By Harold Leusink
Data is like Truth.
It is always from a certain point of view.
I contend that a data warehouse can be a great way to burn money. The moment we business leaders agree to start moving and transforming data into a central location is the precise moment that our costs can go up and the value go down.
There are three reasons that explain this reality:
1. Data only has meaning in context
2. Transformations by definition take data out of context
3. Central teams tasked with managing the data do not understand the context
At the core, data has meaning only in context. For example, sales numbers have context for the sales department only in terms of sales date. Sales data gathered by the department follows a process deliberately designed to keep it in that context. Every step through each system, every interaction and every database table where it is stored, is structured in the context of sales date. Within the sales department, any time that it is moved, processed, aggregated or reported, data always remains in context.
Those same sales numbers within the shipping department only have context in terms of shipping date. Shipping performance metrics are tied to shipping date. SLAs are defined in terms of shipping date or perhaps delivery date. Every process within shipping is designed to keep the data in that context. So if a sales manager decided to query data from a shipping database table, or view sales performance on a shipping report, the data presented by shipping date would be out of context for sales, and it would have little meaning for the sales manager.
Transformations by definition move data out of context. BI or analytics teams have to change data they get from one department, in order for it to gain context to others. And herein lies both the challenge and the danger. BI teams won’t understand the complete context. In fact, they don’t even need to care about that context. All the BI team needs to know is from where to pull what data, and how that data needs to be stored. For example, the BI team doesn’t need to care about the business ‘why’ behind the data, their responsibility is only to ensure that the data as stored, meets the specified requirements. They say things like, “Ok, we can store sales numbers by both sales date and shipping date, and then the BI tool will let you decide which you want to see at the time.”
The result is that in the data warehouse, we are at risk of accidentally erasing or mixing up context. And it is never as simple as just a date. We encounter situations where a sales invoice in one department is associated with three work orders in another department and no machine way of making the association. And the way finance needs to see the transaction is by salesperson regardless of date, unless a return happened, and in that case the value of the transaction is determined only after a specified formula is applied. You get the idea, and we haven’t even discussed any of the challenges that creep in as we wrestle with the large scope of corporate wide data. The challenges are multiplied with the addition of each department’s context.
As BI teams gather requirements, load and transform data, build reports and dashboards, we quickly realize that it’s difficult to account for all context, especially up front. And over the next few years we continue to find exceptions and make context tweaks or adjustments. After long projects, numerous meetings, validations, data fixes, custom reports, context sensitive calculations and transformations, we end up with a mix of reports, dashboards and exports to Excel.
So what’s the answer? At very least make sure that we understand the challenges that come with moving data out of context. And the reality that transformations by definition do exactly that. But what if we approach it differently? What if we keep the data in context? We let the people who enter and use the data define that context. The trusted data stays within the area, after all, they know it best. And by keeping data in context we ensure that the meaning is preserved; high quality data, all the time. To handle cross-departmental reporting, we create a standard architecture that each area must use to expose their data to the entire organization. It’s mandated. Context is defined and provided via a data catalog. So any area may pull data using the known common architecture, referencing the data catalog as the context guide. At any time they can combine any data with their own, and the base data always remains neatly in context within the source area. Data scope is also reduced as each area now only concerns itself with the scope they need to support their necessary decisions.
The issue is never how fast we can get data, it’s always been about the quality of data. The objective of the first adding machines in the early 1900s was increased accuracy, not efficiency. Regardless of how fast we get it, we only make good decisions from good data.
For additional information