Quality Data: A foundation for BI Success
Data Quality refers to the degree of excellence exhibited by the data in relation to the portrayal of the actual scenario. Data are of high quality ‘if they are fit for their intended uses in operations, decision making and planning’ (J. M. Juran). Alternatively, the data are deemed of high quality if they correctly represent the real-world construct to which they refer. Data quality is really the only truly unique asset that a company has.
A company needs to care about its information because:
- Information volume is exploding
- Lot of information is generated real time
- Information is a key business asset
A company can have products, but those products can be and are regularly copied. A company can distinguish itself by its service, but its service is highly reliant upon good data. Data is very strategic, because it’s used for both internal and external decision-making. You need that high degree of reliability from and high degree of confidence in your data because it impacts your operation capabilities on a day-to-day basis.
Business Intelligence (BI) is the buzzword of the moment. BI is built on a foundation of high quality data. A good BI can have a major impact on driving improvements in data quality. According to industry analyst firm Gartner, more than 50 percent of business intelligence and customer relationship management deployments will suffer limited acceptance, if not outright failure, due to lack of attention to data quality issues.
The impact of poor data quality is far reaching and its affects are both tangible and intangible. If data quality problems are allowed to persist, executives grow to mistrust the information in the data warehouse and will be reluctant to use it for decision-making.
There are various ways in which Data quality can be improved to increase trust in data:
A. Improve how data is entered: In case of legacy systems, it becomes harder to evaluate data, but in case of modern systems, various ways of validating data like setting conditions and providing drop down option can be used. However validating each field is only the start, entries that are valid may be nonsensical. A typical example here is with dates. It is unlikely that an order was placed in 1910 for example, or – maybe more typically – that an item was delivered before it was ordered. This leads us into the issue of combinations of fields.
B. Problem of Interface: In many organisations the IT architecture is much more complex than a simple flow of data from a front-end system to BI and other reporting applications. Often modern front end systems wrap older legacy systems that are too expensive to replace or too integrated into the infrastructure of organization. Also there may be number of different systems handling different parts of organization.
Many a times there is no common organization wide dictionary defining the naming and validation of field across all systems and hence data may be recorded in different forms/ways in different systems. Mapping has to be made for data passing between various systems. This leads to a lack of transparency between what goes in to one end of the “machine” and what comes out of the other. It also creates multiple vulnerabilities to data being corrupted or meaning being lost along the way.
C. Track source of bad data: Audit reports can help track source of bad data but just knowing will not make a difference. Action needs to be taken to curb instances of bad data. Data quality is either important to the organisation, or it is not. There is either a unified approach by all management, or we accept that our data quality will always be poor. In summary, there needs to be a joined-up approach to the policing of data and people need to be made accountable for their own actions in this area.
D. Highlight bad data in BI: Many a times bad data is filtered out of BI reports. Instead display bad data in BI report as they help in pointing out and help in finding the source of bad data. The act of suppressing the bad data is similar to hiding the truth. The senior management are denied of the opportunity to see true picture of what is happening in company. Quality of data improves dramatically if executives become aware about how it affects the information they need to run the business.
But for this to happen, senior management need to place value on the BI they receive and realize the importance of its accuracy.
E. Transparency: Information transparency is way to maintaining high data standards. Executives need to be able to see exactly where the data came from, how reliable it is, how it was defined or manipulated, and when it was last updated and BI can help executives with the same.
F. Bench-marking and external data: Your own systems will rarely contain all the information you need to make decisions. In order to determine your real performance, and make the right decisions, you need to be able to compare your numbers with the economy, the market, or your competitors. BI tool can be designed to achieve that.
When creating a data quality strategy, there are six factors, or aspects, of an organization’s operations that must be considered. The six factors are:
- Context—the type of data being cleansed and the purposes for which it is used
- Storage—where the data resides
- Data flow—how the data enters and moves through the organization
- Workflow—how work activities interact with and use the data
- Stewardship—people responsible for managing the data
- Continuous monitoring—processes for regularly validating the data
Posted on January 16th, 2012 by Sanjay Mehta
Filed under: Business Intelligence, Emerging Trends, View Points & Perspective
Very informative site with lots of resources for data quality, MIS and business intelligence. Thanks
@Palesa, You can find more on KPI for Educational institutions here: http://blog.maia-intelligence.com/2009/09/07/kpi-for-educational-institutions/
I came across this before, but this gave me more information on data quality for business intelligence. Thanks for the post!