Ivan Chong
Data Quality – Ensuring Trust in Your Data
TDWI once published a study that poor data quality was costing companies an estimated $600B annually in the US alone. And yet, industry analysts estimate the data quality tools market to be between $300-400M annually. The demand for better data quality is clearly there, but the size of the data quality tools market does not appear to reflect that same demand. Could it be that traditional Data Quality tools are not delivering enough value for customers? Perhaps instead of focusing merely on data cleansing and data profiling, tools vendors need to be reminded that the goal, ultimately, is getting customers to the point where they can trust their data.
I had a nice conversation with a customer after giving the Data Quality keynote. This person was using PowerCenter for data integration, but a Data Quality product from another company. Until he had the attention and involvement of the business, he stated that he had no reason to trust the data. I was really struck by this statement. Here was a customer who was clearly an expert in the use of matching technology. He had successfully completed an IT project that was processing very large volumes of data – and yet, without the availability of business owners for the data, he was unable to get enough validation on the data so that he could trust it.
We hear these types of comments very often. A great deal has to happen before customers feel they can trust their data. Eventually, data quality vendors will understand how to address the value gap that exists in this market – but the first to figure this out will have a golden opportunity to overtake the incumbents. Until then, data quality will continue to be an underserved market.
Better management through measuring data quality
I recently asked a customer of ours why they invested so much in monitoring and publishing key performance indicators for their data quality. “Believe it or not, the biggest reason we measure data quality is not to correct bad data” came the reply. “The reason we monitor data quality is to detect problems with our business processes.”
Indeed, as I mentioned in my last blog post, business users look to investments in people and processes in addition to technology in order to address poor data quality. For example, if a bank branch manager received a report showing that customer data originating from his branch office had much higher incidents of duplicate entries and was putting the entire bank at risk of massive regulatory fines, he is not going to throw technology at the problem. His response might be mandatory training for tellers or better hiring practices to screen for adequate computer skills.
Experts in quality control methodology refer to this as addressing “root cause.” Common starting points of measurement involve completeness, accuracy, consistency, conformity, duplication, and integrity. Eventually, as the business culture matures its data quality practices, timeliness and data lineage (origination) are used to evaluate quality of data. Of course, software technology that automates the process of parsing, standardizing, matching and consolidating data is of immense value and is an absolute requirement in any data integration project. However, the issue of data quality goes beyond these IT projects. Ongoing measurement and monitoring of data quality provides value directly to the business because it helps them to better manage their people and processes.
Business and IT Collaboration is Essential for Data Quality
A recent InformationWeek article* described the growth in IT employment across the US as a result of a shift in skills. Rather than focusing on pure IT proficiency, organizations are looking for talent with “a more hybrid mix of technology skills, along with an understanding of the business and its customers.”
IT departments are highly motivated to increase the level of collaboration with their counterparts in the business. Nowhere is this more critical than in the area of data quality and the trend is causing a shift in the way companies are looking to solve their data quality issues. First generation data quality tools had a natural focus on technology, instead of business. Here are some of the differences between technology focused data quality solutions and business-focused data quality solutions.
Tools vs. Process
Technology focused data quality solutions provide tools that automate data processing. Evidence of this type of focus can be seen in the way that vendors will tout the sophistication and type of their algorithms over and above their ability to support ongoing data quality management processes. While technology is extremely important, its relevance cannot eclipse the overall data quality management process. Even if your data quality tool can automate the correction of 95 percent of the data, if the remaining five percent cannot be managed properly, you will continue to suffer from poor data quality.
(more…)
Loading ...