Informatica Perspectives

A recent InformationWeek article* described the growth in IT employment across the US as a result of a shift in skills. Rather than focusing on pure IT proficiency, organizations are looking for talent with “a more hybrid mix of technology skills, along with an understanding of the business and its customers.”

IT departments are highly motivated to increase the level of collaboration with their counterparts in the business. Nowhere is this more critical than in the area of data quality and the trend is causing a shift in the way companies are looking to solve their data quality issues. First generation data quality tools had a natural focus on technology, instead of business. Here are some of the differences between technology focused data quality solutions and business-focused data quality solutions.

Tools vs. Process
Technology focused data quality solutions provide tools that automate data processing. Evidence of this type of focus can be seen in the way that vendors will tout the sophistication and type of their algorithms over and above their ability to support ongoing data quality management processes. While technology is extremely important, its relevance cannot eclipse the overall data quality management process. Even if your data quality tool can automate the correction of 95 percent of the data, if the remaining five percent cannot be managed properly, you will continue to suffer from poor data quality.
Physical Data Attributes vs. Semantics
Once data is separated from its original host application it has very little context. Therefore, it is natural that technology focused data quality offerings center on evaluating the physical data structures instead of evaluating consistency and accuracy of the meaning attached to data. There are exceptions in areas where common reference data is applicable (e.g. address validation). However, for much of the data within the enterprise, there is not enough attention being paid to the data quality of information in context. A field containing the number 10044 out of context is not much good to us from a data quality standpoint. A business analyst would ask – “10044 what?” 10044 pounds? 10044 dollars? 10044 liters? Is 10044 a specific code relating to a product attribute? Without the context data quality is meaningless.

See also the interesting post from my colleague Chris McCauley Data Quality Metadata; a lot more than just "data about data" for more on this important subject.

Fixes applied in Technology vs. Fixes applied in People, Processes and Technology
Technology focused data quality offerings limit the type of data quality improvements to fixing data using developer tools. Properly addressing data quality requires more than just fixing defective data; it also requires improving the processes that lead to poor quality data entering the system in the first place. Bridging the gap between technology and business is essential if this is to be achieved. As data quality offerings evolve to have more of a business focus, they offer user interfaces designed with the business professional or knowledge worker in mind and they allow for the type of Business-IT collaboration that was highlighted by InformationWeek. Business users have a very different outlook on data quality. They are just as likely to invest in training, hiring, and business process reengineering to improve data quality as they are to invest in additional technology. The resulting effect is the data quality management process, information in context and data quality metrics get increased attention and investment.

* You can read the InformationWeek article at In Growing Job Market, IT Pros Get More For The Soft Skills