Recently I spoke at a User Group Meeting on the topic “Align for Success: The critical part Data Quality plays in complex Business and IT Initiatives.” I began the discussion by polling the group to find out how many of the organizations represented had a data quality solution in place. The response to the question was mixed, with approximately half the audience indicating they either had a solution or were considering one, while the other half indicated they weren’t currently considering data quality (or the person was unaware of any data quality initiatives). Although this was a very unscientific survey, it set the tone for my presentation, as I attempted to explain the concept of a data quality maturity model. By understanding where an organization is today from the standpoint of the model, management can begin to develop plans as to where they want to end up both in the short and long term.

An Eye-opener for Your Business: How Data Profiling Can Build Support for Data Quality within Data Management Projects

Q: Who in the organization should be responsible for data quality? And who should sign off on the scope document?

Today I am doing a webcast with Informatica on just this topic. The answer lies in your ability to demonstrate the true state of the data -- "seeing" the problems by performing data profiling on your data. Nothing says "Houston, we have a problem" more than viewing the profiling results. It will make believers out of the most difficult people!

I hope you enjoy the presentation.

Update: Questions and answers from this event can be in the following post - Profile Early, Profile Often

The Webopedia.com CDI definition went: “Short for Customer Data Integration, it is the combination of the technology, processes, and services needed to create and maintain an accurate, timely and complete view of the customer across multiple channels, business lines, and, potentially, enterprises, where there are multiple sources of customer data in multiple application systems and databases.”

A bit long winded perhaps, but the three words that shone out at me through the glare of the florescent lights in San Francisco airport were “accurate, timely and complete”; all data quality issues. Despite this, few if any of the Customer Data Integration (CDI) vendors in the market today have truly addressed the data quality issues in their CDI solutions. And anyone who has gone down the route of developing their own custom-built CDI application will be all too familiar with the data quality demands involved.

This blog is the first of a series of three blogs on the critical quality characteristics (or measures) of information quality required to achieve Total Information Quality Management.

1. Information Product Specification data quality
2. Data content quality
3. Information presentation quality

What constitutes the “Information Product Specifications” data?

• Information standards
• Data names
• Data definitions
• Attribute valid value set or range of values
• Value format for structured attributes (VIN, SSN, Product Codes)
• Business rule specifications of constraints on data
• Information Steward accountable for data definition quality

While grabbing a bite to eat prior to a flight, I overheard two gentlemen who were both retired from the newspaper business discussing how people don’t really take the time to read a newspaper like they used to. They were lamenting that people today preferred quick sound bites of information – whether it be from television or from reading one of the various news sites on the Internet.

Indeed, as I mentioned in my last blog post, business users look to investments in people and processes in addition to technology in order to address poor data quality. For example, if a bank branch manager received a report showing that customer data originating from his branch office had much higher incidents of duplicate entries and was putting the entire bank at risk of massive regulatory fines, he is not going to throw technology at the problem. His response might be mandatory training for tellers or better hiring practices to screen for adequate computer skills.

Experts in quality control methodology refer to this as addressing “root cause.” Common starting points of measurement involve completeness, accuracy, consistency, conformity, duplication, and integrity. Eventually, as the business culture matures its data quality practices, timeliness and data lineage (origination) are used to evaluate quality of data. Of course, software technology that automates the process of parsing, standardizing, matching and consolidating data is of immense value and is an absolute requirement in any data integration project. However, the issue of data quality goes beyond these IT projects. Ongoing measurement and monitoring of data quality provides value directly to the business because it helps them to better manage their people and processes.

The fact is even when the customer is responsible for “bad data” the vendor still has to shoulder the blame and do something about it.

Take the example of a friend of mine who recently signed up with a well known social networking website. Let’s ignore the reasons why he signed up – he’s not entirely sure – or what social networking sites are for – none of us is entirely sure yet, but I guess we’ll start to get the point as more and more of them are sold to industry giants for n billion dollars.

It all got me thinking about a recent blog post by my esteemed colleague Garry Moroney. His post Mobilizing the Data Quality Army outlined the level of effort, thought and planning that the US Department of Education is putting into data quality.

As Garry points out dealing with data quality in a large, disconnected organization such as the US schools system is not a trivial exercise. But if you were to only read that one post you might be overwhelmed by the potential size of the data quality task in front of you.

IT departments are highly motivated to increase the level of collaboration with their counterparts in the business. Nowhere is this more critical than in the area of data quality and the trend is causing a shift in the way companies are looking to solve their data quality issues. First generation data quality tools had a natural focus on technology, instead of business. Here are some of the differences between technology focused data quality solutions and business-focused data quality solutions.

Tools vs. Process
Technology focused data quality solutions provide tools that automate data processing. Evidence of this type of focus can be seen in the way that vendors will tout the sophistication and type of their algorithms over and above their ability to support ongoing data quality management processes. While technology is extremely important, its relevance cannot eclipse the overall data quality management process. Even if your data quality tool can automate the correction of 95 percent of the data, if the remaining five percent cannot be managed properly, you will continue to suffer from poor data quality.

As a new contributor to the data quality blog site, I wanted to start by introducing myself and highlighting the types of topics I plan to discuss on a semi-frequent basis. I am a Principal Consultant with Informatica Professional Services and have spent the past 6 years in the data quality space in a variety of sales and post sales roles. During this time I have seen the data quality market continue to evolve and mature. Thus, I would like to use this column to reflect on the types of use cases I have seen and continue to see when meeting with organization’s faced with data quality problems. I hope these posts can start an active dialogue, regardless if your company is trying to tackle their first data quality initiative or looking to build out a formal center of excellence around data quality.

To start, I wanted to pose a common question I am often asked by clients and prospects – how do I build a business case for data quality? Although an organization may think (or even know) there is a problem, the need to justify the cost around procuring a data quality solution often exists. This justification requirements often comes from the idea that data quality issues aren’t necessarily a core business issue (how wrong this is!) or something that can be handled through manual intervention (this is true – if you have unlimited time and money, but even then your results will be limited). Thus, the following points are meant to help start an organization down the path to building the internal business case through a Data Quality Audit. Note - if you have access to Informatica’s Velocity Methodology, I go into these steps in further detail in the best practice document, “Developing the Data Quality Business Case.”

Hospital billing mistakes have become so prevalent that a niche industry has evolved to help patients decipher their bills and help correct the errors. ”Pat Palmer, founder of Medical Billing Advocates of America, estimates that she finds multiple errors in 8 out of every 10 hospital bills she reviews” (Dina ElBoghdady, “Killer Billing Errors,” Washington Post, June 27, 2004, p. F01. Accessed Aug 9, 2007 at: http://www.washingtonpost.com/wp-dyn/articles/A7351-2004Jun26.html ).

One patient saw the bill of $25,652.14 for her 2-hour routine operation, reduced to a cost of $17,000 billed to the insurance company and her cost reduced to $2,148, after correcting errors and overcharges.
While this error-rate was not statistically measured, nor would it apply to all hospital bills, it points out a huge problem in Information Quality in the health care system, not counting actual medical errors caused by information defects.

The consequence of health care over billing is significant. It contributes to the high costs of health care, which is one major cause of personal bankruptcy.

“The thing that they all have in common is a desire to cut corners and deal with quality later. It takes a lot of energy to be the information quality cheerleader, and I find it discouraging and overwhelming at times. Keep writing your articles and books to encourage all the people like me who are dealing with these issues every day.” P. G.

The discovery that P. G. has experienced is, unfortunately, the norm—not the exception. There are two critical elements in this experience.