Informatica Data Quality

If you ever find yourself discussing the benefits of data quality for your business and one of your associates asks rhetorically, "Yes, but can it solve world hunger?" you now have an answer for them.

The Food and Agriculture Organization of the United Nations records the level of completeness for data collection from each member nation. On their website, their stated mission is to work towards "a world without hunger." A key element in their fight against hunger is the FAO Stat database and a key means of maintaining the efficacy of the data is their data quality dashboard.

For organizations working with the FAO, it's important that the data be accurate - otherwise perishable goods may be wasted by getting shipped to locations not suffering from malnourished populations. This example highlights something that I've seen very often in the context of enterprise data quality initiatives. Many prospective customers come to us and ask "how do we get started, given the complexities of coordinating across multiple organizations inside our company?" Within the Informatica customer base, there are many examples of successful initiatives starting off with Data Quality metrics and dashboards. The metrics offer a great way for organizations to maintain a dialog on how to prioritize their investment in data quality.

Already, I've received email comments on my posting. "Can Data Quality allow us to live longer? Facilitate the exploration of outer space?" Great questions… stayed tuned for future postings!

I recently asked a customer of ours why they invested so much in monitoring and publishing key performance indicators for their data quality. “Believe it or not, the biggest reason we measure data quality is not to correct bad data” came the reply. “The reason we monitor data quality is to detect problems with our business processes.”

Indeed, as I mentioned in my last blog post, business users look to investments in people and processes in addition to technology in order to address poor data quality. For example, if a bank branch manager received a report showing that customer data originating from his branch office had much higher incidents of duplicate entries and was putting the entire bank at risk of massive regulatory fines, he is not going to throw technology at the problem. His response might be mandatory training for tellers or better hiring practices to screen for adequate computer skills.

Experts in quality control methodology refer to this as addressing “root cause.” Common starting points of measurement involve completeness, accuracy, consistency, conformity, duplication, and integrity. Eventually, as the business culture matures its data quality practices, timeliness and data lineage (origination) are used to evaluate quality of data. Of course, software technology that automates the process of parsing, standardizing, matching and consolidating data is of immense value and is an absolute requirement in any data integration project. However, the issue of data quality goes beyond these IT projects. Ongoing measurement and monitoring of data quality provides value directly to the business because it helps them to better manage their people and processes.

This week NASA announced that it may have discovered evidence of water flowing on the surface of Mars in the recent past. This raises the possibility of life existing on Mars in the past and even to the present day.

A lot of talent, time and money has gone into addressing the fundamental question - is there life on Mars. In addition to the two rovers on the surface, there are currently three spacecraft in operation around Mars; Mars Odyssey, Mars Reconnaissance Orbiter and Mars Express – sadly the Mars Global Surveyor which is responsible for this weeks exciting news has probably suffered a severe failure and is in effect lost. Each spacecraft has been sent out to gather some basic statistics about the Red Planet such as; how the surface changes over time, the percentage of carbon-dioxide at the poles or how the temperature varies throughout the atmosphere. All of this in the hope that we move closer to a definitive answer to that fundamental question. But forget the answer: are we asking the right questions?

[Read more]

I attended my first parent-teacher meeting the other day for my five-year old daughter. Another one of those “life stage” events done and dusted – I remember dreading the annual meeting when I was a kid. The notion of my parents and my teacher comparing notes on my behaviour was too much to bear – somebody was eventually going to put two and two together and find out I was up to no good.

It all got me thinking about a recent blog post by my esteemed colleague Garry Moroney. His post Mobilizing the Data Quality Army outlined the level of effort, thought and planning that the US Department of Education is putting into data quality.

As Garry points out dealing with data quality in a large, disconnected organization such as the US schools system is not a trivial exercise. But if you were to only read that one post you might be overwhelmed by the potential size of the data quality task in front of you.
[Read more]

I recently received an email from one of my early clients. After having worked in four different companies in four different industries, she came to a sad conclusion, writing:

“The thing that they all have in common is a desire to cut corners and deal with quality later. It takes a lot of energy to be the information quality cheerleader, and I find it discouraging and overwhelming at times. Keep writing your articles and books to encourage all the people like me who are dealing with these issues every day.” P. G.

The discovery that P. G. has experienced is, unfortunately, the norm—not the exception. There are two critical elements in this experience.

[Read more]

“In e-Business, the Information IS the Business”
Having just completed writing a chapter on “IQ in the Internet and e-Business Environments” in my forthcoming book, Information Quality Applied: Best Practices for Improving Business Information, Processes and Systems (John Wiley & Sons), I wanted to share a few excerpts from this chapter. This is one of ten chapters focused on applying sound quality principles to the unique quality issues in various information value “circles” such as “Prospect to Satisfied Customer,” “Order to Cash” Supply chain, for example.
There are three categories of information in the Internet environment to which quality principles must be applied:
* Web-Based Documents and Web Content
* Data “Shared” by Internal Processes and Internet Processes
* Information Collected or Created in e-Commerce and e-Business value chains, including third party business partners
The major problem with IQ in the Internet is that business is conducted in “cyberspace” with no person “minding the store” or monitoring the e-Business transactions.
Here I will address some problems and improvements in the first category.
[Read more]

These days, savvy business executives understand that a report or analysis on their customers, markets, products or anything else is only as good as the data used to compile it. There is always a risk that the data used in the report may not be of sufficiently high quality. Similarly, business partners realize that integrating their systems with another company’s will only add value if the data flowing through the integration meets the required standards.

So more and more the message that data consumers are giving to data providers is: “before I accept this data from you, before I use it in my decision making processes or write it into my systems or pass it on to another party, prove to me that this is high quality data. Prove to me that the data is fit for purpose.
[Read more]

IQ in the News
Most people have probably heard about the highly reputable Consumer Reports' recall of its flawed testing of infant car seat safety. The report, issued January 5, 2007, found that many car seats failed the high-speed side impact test it conducted (the government requires passing of frontal crashes of 30 mph. Consumer Reports tested at 35 mph for frontal crashes and 38 mph (so they thought) in side impact crashes. The findings seemed to indicate a high degree of failure with nine failing some or all of the crash tests, and only two doing well in all tests.
However, the government found a problem with the way the testing was conducted. Instead of a 38 mph side crash, the test simulated a side-impact crash of over 70 mph with very inconsistent results that would have come from 38 mph tests. Consumer Reports recalled the entire report January 18.
IQ Lessons Learned From the Consumer Reports Recall:
• Negative impact on consumers and their confidence in the organization:
The impacts of the faulty testing where dramatic and swift. The Executive Director of the Washington State Safety Restraint Coalition exclaimed that "Consumer Reports screwed up….They really upset people and created enormous confusion."
• When designing tests, as you will with IQ assessments, you must assure you design the tests properly. Measuring validity and accuracy are two distinctly different measurements. You can test validity by defining the business rules, valid values or ranges the data must conform to, and conduct these tests electronically with IQ assessment software or your own validity routine tests. But to measure accuracy, you must confirm the data values correctly correspond to the characteristic of the real world object or event, the data represents. To perform this test, you must compare the data with the characteristic of the real world object itself.
[Read more]

In my last blog post, I discussed the topic of building the business case for data quality. As such, one of the points I mentioned was the need to highlight resulting issues. Since my last post, I have had a number of discussions with clients and prospects on this topic. At the core of these discussions is the same fundamental question – what is the best way to package up the issues we uncover? In answering this, I often discuss the six dimensions of data quality (Completeness, Conformity, Consistency, Accuracy, Integrity, and Duplicates) and how to use a data quality scorecard to present the information in a meaningful way that it can be shared with key stakeholders within the organization. Although my response to this question remains the same, a conversation I overheard at the airport a couple of weeks ago made me look at the need for a DQ scorecard a little differently.

While grabbing a bite to eat prior to a flight, I overheard two gentlemen who were both retired from the newspaper business discussing how people don’t really take the time to read a newspaper like they used to. They were lamenting that people today preferred quick sound bites of information – whether it be from television or from reading one of the various news sites on the Internet.
[Read more]