Informatica Data Quality Blog

Data Quality Maturity Model – How Does Your Organization Rate?

2008-05-09T18:23:12Z

Recently I spoke at a User Group Meeting on the topic “Align for Success: The critical part Data Quality plays in complex Business and IT Initiatives.” I began the discussion by polling the group to find out how many of the organizations represented had a data quality solution in place. The response to the question was mixed, with approximately half the audience indicating they either had a solution or were considering one, while the other half indicated they weren’t currently considering data quality (or the person was unaware of any data quality initiatives). Although this was a very unscientific survey, it set the tone for my presentation, as I attempted to explain the concept of a data quality maturity model. By understanding where an organization is today from the standpoint of the model, management can begin to develop plans as to where they want to end up both in the short and long term.

]]> Gartner’s Data Quality Maturity Model (http://www.gartner.com/DisplayDocument?doc_cd=139742&ref=g_rss) is comprised of five phases – Aware, Reactive, Proactive, Managed, and Optimized. In order for an organization to move along the curve from the awareness phase an attitude shift needs to occur. An organization must move from seeing data quality as an initiative that provides few benefits to something that is considered a core asset to the organization (especially in the proactive and managed phases). As attitudes begin to change, so does the value proposition for data quality, as it moves from being viewed as a cost that could be eliminated to being considered a strategic initiative within the company. Understanding where your organization fits in this model is crucial, as any long term plans that an organization may be considering, such as a Data Governance or MDM initiative, will be hampered if the concept of data quality is still somewhat immature within the company.

In my previous post I discussed the cultural aspect of data quality and how change doesn’t happen over night. Similarly, moving along the maturity model to the point that data quality is seen as a strategic component will take time and require an attitude shift. Even with support from upper management, the best approach to implement a data quality strategy in the organization is something that will need to be hashed out through a series of discussions and iterations of data quality analysis. Although this might seem to be a time consuming process, the end result will be a better understanding of the underlying data that is ultimately driving the key business decisions within your organization. In addition, the process of moving through the maturity model should provide an added benefit - increased collaboration between business and IT stakeholders within the organization.

Until next time...

Profile Early, Profile Often

2008-04-08T16:12:54Z

Dr. Claudia Imhoff, President & Founder, Intelligent Solutions and Ed Lindsey, National Product Specialist, Informatica answer some questions that were raised during our recent web seminar. If you missed the web seminar, you can listen to it by clicking the following link:

An Eye-opener for Your Business: How Data Profiling Can Build Support for Data Quality within Data Management Projects

Q: Who in the organization should be responsible for data quality? And who should sign off on the scope document?

]]>
Claudia Imhoff, Intelligent Solutions:
1. There are several organizations responsible for data quality – data stewards, database administrators, and data administrators. I suggest that you look into creating a data quality program in which all these groups have representation. The program manager should report to the CFO or COO.

2. The scope document should be signed by the IT and business sponsors. If there are high level influencers (like a VP of Sales, CFO or COO) then they should sign as well.

Q: Last year my company identified that we needed a data governance organization and policy. The challenge is in identifying data stewards as rolls and responsibilities are changed through company reorganization, or through people changing roles. What is the best way to identify data stewards and manage the changes over time?

Claudia Imhoff, Intelligent Solutions: Data Stewards should come from the business. Generally they are the few people who demonstrate a true interest in the data and information you are generating. I suggest you appoint (either formally or informally) one person per subject area to be the main steward for that subject area. They may choose to have others in their committee but they remain responsible for any issues within their area.

Q: What tools/methods are available to come up with ‘effort estimates’ for source system analysis?

Claudia Imhoff, Intelligent Solutions: I suggest using data profiling to give you an idea of the effort it will take to do the source system analysis. The profiling results give you the right information to let you know what to expect from the SSA.

Q: Can I do Data Profiling on a table (DB2, Oracle, SQL Server)? Or does it have to be a flat file?

Ed Lindsey, Informatica: Yes, we profile data in a RDBMS system via ODBC or native connectivity for DB2 load format, Informix, Oracle, Sybase and UDB.

Q: You talk about doing analysis on a database; can your tool also do analysis on files outside of a database? e.g. Excel, text files, etc.

Ed Lindsey, Informatica: Yes, we profile DB2 load format, delimited or fixed length and XML format.

Q: Can data profiling be done on mainframes (IMS)?

Ed Lindsey, Informatica: Normally we extract the IMS data to a flat file or stage it in a RDBMS and profile it there.

Q: How well does Informatica's offering support injecting business rules from Data Quality into the ETL process if PowerCenter is used?

Ed Lindsey, Informatica: Power Center can embed a Data Quality plan in a mapping and run in on the Power Center Server. This way you can have a business user work on the business rules and have the same rules executed as a DQ Scorecard, DQ process and the ETL process.

Q: What are best ways to fix identified DQ issues? When would you choose one over the other?

Ed Lindsey, Informatica: The DQ process is different from customer to customer. The best place to fix a data quality problem is at the source. However, many customers will not allow modifications of the data at the source for fear of breaking the original system. Also, a lot of the data is not under the control of the department using the feeds because it comes from outside the company or business unit. Most of the time the data is corrected as it enters the business unit as part of an operational data store, data warehouse or enterprise application. Over time, as DQ issues are corrected downstream, the data customer gives their feedback to the provider and hopefully they initiate their own DQ process so that over time quality ultimately finds it way back to the source system.

Q: Would you suggest doing preliminary data profiling across data sets which collects the same subject data into many different data warehouses when you need to move towards an Enterprise Datawarehouse?

Ed Lindsey, Informatica: I would profile the data from any source system before it is loaded into the Data Warehouse. The closer to the source system the better. However, one difference here is you must make sure that other downstream systems are not injecting DQ issues into the process. And yes, we have customers that profile the data before and after the ETL process to make sure that that process has not injected issues as well. As I said during our recent web seminar: Profile early, Profile often.

Seeing is Believing

2008-03-12T16:40:05Z

You know the expression "Seeing is believing"? Well, it is even more true for data management projects. Getting sponsorship much less funding for data management projects like data quality, CRM, MDM, data warehousing, or other such enterprise-encompassing initiatives is very difficult at best. What should you do?

Today I am doing a webcast with Informatica on just this topic. The answer lies in your ability to demonstrate the true state of the data -- "seeing" the problems by performing data profiling on your data. Nothing says "Houston, we have a problem" more than viewing the profiling results. It will make believers out of the most difficult people!

I hope you enjoy the presentation.

Update: Questions and answers from this event can be in the following post - Profile Early, Profile Often

Rome Wasn’t Built in a Day and Neither is a Data Governance Initiative

2008-02-15T17:44:19Z

In my previous posts, I have discussed building the business case for data quality as well as the role that a data quality dashboard plays in supporting this case. As previously noted, these efforts will directly impact your ability to articulate the need to pursue a data quality initiative. The reason for returning to this topic is that I have recently participated in multiple discussions with a variety of companies that were either in the process of forming a data governance council or in the process of building the internal business case to support exploring a data governance initiative. In these discussions two common threads were present – the role of data quality in the data governance initiative and the need to change the culture within the organization if data governance is going to succeed. Although these are only two aspects to consider when pursuing a data governance initiative, they are directly tied to the underlying success or failure of the program.]]> As it pertains to data quality, the key point I often discuss in these meetings is that without a firm understanding of the types of issues in the underlying data and an action plan to remediate these issues, the governance program will ultimately be flawed. The reason is that any decisions made based upon the data assumes it is correct and that a single version of the truth exists within an organization. The concept of governance is to drive to this common version of the truth where data definitions and processes are standardized at the enterprise level. In each of these scenarios, data quality serves a key role, as it allows an organization to monitor and then remediate issues as they arise. By leveraging data quality software and processes on an ongoing basis, the organization is able to monitor and address data quality on an iterative basis. In a sense, you can consider the data quality scorecard a report on the overall health of your organization’s data. If data issues are not addressed, the data related processes or definitions are likely to have issues as well, which compromises the intent of a data governance initiative.

Although data quality plays an important role in any data governance initiative, another key aspect that should be addressed is the cultural impact within an organization. Although processes can be put in place and data quality software can be licensed and implemented, the ultimate success or failure of the initiative will be determined by the acceptance (or lack thereof) within your organization. To that point, data governance needs executive sponsorship within an organization and should be initiated via a top down approach. Without this executive sponsorship and the willingness to make data governance a strategic initiative to the organization, the program is unlikely to gain the traction needed. The reason is often cultural, as people find it easier to continue to do business as usual. To address this, I often request that the various key stakeholders from across an organization are present when we conduct a data governance workshop, as I want to observe how everyone interacts as well as listen to any concerns, criticisms or objections that might be present, as these are the very issues that must be addressed if a governance program is to succeed.

Just as the title of this blog states, “Rome Wasn’t Built in a Day and Neither is a Data Governance Initiative” - as your organization begins to consider a data governance initiative this is something you should remain cognizant of, as the time to initiate a governance program is going to vary based on the types of issues noted in this post. If you do not have executive buy-in and if significant resistance to governance exists, the time to implement a program is going to be longer, as there is going to be a need for educational workshops to address concerns. If you have this sponsorship in place, you should begin to examine the quality of the data that the processes, policies and standards will be built upon.

Until next time….

You can’t have CDI without Data Quality

2008-02-01T10:13:00Z

Looking in Webopedia.com recently I came across a definition for CDI. Yes webopedia.com - it bills itself as the #1 online encyclopaedia dedicated to computer technology. You might wonder what I was doing surfing this font of knowledge – well I had time on my hands between delayed flights coming back to Europe from the US. You know what they say “time to spare, travel by air.”

The Webopedia.com CDI definition went: “Short for Customer Data Integration, it is the combination of the technology, processes, and services needed to create and maintain an accurate, timely and complete view of the customer across multiple channels, business lines, and, potentially, enterprises, where there are multiple sources of customer data in multiple application systems and databases.”

A bit long winded perhaps, but the three words that shone out at me through the glare of the florescent lights in San Francisco airport were “accurate, timely and complete”; all data quality issues. Despite this, few if any of the Customer Data Integration (CDI) vendors in the market today have truly addressed the data quality issues in their CDI solutions. And anyone who has gone down the route of developing their own custom-built CDI application will be all too familiar with the data quality demands involved.

]]> CDI or customer data hubs, a subset of the wider master data management field, have become a focus for large organizations struggling to improve customer service and the operational efficiency of all customer interactions. Like all complex IT projects the CDI vision can be limited by the usual constraints of inadequate budgets, tight timelines and capabilities. In the pressure to get the CDI project completed organizations have a tendency to push the data quality issue to the side, and also to take more tactical than strategic views of the CDI implementation.

I would argue that this is a mistake; if you want to create a successful customer hub it is essential to focus squarely on data quality and take as broad a view as possible in terms of CDI strategy – especially in the planning phase.

So even if you don’t have time on your hands, and you are pressed to the pin of your collar to implement the project sooner rather than later, don’t forget the data quality. Defective data quality not only limits return on investment (ROI) from a CDI implementation, but it will ultimately lead to outright failure of the solution. The CDI project will not alleviate the problems that can mean poor relationships with customers, vendors, suppliers, regulators, and other stakeholders which can result in poor decisions and missed business opportunities.

On the other hand when data quality is central to the CDI implementation, the business benefits can be vast: increased customer satisfaction, greater customer loyalty, improved revenue and profit, decreased operational costs, and greater regulatory compliance.

At the end of the day I believe that the ultimate success of CDI hinges as much on good data quality as it does on anything else.

IQ and Information Product Specifications Quality

2008-01-20T12:15:06Z

One of the root causes of poor quality information is defects in the data definition, specifically the “information product specifications.” Because information is a product of our business, manufacturing and service processes, the analogy of an “information product” is real, and the requirement for quality in “information product specifications” is a critical requirement for Information Quality.

This blog is the first of a series of three blogs on the critical quality characteristics (or measures) of information quality required to achieve Total Information Quality Management.

1. Information Product Specification data quality
2. Data content quality
3. Information presentation quality

What constitutes the “Information Product Specifications” data?

• Information standards
• Data names
• Data definitions
• Attribute valid value set or range of values
• Value format for structured attributes (VIN, SSN, Product Codes)
• Business rule specifications of constraints on data
• Information Steward accountable for data definition quality

]]> Quality of Information Product Specifications data includes:

• Information standard quality: Standards are enterprise-focused, adopted by all information stakeholders and developers. Enterprise information standards are always used to ensure consistency of data names, definitions, business rule specifications in the same way that Financial standards (GL Chart of Accounts) are adopted by all in developing budgets.

• Data name quality: Data names are common and intuitive to all information stakeholders in labelling classifications of objects or events (entity types) and attributes (facts) that the enterprise needs to know about.

• Data definition quality: Definitions of business concepts (business terms), entity types and attributes completely, correctly and clearly define the real world object (entity type), fact (attribute) or business concept (business term) that the enterprise needs to know about, so there is no misinterpretation of data across business areas. Definitions must be of a single class of real world objects (entity type), characteristic (attribute) of a real world object or business concept that generally does not show up as an entity type or attribute, but may be used in an entity type or attribute definition.

• Attribute valid value set or range of value quality: Classifications of things, such as country code, gender, territory, industry code or medical procedure, require standardized codes or names to assure consistency and avoid value synonyms (different codes mean the same thing) or homonyms (same code means different things).

• Value format for structured attributes (VIN, SSN, Product Codes) quality: Value structures should be standardized at the most global level possible, and must be standardized at an enterprise level. Well-defined formats for certain types of information generally allow easy recognition and memory of values and their meaning. Such formats should be able to reduce errors in data capture or in information presentation.

• Quantitative attribute value UoM (measurement, currency amounts) quality: Quantitative attribute values have explicit and unambiguous Unit-of-Measure associated with the value. This prevents misinterpretation of numeric values.

• Business rule specification quality: The specifications for constraints on data should represent the inherent constraints on the real world objects and events and their characteristics and secondarily the enterprise’s business policies. Failing to define business rules that are inherent to the real world can lead to faulty information model relationships and database designs. Faulty information model relationships and database designs can lead to unstable designs and high modification costs—and at worst—inability to share information and high costs of redundant databases and data transformation and movement. The business rule specifications must be able to be implemented in both manual tests and within the database designs and software in a way that is easily verifiable and modifiable should they change.

• Business Information Steward accountable for data definition quality: Every data object (business term, entity type and attribute) should have both a business subject matter expert and an Information Resource specialist approve all information product specification data for the set of Information Resource Data they oversee for the well-being of the enterprise and its customers and stakeholders.

For more about Information Product Specification Quality, see Chapter 5, “Assessing Data Definition and Information Architecture Quality,” in Improving Data Warehouse and Information Quality. This contains a more comprehensive list of quality characteristics with examples. It also describes how to measure these quality characteristics, including how to measure data reuse.

What do you think? Share your thoughts about “Information Product Specification Data” and how to help bring in the Information Quality Revolution!!! The next blog will discuss data content quality characteristics.

Data Quality Dashboard – Capture Your Audience’s Attention

2007-12-22T01:11:47Z

In my last blog post, I discussed the topic of building the business case for data quality. As such, one of the points I mentioned was the need to highlight resulting issues. Since my last post, I have had a number of discussions with clients and prospects on this topic. At the core of these discussions is the same fundamental question – what is the best way to package up the issues we uncover? In answering this, I often discuss the six dimensions of data quality (Completeness, Conformity, Consistency, Accuracy, Integrity, and Duplicates) and how to use a data quality scorecard to present the information in a meaningful way that it can be shared with key stakeholders within the organization. Although my response to this question remains the same, a conversation I overheard at the airport a couple of weeks ago made me look at the need for a DQ scorecard a little differently.

While grabbing a bite to eat prior to a flight, I overheard two gentlemen who were both retired from the newspaper business discussing how people don’t really take the time to read a newspaper like they used to. They were lamenting that people today preferred quick sound bites of information – whether it be from television or from reading one of the various news sites on the Internet.

]]> This discussion struck a chord with me, as I had just wrapped up a discussion with a client where their means to build a business case for data quality was going to be driven by their ability to compile the results of their analysis into a quick snapshot or dashboard for key stakeholders to make a decision. Thus, just like the comment about people wanting “just the facts” when they read the news, the same holds true for data quality. Although some people still want to read the entire paper from front to back, the vast majority might choose to start with the headlines and only read an entire story if it appears to be of interest. In the case of data quality, a dashboard provides you with the overall state of data quality in your organization, while providing the capability to drill down as needed. Thus, just like in a newspaper article where the headline might influence whether someone decides to read the entire article or not, the types of issues captured in a dashboard will likely influence whether you are in tune with the types of key data quality in your organization, or if this is the type of “headline” that doesn’t entice the reader (or key stakeholders) to want to learn more. How successful you are in garnering attention to data quality issues through the dashboard results will directly impact your ability to articulate the need to pursue a data quality initiative.

Until next time…

Better management through measuring data quality

2007-12-20T11:57:29Z

I recently asked a customer of ours why they invested so much in monitoring and publishing key performance indicators for their data quality. “Believe it or not, the biggest reason we measure data quality is not to correct bad data” came the reply. “The reason we monitor data quality is to detect problems with our business processes.”

Indeed, as I mentioned in my last blog post, business users look to investments in people and processes in addition to technology in order to address poor data quality. For example, if a bank branch manager received a report showing that customer data originating from his branch office had much higher incidents of duplicate entries and was putting the entire bank at risk of massive regulatory fines, he is not going to throw technology at the problem. His response might be mandatory training for tellers or better hiring practices to screen for adequate computer skills.

Experts in quality control methodology refer to this as addressing “root cause.” Common starting points of measurement involve completeness, accuracy, consistency, conformity, duplication, and integrity. Eventually, as the business culture matures its data quality practices, timeliness and data lineage (origination) are used to evaluate quality of data. Of course, software technology that automates the process of parsing, standardizing, matching and consolidating data is of immense value and is an absolute requirement in any data integration project. However, the issue of data quality goes beyond these IT projects. Ongoing measurement and monitoring of data quality provides value directly to the business because it helps them to better manage their people and processes.

The self-service data quality minefield

2007-12-14T12:08:33Z

Marshall Field, the American department-store owner and retail merchandizing pioneer, is usually credited with coming up with the now much abused adage “Right or wrong, the customer is always right”. These days the saying is usually shortened to the more functional “the customer is always right”. But with self-service data entry, where more and more customers are responsible for supplying their own contact and order details via web applications, it is worth revisiting the entire quote.

The fact is even when the customer is responsible for “bad data” the vendor still has to shoulder the blame and do something about it.

Take the example of a friend of mine who recently signed up with a well known social networking website. Let’s ignore the reasons why he signed up – he’s not entirely sure – or what social networking sites are for – none of us is entirely sure yet, but I guess we’ll start to get the point as more and more of them are sold to industry giants for n billion dollars.

]]> My friend (let’s call him Paul) definitely signed up once, probably twice and possibly three times – so now we have a classic data quality problem, duplicate customer entries. When I find one of Paul’s entries on the site, I send him a message to say hello – after all even though I have his phone number and could just call him, this is the new era of Internet social networking and I don’t want to be left out. He gets an email to tell him he has a message and logs on to one of his multiple personalities to see who it’s from. But because the account he logs on to is not the one I found he doesn’t see my message. He leaves feeling disappointed and under whelmed by the whole social networking experience. At the same time because I don’t get a reply to my cheery greeting I feel neglected and don’t invite Paul and his wife to dinner the following weekend!

Luckily because we live in a small enough town, we bump into each other in a real life social situation (i.e. the local pub), make up and all live happily ever after.

Despite the happy ending, there is a lesson here that our old friend Marshall Field would not miss, and that is despite the fact Paul is a smart guy who knows that ultimately he messed up a bit with his data entry procedures, he still blames the social networking site to some degree for letting it happen.

This was just something Paul was trying out in his spare time (not on work time I promise boss). But the same sort of scenario happens more and more as people are faced with self-service Internet applications for banking, paying bills, handling their telephone accounts and much more. The problem for the bank, insurance company, government agency or utility company is that their customers will still expect them to get their data right, even when it’s the customer who made the mistake in the first place.

In fact having stringent automated data quality processes and procedures in place at the point of entry becomes even more important in the self-service age. Most organizations will know that even when they have say a couple of hundred trained call center staff handling front line data entry, they still end up with multiple data quality problems. What is it going to be like when thousands of customers, none of them trained for the job, take over the task?

Not to worry all you have to remember is, right or wrong the customer is always right!

Start small with monitoring, but always think big to achieve data quality goals

2007-12-06T15:50:31Z

I attended my first parent-teacher meeting the other day for my five-year old daughter. Another one of those “life stage” events done and dusted – I remember dreading the annual meeting when I was a kid. The notion of my parents and my teacher comparing notes on my behaviour was too much to bear – somebody was eventually going to put two and two together and find out I was up to no good.

It all got me thinking about a recent blog post by my esteemed colleague Garry Moroney. His post Mobilizing the Data Quality Army outlined the level of effort, thought and planning that the US Department of Education is putting into data quality.

As Garry points out dealing with data quality in a large, disconnected organization such as the US schools system is not a trivial exercise. But if you were to only read that one post you might be overwhelmed by the potential size of the data quality task in front of you.

]]> The good news is you don’t have to do it all in one go. At the same time tactical solutions aren’t the answer either. Data quality processes can be phased in over time, but it is important to start with a holistic view and solid process-oriented approach that can be reused and ultimately deployed across the entire organization.

Data quality improvement is not just about fixing data. First you need to understand the true level of data quality within your organization, find ways to clean up the data already in use and then stop low quality data from getting into systems in the future.

Establishing a baseline of your current state of data quality; so that one can identify the critical failure points and determine improvement targets should be your starting point. But, achieving the high levels of data quality needed to improve business efficiency and transparency is an iterative process that needs to be tracked, managed, and monitored. Therefore, being able to measure and monitor data quality throughout the data quality lifecycle and compare the results over time is an essential ingredient in the proactive management of ongoing data quality improvement and data governance. So a bit of effort put into planning for measurement and monitoring at the start will pay dividends for a long time to come, and enable you to keep an eye on the big picture no matter how small you start.

Business and IT Collaboration is Essential for Data Quality

2007-12-05T08:05:42Z

A recent InformationWeek article* described the growth in IT employment across the US as a result of a shift in skills. Rather than focusing on pure IT proficiency, organizations are looking for talent with “a more hybrid mix of technology skills, along with an understanding of the business and its customers.”

IT departments are highly motivated to increase the level of collaboration with their counterparts in the business. Nowhere is this more critical than in the area of data quality and the trend is causing a shift in the way companies are looking to solve their data quality issues. First generation data quality tools had a natural focus on technology, instead of business. Here are some of the differences between technology focused data quality solutions and business-focused data quality solutions.

Tools vs. Process
Technology focused data quality solutions provide tools that automate data processing. Evidence of this type of focus can be seen in the way that vendors will tout the sophistication and type of their algorithms over and above their ability to support ongoing data quality management processes. While technology is extremely important, its relevance cannot eclipse the overall data quality management process. Even if your data quality tool can automate the correction of 95 percent of the data, if the remaining five percent cannot be managed properly, you will continue to suffer from poor data quality.

]]> Physical Data Attributes vs. Semantics
Once data is separated from its original host application it has very little context. Therefore, it is natural that technology focused data quality offerings center on evaluating the physical data structures instead of evaluating consistency and accuracy of the meaning attached to data. There are exceptions in areas where common reference data is applicable (e.g. address validation). However, for much of the data within the enterprise, there is not enough attention being paid to the data quality of information in context. A field containing the number 10044 out of context is not much good to us from a data quality standpoint. A business analyst would ask – “10044 what?” 10044 pounds? 10044 dollars? 10044 liters? Is 10044 a specific code relating to a product attribute? Without the context data quality is meaningless.

See also the interesting post from my colleague Chris McCauley Data Quality Metadata; a lot more than just "data about data" for more on this important subject.

Fixes applied in Technology vs. Fixes applied in People, Processes and Technology
Technology focused data quality offerings limit the type of data quality improvements to fixing data using developer tools. Properly addressing data quality requires more than just fixing defective data; it also requires improving the processes that lead to poor quality data entering the system in the first place. Bridging the gap between technology and business is essential if this is to be achieved. As data quality offerings evolve to have more of a business focus, they offer user interfaces designed with the business professional or knowledge worker in mind and they allow for the type of Business-IT collaboration that was highlighted by InformationWeek. Business users have a very different outlook on data quality. They are just as likely to invest in training, hiring, and business process reengineering to improve data quality as they are to invest in additional technology. The resulting effect is the data quality management process, information in context and data quality metrics get increased attention and investment.

* You can read the InformationWeek article at In Growing Job Market, IT Pros Get More For The Soft Skills

Building the Business Case for Data Quality

2007-11-28T22:14:40Z

As a new contributor to the data quality blog site, I wanted to start by introducing myself and highlighting the types of topics I plan to discuss on a semi-frequent basis. I am a Principal Consultant with Informatica Professional Services and have spent the past 6 years in the data quality space in a variety of sales and post sales roles. During this time I have seen the data quality market continue to evolve and mature. Thus, I would like to use this column to reflect on the types of use cases I have seen and continue to see when meeting with organization’s faced with data quality problems. I hope these posts can start an active dialogue, regardless if your company is trying to tackle their first data quality initiative or looking to build out a formal center of excellence around data quality.

To start, I wanted to pose a common question I am often asked by clients and prospects – how do I build a business case for data quality? Although an organization may think (or even know) there is a problem, the need to justify the cost around procuring a data quality solution often exists. This justification requirements often comes from the idea that data quality issues aren’t necessarily a core business issue (how wrong this is!) or something that can be handled through manual intervention (this is true – if you have unlimited time and money, but even then your results will be limited). Thus, the following points are meant to help start an organization down the path to building the internal business case through a Data Quality Audit. Note - if you have access to Informatica’s Velocity Methodology, I go into these steps in further detail in the best practice document, “Developing the Data Quality Business Case.”

]]>
1. Identify a test source - A manageable, yet representative, sample set of data should be evaluated. This can be a cross-section of an enterprise data set or data from a specific department that a potential data quality issue is expected to be found.

2. Identify issues - Any anticipated issues with the data should be identified prior to conducting the Audit, in order to ensure the known use cases are investigated.

3. Define scope - The scope of the Audit should be defined in order to ensure that a business case can be made for a data quality initiative within weeks, not months. The project should be seen as a pilot in order to validate the anticipated ROI if an enterprise initiative is pursued. Just as the scope should be well defined, commitments should be agreed upon prior to starting the project that the required resources (such as the data steward, IT representative, business user) will be available as needed during the duration of the project in order to ensure the activities such as data and business rule review remain on schedule.

4. Highlight resulting issues - Upon the conclusion of the Audit, the issues uncovered during the project should be summarized and presented to key stakeholders in a workshop setting. During the workshop, the results should be highlighted, as well as any anticipated impact to the business if a data quality initiative is not enacted within the organization.

5. Build knowledge - As previously stated, the intent of the Audit is to quantify the anticipated ROI within an organization if a data quality strategy is implemented. In addition, the knowledge about the data, the business rules and the potential strategy that can be leveraged throughout the entire organization should be captured.

The intent of these five steps is to ensure the initial business case can be quantified in a relatively short period of time without a significant investment being incurred by the organization. In addition, the information gathered during the activity will be the starting point for the actual data quality initiative.

In future posts I will discuss where to go after you have quantified the problem. You now have the knowledge and tools available, but how do you start attacking the 800-pound data quality gorilla in the room?

Until next time…

Hospital Billing Errors 'Kill' Patients

2007-08-29T21:45:21Z

Hospital billing mistakes have become so prevalent that a niche industry has evolved to help patients decipher their bills and help correct the errors. ”Pat Palmer, founder of Medical Billing Advocates of America, estimates that she finds multiple errors in 8 out of every 10 hospital bills she reviews” (Dina ElBoghdady, “Killer Billing Errors,” Washington Post, June 27, 2004, p. F01. Accessed Aug 9, 2007 at: http://www.washingtonpost.com/wp-dyn/articles/A7351-2004Jun26.html ).

One patient saw the bill of $25,652.14 for her 2-hour routine operation, reduced to a cost of $17,000 billed to the insurance company and her cost reduced to $2,148, after correcting errors and overcharges.
While this error-rate was not statistically measured, nor would it apply to all hospital bills, it points out a huge problem in Information Quality in the health care system, not counting actual medical errors caused by information defects.

The consequence of health care over billing is significant. It contributes to the high costs of health care, which is one major cause of personal bankruptcy.

]]> The causes of hospital billing errors are many:
• The complexity of diagnosis codes and health care procedure codes is complex and time consuming
• The practice of billing for every detailed item is very error prone. Items are created for the use of each instrument, sponge, and medication component are difficult to document accurately
• Keying errors in data entry can be quite common
• The ultimate root cause is the desire to increase revenue in health care services leading to poor controls in the billing process

From a consumer standpoint, my recommendations are:
• Always get an itemized hospital or healthcare service bill
• Review it as quickly as possible while the service experience is fresh in your mind
• Do not hesitate to ask the health care provider to go over the bill to explain any apparent anomalies
• Work with your insurer to assure the medical procedure costs are appropriate from their perspective.

From your perspective as an Information Quality practitioner, ask yourself how this anecdote of one kind of IQ problem might apply in your organization:
• Where might your organization have broken information processes?
• In what way might those broken processes negatively affect your customers?
• What are the root causes of poor quality information in those processes?
• What can you do to improve the processes to prevent information defects that cost your organization in process failure and “information scrap and rework” and in potential harm or dissatisfaction among your customers?

What do you think? Share your thoughts on this blog to help bring in the Information Quality Revolution!!!

Information Quality & Management Transformation

2007-06-28T15:59:33Z

I recently received an email from one of my early clients. After having worked in four different companies in four different industries, she came to a sad conclusion, writing:

“The thing that they all have in common is a desire to cut corners and deal with quality later. It takes a lot of energy to be the information quality cheerleader, and I find it discouraging and overwhelming at times. Keep writing your articles and books to encourage all the people like me who are dealing with these issues every day.” P. G.

The discovery that P. G. has experienced is, unfortunately, the norm—not the exception. There are two critical elements in this experience.

]]> The first is embedded in W. Edwards Deming’s Second Point of (his 14 Points of Quality), “Adopt the New Philosophy: ‘Reliable product and service reduces costs.’” However, Dr. Deming said Point 2 “really means a transformation of [American style] management” (The Deming Management Method, p. 59.) “We can no longer tolerate commonly accepted levels of mistakes, defects, material not suited for the job….” (Out of the Crisis, p. 26.)

Mistakes, errors, untimely information all increase costs of doing business. Management that does not go to Gemba may insulate it from knowing how wasteful their processes are when defective information causes processes to fail, incurring costs of failure itself, the cost to recover from the failure and data correction (data cleansing) required.

The second part of this is that IQ professionals must not just complain that management does not listen to them and change their “defective” management practices. We must be more than “cheerleaders;” we must be missionaries, calling for a conversion from the practices of managing based upon time to delivery and costs alone, that invariably reduces quality while increasing costs of failure and information scrap and rework.

Kaoru Ishikawa, one of the eminent Japanese quality gurus used to say, if we are going to solve problems, we must, “Speak with Data.” We must remember that management needs data that illuminates a problem, so they can make the right decision. IQ professionals, need to understand their manager’s need for information and provide that data in a way that is compelling and true.

Most managers believe that their processes are working properly because our organization is making a profit. If they have never measured the costs caused by defective information, they feel there is no need to change.
So, the most important quality management process to address this management situation is TIQM® (Total Information Quality Management) Process, P3, “Measure the Cost of Poor Quality Information” (Improving Data Warehouse and Information Quality, pp. 199-235.)

Measuring and quantifying the costs of poor quality information can be challenging and professionals in my field, including myself, provide training in these areas.

Go find a business manager who is feeling some pain in their business area caused by poor quality information. They will be delighted if you can help them measure the costs of poor quality and then improve the upstream processes to eliminate the root cause, so they can eliminate the wasteful information scrap and rework activities and become more productive.

Management has basically three choices in the emerging “realized” Information Age. They can:
1. Maintain the status quo, maintaining the current “cost-of-failure” and information scrap and rework that creates competitive disadvantage and risk of business failure
2. Play at Data Quality, implementing “data cleansing” or “inspect and correct” that only attacks the symptoms, again leading to competitive disadvantage and the risk of business failure
3. Lead the transformation, recognizing that proactive, Information Quality Management with process improvement to eliminate root causes and with it the defective information that drives up business costs. This leads to competitive Advantage with increased profits, increased end-customer satisfaction and internal knowledge worker satisfaction as people no longer have to waste time in information scrap and rework and can concentrate on performing their value work more effectively and on innovation for tomorrow’s products and services.

What do you think? Share your thoughts on this blog to help bring in the Information Quality Revolution!!!

Alice in “Qualityland"

2007-06-22T14:20:47Z

Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to
Alice: I don't much care where.
The Cheshire Cat: Then it doesn't much matter which way you go
– Lewis Carroll, Alice's Adventures in Wonderland

When confronted with the problem of how to address their data quality issues many organisations are faced with a similar dilemma to that which confronted Alice during her travels in Wonderland; “I know that I need to do something, but I don’t know where to start”. Knowing where to start and, equally importantly, the size of the problem as well as where an organisation needs to go are critical factors in ensuring that their data quality journey takes them where they need to be at the price they are prepared to pay.

When planning their “journey” organisations need to address the issue of data quality holistically by considering each of the three DQ pillars in turn; firstly “People”, then “Ideas” and finally “Technology”. Many DQ initiatives have failed as the primary focus has been on delivering a technical solution. However without the right framework in place and operated by the right people this approach will never deliver the results that organisations need. Time and time again within the IT industry it has been proved that the pure application of technology will never solve business issues, as technology in itself will never win the “war”, it is always the right people with the right ideas who use the technology in the right way.

]]> The “People” pillar represents the data quality function within an organisation and will typically involve, amongst other activities, the identification of data owners, the appointment of data stewards and the establishment of appropriate data governance on a scale that is appropriate to an organisation. A successful “People” pillar will encompass both the empowerment of the right people in the right places as well as address the cultural shift required to accommodate the concepts of data ownership and accountability; i.e. moving the business perception of the role that IT plays, from one of owner to that of custodian while communicating a clear vision on where it wants to be.

Once the “People” are in place then appropriate “ideas” can be developed. By “ideas” we mean the data quality initiative, program or strategy that encapsulates the key quality principles that are relevant to a particular organisation, i.e. their critical success factors, as well as defining how they will implemented and by whom. The “People” component cannot operate in a vacuum and therefore it is crucial that they function inside of a defined framework which is understood by them and conversely, can be easily communicated to others.

A critical step in this process is defining exactly what constitutes data quality for an organisation for as John Guaspari's discusses in his book, “The Customer Connection”, quality is not just the absence of defects as determined by the producer but also the presence of value as determined by customers. Integral to this process is the identification of the key metrics that will be used to underpin this definition and thus enable objective rather than subjective measurement of the problem.

However whatever the final outcome of this exercise it must be recognised that not every problem in every instance can or for that matter should be addressed. The “People” driving the process should review the work being performed from a perspective of both appropriate risk mitigation and overall pragmatism.

Once the first two pillars are in place then the appropriate “technology” can be identified which can both deliver the “Ideas” and be simply and cost-effectively used by the “People” to enable, and not drive, the DQ process. When reviewing which particular “technology” an organisation should adopt it is important to consider the following:
• Its capacity to deliver the generic functionality of contact efficiency and relationship identification but also it’s extensibility to perform across all business data types and it’s ability to deliver sophisticated data analysis
• Its ability to empower business users so the focus of the work can be moved from the IT custodians to the data owners with its related shift of accountability and cultural mindset
• Its ability to be deployed across the full spectrum of business data and not be constrained to only name and address data, as are many of today’s products.
• Its ability to support vertical solutions which address the specific requirements of many industries be it Sarbanes-Oxley reporting for Financial Services, Global Data Synchronisation for CPG or Product Cataloguing for Pharmaceuticals
• Its support of open data repositories and “out-of-the-box” functionality that can be quickly deployed and integrated easily into an existing IT environment

Finally, central to an organisation’s successful adoption of data quality principles is the ability to communicate to all stakeholders not only what work is being performed but also how it is progressing be it a discrete vertical solution such as Basel II compliance project or a horizontal data integration project, e.g. data migration . Increasingly the use of benchmarking and score carding is being adopted to achieve this and organisations should take note of a product’s capability to generate such reports and how easily they can be integrated into their normal business intelligence portfolio.

Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to
Alice: I don't much care where.
The Cheshire Cat: Then it doesn't much matter which way you go
Alice: `--so long as I get SOMEWHERE,' Alice added as an explanation.