Tag Archives: Data Quality
Nine years ago when I started in the data integration and quality space, data quality was all about algorithms and cleansing technology. Data went in, and the “best” solution was the one that could do the best job of fuzzy matching the data and cleaning more data than the other products. Of course, not data quality solution could clean 100% of the data so “exceptions” were dumped into a file that were left as an “exercise for the user” to deal with on their own. This usually meant using the data management product of choice, when there is nothing else…. Data goes into a spreadsheet, and then users would remediate the mistakes by hand in the spreadsheet. Then someone would write an SQL query to write the corrections back into the database. In the end, managing the exceptions was a very manual process with very little to no governance to the process.
The problem with this of course is that for very many companies, data stewardship is not the person’s day job. So if they have to spend time checking to see if someone else has corrected an error in the data, or getting approval to make a data change, or spending time then consolidating all the manual changes they made and then communicating those changes to management, then they don’t have much time left to sleep, much less eat. In the end, but business of creating quality data just doesn’t get done, or doesn’t get done well. In the end, data quality is a business issue, supported by IT, but the business facing part of the solution has been missing.
But that is about to change. Informatica already provides the most scalable data quality product for handling the automated portion of the data quality process. And now, in the latest release of Informatica Data Quality 9.6, we have created a new edition called the Data Quality Governance Edition to fully manage the exception process. This edition provides a completely governed process for managing remediation of data exceptions by business data stewards. It allows organizations to create their own customized process with different levels of review. Additionally, it makes it possible for business users to create their own data quality rules, describing the rules in plain language…. no coding necessary.
And of course, every organization wants to be able to track how they are improving. And Informatica Data Quality 9.6 includes embeddable dashboards that show the progress of how data quality is improving and impacting the business in a positive way.
Great data isn’t an accident. Great data happens by design. And for the first time, data cleansing has been combined with a holistic data stewardship process, allowing business and IT to collaborate to create quality data that supports critical business processes.
Over the last 40 years, data has become increasingly distributed. It used to all sit on storage connected to a mainframe. It used to be that the application of computing power to solve business problems was limited by the availability of CPU, memory, network and disk. Those limitations are no longer big inhibitors. Data fragmentation is now the new inhibitor to business agility. Data is now generated from distributed data sources not just within a corporation, but from business partners, from device sensors and from consumers Facebook-ing and tweeting away on the internet.
So to solve any interesting business problem in today’s fragmented data world, you now have to pull data together from a wide variety of data sources. That means business agility 100% depends on data integration agility. But how do you do deliver that agility in a way that is not just fast, but reliable, and delivers high quality data?
First, to achieve data integration agility, you need to move from a traditional waterfall development process to an agile development process.
Second, if you need reliability, you have to think about how you start treating your data integration process as a critical business process. That means thinking about how you will make your integration processes highly available. It also means you need to monitor and validate your operational data integration processes on an ongoing basis. The good news is that the capabilities you need for data validation as well as operational monitoring and alerting for your data integration process are now built into Informatica’s newest PowerCenter Edition, PowerCenter Premium Edition.
Lastly, the days where you can just move data from A to B without including a data quality process are over. Great data doesn’t happen by accident, it happens by design. And that means you also have to build in data quality directly into your data integration process.
Great businesses depend on great data. And great data means data that is delivered on time, with confidence and with high quality. So think about how your understanding of data integration and great data can make your career. Great businesses depend on great data and people like you who have the skills to make a difference. As a data professional, the time has never been better for you to make a contribution to the greatness of your organization. You have the opportunity to make a difference and have an impact because your skills and your understanding of data integration has never been more critical.
I love exploring new places. I’ve had exceptional experiences at the W in Hong Kong, El Dorado Royale in the Riviera Maya and Ventana Inn in Big Sur. I belong to almost every loyalty program under the sun, but not all hospitality companies are capitalizing on the potential of my customer information. Imagine if employees had access to it so they could personalize their interactions with me and send me marketing offers that appeal to my interests.
Do I have high expectations? Yes. But so do many travelers. This puts pressure on marketing and sales executives who want to compete to win. According to Deloitte’s report, “Hospitality 2015: Game changers or spectators?,” hospitality companies need to adapt to meet consumers’ increasing expectations to know their preferences and tastes and to customize packages that suit individual needs.
In this interview, Jeff Klagenberg, senior principal at Myers-Holum, explains how one of the largest, most customer-focused companies in the hospitality industry is investing in better customer, product, and asset information. Why? To personalize customer interactions, bundle appealing promotion packages and personalize marketing offers across channels.
Q: What are the company’s goals?
A: The executive team at one of the world’s leading providers of family travel and leisure experiences is focused on achieving excellence in quality and guest services. They generate revenues from the sales of room nights at hotels, food and beverages, merchandise, admissions and vacation club properties. The executive team believes their future success depends on stronger execution based on better measurement and a better understanding of customers.
Q: What role does customer, product and asset information play in achieving these goals?
A: Without the highest quality business-critical data, how can employees continually improve customer interactions? How can they bundle appealing promotional packages or personalize marketing offers? How can they accurately measure the impact of sales and marketing efforts? The team recognized the powerful role of high quality information in their pursuit of excellence.
Q: What are they doing to improve the quality of this business-critical information?
A: To get the most value out of their data and deliver the highest quality information to business and analytical applications, they knew they needed to invest in an integrated information management infrastructure to support their data governance process. Now they use the Informatica Total Customer Relationship Solution, which combines data integration, data quality, and master data management (MDM). It pulls together fragmented customer information, product information, and asset information scattered across hundreds of applications in their global operations into one central, trusted location where it can be managed and shared with analytical and operational applications on an ongoing basis.
Q: How will this impact marketing and sales?
A: With clean, consistent and connected customer information, product information, and asset information in the company’s applications, they are optimizing marketing, sales and customer service processes. They get limitless insights into who their customers are and their valuable relationships, including households, corporate hierarchies and influencer networks. They see which products and services customers have purchased in the past, their preferences and tastes. High quality information enables the marketing and sales team to personalize customer interactions across touch points, bundle appealing promotional packages, and personalize marketing offers across channels. They have a better understanding of which marketing, advertising and promotional programs work and which don’t.
Q: What is the role did the marketing and sales leaders play in this initiative?
A: The marketing leaders and sales leaders played a key role in getting this initiative off the ground. With an integrated information management infrastructure in place, they’ll benefit from better integration between business-critical master data about customers, products and assets and transaction data.
Q. How will this help them gain customer insights from “Big Data”?
A. We helped the business leaders understand that getting customer insights from “Big Data” such as weblogs, call logs, social and mobile data requires a strong backbone of integrated business-critical data. By investing in a data-centric approach, they future-proofed their business. They are ready to incorporate any type of data they will want to analyze, such as interaction data. A key realization was there is no such thing as “Small Data.” The future is about getting very bit of understanding out of every data source.
Q: What advice do you have for hospitality industry executives?
A: Ask yourself, “Which of our strategic initiatives can be achieved with inaccurate, inconsistent and disconnected information?” Most executives know that the business-critical data in their applications, used by employees across the globe, is not the highest quality. But they are shocked to learn how much this is costing the company. My advice is talk to IT about the current state of your customer, product and asset information. Find out if it is holding you back from achieving your strategic initiatives.
Also, many business executives are excited about the prospect of analyzing “Big Data” to gain revenue-generating insights about customers. But the business-critical data about customers, products and assets is often in terrible shape. To use an analogy: look at a wheat field and imagine the bread it will yield. But don’t forget if you don’t separate the grain from the chaff you’ll be disappointed with the outcome. If you are working on a Big Data initiative, don’t forget to invest in the integrated information management infrastructure required to give you the clean, consistent and connected information you need to achieve great things.
Do you know what year the first steam engine locomotive was invented? 1804. It traveled 9 miles in two hours. Now, you and I would be pretty upset of we boarded a train and it took 2 hours to go 9 miles. But, 200 years ago, this was a huge innovation and led to the invention of the modern day train and railway.
Tremendous Growth In Demand for Rail Travel Puts Pressure on Rail Infrastructure
Today, Britain is experiencing tremendous growth in demand for rail travel. One million more trains and 500 million more passengers travel by train than just 5 years ago. Over the next 30 years passenger demand for rail will more than double and freight demand is expected to go up by 140%. This puts tremendous pressure on the rail infrastructure.
Network Rail is in the modern-day rail business. Employees work day and night running, maintaining and updating Britain’s rail infrastructure, including millions of assets, such as 22,000 miles of track, 6,500 crossings, 43,000 bridges, viaducts and tunnels. Improving the rail network provides faster, more frequent and more reliable journeys between Britain’s towns and cities.
Network Rail is investing more in the rail infrastructure than in Victorian times. In the last six months, they spent about $25 million a day! In a recent news release, Patrick Bucher, group finance director said, “We continue to invest record amounts to deliver a bigger, better railway for passengers and businesses across Britain. We are also driving down the cost of running Britain’s railway to help make it more affordable in the years ahead.”
Employees Need to Trust Asset Information to Pinpoint and Fix Problems Quickly
To pinpoint and fix problems quickly, keep their operating costs low and maintain a strong safety record, Network Rail’s employees need to trust their mission-critical asset information, such as:
- What is the problem?
- Where is it?
- What equipment, tools and skills are needed to fix it?
- Who is closest to the problem that could fix it?
Difficult to Make Sense of Asset Information Scattered across Applications
Similar to many companies their size, Network Rail’s mission-critical asset information was scattered across many applications, which made it difficult for employees to make sense of asset information and the interaction between assets.
The asset information team recognized the limitations of employees depending on an application-centric view of their business. To operate more efficiently and effectively, they needed clean asset information, consistent asset information, and connected asset information.
Investing in Rail Infrastructure AND the Information Infrastructure to Support It
Network Rail now uses a combination of data integration, data quality, and master data management (MDM) to manage their mission-critical asset information in a central location on an ongoing basis, to:
- make sense of asset information,
- understand the relationships between assets, and
- track changes to asset information.
In a news release, Patrick Bossert Director of Network Rail’s Asset Information services business said, “With more accurate and reliable information about assets and their condition our team can make better business decisions, enable innovation in our asset management policy, planning and execution, and improve rail-system-wide investment decisions that benefit the rail industry as a whole.”
If you work for a company that revolves around mission-critical asset information, ask yourself these questions:
- Can our employees makes sense of our asset information?
- Can they easily see relationships between assets and how they interact?
- Can they see the history of changes to asset information over time?
Or are are they limited by an application-centric view of the business because asset information is scattered across in multiple systems?
Have a similar story about how you are managing your mission-critical asset information? Please share it in the comments below.
The Physician Payments Sunshine Act shines a spotlight on the disorganized state of physician information, which is scattered across systems, often incomplete, inaccurate and inconsistent in most pharmaceutical and medical device manufacturing companies.
According to the recent Wall Street Journal article Doctors Face New Scrutiny over Gifts, “Drug companies collectively pay hundreds of millions of dollars in fees and gifts to doctors every year. In 2012, Pfizer Inc., the biggest drug maker by sales, paid $173.2 million to U.S. health-care professionals.”
The Risks of Creating Reports with Inaccurate Physician Information
There are serious risks of filing inaccurate reports. Just imagine dealing with:
- An angry call from a physician who received a $25 meal, which was inaccurately reported as $250 or who reportedly, received a gift that actually went to someone with a similar name.
- Hefty fines and increased scrutiny from the Centers for Medicare and Medicaid Services (CMS). Fines range from $1,000 to $10,000 for each transaction with a maximum penalty of maximum $1.15 million.
- Negative media attention. Reports will be available for anyone to access on a publicly accessible website.
How prepared are manufacturers to track and report physician payment information?
One of the major obstacles is getting a complete picture of the total payments made to one physician. Manufacturers need to know if Dr. Sriram Mennon and Dr. Sri Menon are one and the same.
On top of that, they need to understand the complicated connections between Dr. Sriram Menon, sales representatives’ expense report spreadsheets (T&E), marketing and R&D expenses, event data, and accounts payable data.
3 Steps to Ensure Physician Information is Accurate
In recent years, some pharmaceutical manufacturers and medical device manufacturers were required to respond to “Sunshine Act” type laws in states like California and Massachusetts. To simplify, automate and ensure physician payment reports are filed correctly and on time, they use an Aggregate Spend Repository or Physician Spend Management solution.
They also use these solutions to proactively track and review physician payments on a regular basis to ensure mandated thresholds are met before reports are due. Aggregate Spend Repository and Physician Spend Management solutions rely on a foundation of data integration, data quality, and master data management (MDM) software to better manage physician information.
For those manufacturers who want to avoid the risk of losing valuable physician relationships, paying hefty fines, and receiving scrutiny from CMS and negative media attention, here are three steps to ensure accurate physician information:
- Bring all your scattered physician information, including identifiers, addresses and specialties into a central place to fix incorrect, missing or inconsistent information and uniquely identify each physician.
- Identify connections between physicians and the hospitals and clinics where they work to help aggregate accurate payment information for each physician.
- Standardize transaction information so it’s easy to identify the purpose of payments and related products and link transaction information to physician information.
Physicians Will Review Reports for Accuracy in January 2014
In January 2014, after physicians review the federally mandated financial disclosures, they may question the accuracy of reported payments. Within two months manufacturers will need to fix any discrepancies and file their Sunshine Act reports, which will become part of a permanent archive. Time is precious for those companies who haven’t built an Aggregate Spend Repository or Physician Spend Management solution to drive their Sunshine Act compliance reports.
If you work for one of the pharmaceutical or medical device manufacturing companies already using an Aggregate Spend Repository or Physician Spend Management solution, please share your tips and tricks with others who are behind.
Tick tock, tick tock….
Last week I described how Informatica Identity Resolution (IIR) can be used to match data from different lists or databases even when the data includes typos, translation mistakes, transcription errors, invalid abbreviations, and other errors. IIR has a wide range of use cases. Here are a few. (more…)
Just in time for Halloween, I’m sharing a scary story. Warning: this is a true story. You may wonder:
- Could this happen to me?
- Can this situation be avoided?
- How can I prevent this from happening to me?
Last summer, the worst wildfire in Colorado history burned hundreds of acres, 360 homes, killing two people and forcing 38,000 people to evacuate the area.
Unfortunately, it was during the Colorado wildfire that a large integrated healthcare provider with hospitals, doctors, healthcare providers and employees located throughout the United States (who shall remain nameless) realized they had a problem. They couldn’t respond in real time to the disaster by mobilizing their workforce quickly. They struggled to identify, contact and communicate with doctors, healthcare providers and employees located at the disaster area to warn them not to go to the hospital or redirect them to alternative sites where they could help.
This healthcare provider’s inability to respond to this disaster in real time was an “Aha” moment. What was holding them back was a major information problem. Because their employee information was scattered across hundreds of systems, they couldn’t pull a single, comprehensive and accurate list of doctors, healthcare providers and employees in the disaster area. They didn’t know which employees needed to be evacuated or which could be sent to assist people in other locations. So, they had to email everyone in the company.
The good news is that we’re in the process of helping them create and maintain a central location called an “employee master” built on our data integration, data quality, and master data management (MDM) software. This will be their “go-to” place for an up-to-date, complete and accurate list of employees and their contact information, such as work email, phone, pager (doctors still use them), home phone and personal email as well as their location, so they know exactly who is working where and how best to contact them.
This healthcare provider will no longer be held back by an information problem. In three months, they’ll be able to respond to disasters in real time by mobilizing their workforce quickly.
An interesting side note: Immediately before our Informatica team of experts arrived to talk to this healthcare provider about how we can help them, there was a power outage in the building. They struggled to alert the employees who were impacted. So our team personally experienced the pain of this organization’s employee information problem.
When disaster strikes, will you be ready to respond in real time? Or do you have an information problem that could hold you back from mobilizing your own employees?
I want your opinion. Are you interested in more scary stories? Let me know in the comments below. I’m thinking about making this a regular series.
When I talk to customers about dealing with poor data quality, I consistently hear something like, “We know we have data quality problems, but we can’t get the business to help take ownership and do something about it.” I think that this is taking the easy way out. Throwing your hands up in the air doesn’t make change happen – it only prolongs the pain. If you want to affect a positive change in data quality and are looking for ways to engage the business, then you should join Barbara Latulippe, Director of Enterprise Information Management for EMC and and Kristen Kokie, VP IT Enterprise Strategic Services for Informatica for our webinar on Thursday October 24th to hear how they have dealt with data quality in their combined 40+ years in IT.
Now, understandably, tackling data quality problems is no small undertaking, and it isn’t easy. In many instances, the reason why organizations choose to do nothing about data quality is that bad data has been present for so long that manual work around efforts have become ingrained in the business processes for consuming data. In these cases, changing the way people do things becomes the largest obstacle to dealing with the root cause of the issues. But that is also where you will be able to find the costs associated with bad data: lost productivity, ineffective decision making, missed opportunities, etc..
As discussed in this previous webinar,(link to replay on the bottom of the page), successfully dealing with poor data quality takes initiative, and it takes communication. IT Departments are the engineers of the business: they are the ones who understand process and workflows; they are the ones who build the integration paths between the applications and systems. Even if they don’t own the data, they do end up owning the data driven business processes that consume data. As such, IT is uniquely positioned to provide customized suggestions based off of the insight from multiple previous interactions with the data.
Bring facts to the table when talking to the business. As those who directly interact daily with data, IT is in position to measure and monitor data quality, to identify key data quality metrics; data quality scorecards and dashboards can shine a light on bad data and directly relate it to the business via the downstream workflows and business processes. Armed with hard facts about impact on specific business processes, a Business user has an easier time affixing a dollar value on the impact of that bad data. Here’s some helpful resources where you can start to build your case for improved data quality. With these tools and insight, IT can start to affect change.
Data is becoming the lifeblood of organizations and IT organizations have a huge opportunity to get closer to the business by really knowing the data of the business. While data quality invariably involves technological intervention, it is more so a process and change management issue that ends up being critical to success. The easier it is to tie bad data to specific business processes, the more constructive the conversation can be with the Business.
Even in “good” data there is a lot of garbage. For example a person’s name. John could also be spelled as Jon or Von (I have a high school sports trophy to prove it). Schmidt could become Schmitt or Smith. In Hungarian my name is Janos Kovacs. Human beings entering data make errors in spelling, phonetics, and keypunching. We also have to deal with variations associated with compound and account names, abbreviations, nicknames, prefix & suffix variations, foreign names, and missing elements. As long as humans are involved in entering data there will be a significant amount of garbage in any database. So how do we turn this gibberish into gems of information?
Do you know how good your multichannel data is? This blog covers four business objectives when accelerating multi channel commerce and which quality of product data is needed to deliver to that and a summary of questions to ask when establishing your strategy. These questions help ecommerce managers, category managers and marketers at retailers, distributors and brand manufacturers ask the right questions on product and customer data when establishing a multi channel strategy.
The Multichannel Challenge: Availability of Relevant Information
At every customer touch point, the ready availability of product information has a profound effect on buying decisions. If your customers can’t find what they’re shopping for, don’t understand how well your product meets their needs, or aren’t confident in their choice, they won’t complete their purchase.
When customers are researching or actively online shopping for products, research says 40 is the magic number:
40 % of buyers intend to return their purchase at the time they order it.
40 % order multiple versions of a product.
40 % of all fashion product returns are the result of poor product information (Consumer electronics are 15,3%; Sources: Trusted Shops, 2012, Internet World Business 7.1.2013)
All the high-quality product data in the world is useless if an organization cannot leverage that data for quicker time to market, improved e-commerce performance, and greater customer satisfaction.
Four Business Objectives When Accelerating Multi Channel Commerce
This white paper comes with four common use cases that illustrate typical business objectives within a multichannel commerce strategy. When looking into your product information, here is a list of questions you might consider.
1. Increasing conversions and lowering return rates by ensuring that customers can access product information in an easy-to-consume form.
- Where is the flawed content coming from?
- What tools and incentives can we provide for suppliers to maintain the high quality content?
- Which data quality processes should be automated first?
- Do we need a bespoke data model to fit your requirements?
- Can we effectively use industry standards for communicating with suppliers (such as GS1 or eClass)?
2. Lowering manual processing costs by merging the best product content from multiple suppliers.
- How many product catalogs do we have and what are the processes that slow us down?
- Who is responsible for the quality of the product information?
- How can we define and enforce the objective and measurable policies?
- Which supplier has best descriptions / certain translation, high-quality images / video / etc.?
- How do we collaborate with our large and small suppliers to achieve best data quality?
3. Growing margins through “long tail” merchandising of a broader assortment of products.
- Can we automate product classification?
- Which taxonomy will work best for us?
- Do all stakeholders have visibility of data quality metrics and trends?
- How can we leverage information across all channels and customer touch points, not only ecommerce?
4. Increasing customer satisfaction through more consistent information and corporate identity across sales channels.
- How should we connect customer and product information to provide personalized marketing?
- How can we leverage supplier and location data for regional marketing?
- How do we enable crowd sourcing of comments, reviews and user images?
- What information do internal and external users need to access in real time?
Find more information with the complete white paper on multichannel commerce and data quality.