Informatica Enterprise Data Management

We all know how mandates such as Sarbanes-Oxley place a burden on many businesses, by requiring that they be able to document the reliability and quality of data. Most major mandates, which have now been in place for several years, have given rise to a whole industry dedicated to reporting. In many companies, the equivalents of small departments have been kept busy 52 weeks a year doing little more than generating reports and reviewing data to meet compliance requirements.

Obviously, things can't go on like this. Rather than spending money to just keep simply meeting requirements, many companies are seeking to better integrate compliance into their day-to-day operations in a more automated, systematic form. In doing so, they seek to go far beyond meeting the letter of the law, to take the opportunity to improve and streamline their own processes - which will pay off in battling the challenges of an increasingly competitive marketplace.

By eliminating the silos that have separated data across the enterprise, as well as the silos that have pigeonholed the compliance efforts intended to gather and report this information, organizations can make impressive strides in moving forward with greater agility. In the process, automation can reduce the burden of paperwork and manual processes that drive up the costs of compliance.

Such "sustainable" compliance management can be built on top of three disciplines that already exist within most businesses today. These include governance, or the oversight of corporate activities and processes; risk management, or the identification, assessment and monitoring of risks and controls; and compliance management.  This integrated approach - known as Governance, Risk, and Compliance Management, or GRC, takes its three namesake disciplines and takes a more holistic approach to increasing information visibility and management. [Read more]

To paraphrase the Paul Simon song, there must be 50 ways to integrate your enterprise data. In recent years, companies have made all kinds of attempts to integrate both their applications and data - employing techniques from sophisticated enterprise application integration projects all the way down to manual hand coding. However, while most of these approaches work at least some of the time, few, if any, are delivering real agility for their businesses.

Recently, I had the opportunity to moderate a Webcast - sponsored by Informatica and hosted by ebizQ - which explored in detail an emerging approach, called data services, which ties into service oriented architecture (SOA) and creates a data abstraction layer that addresses the complexities seen across enterprise data environments.

Leading the Webcast were Ash Parikh, principal product marketing manager for Informatica and a highly regarded industry speaker and author, and David Ramos, director of business intelligence and analytics for LinkShare Corporation.

In his presentation, Ash urged closer collaboration between the enterprise data management and emerging service oriented architecture (SOA) worlds. (John Schmidt recently provided a nice overview of SOA here at the EDM blogsite.)

Ash observed that current approaches to enterprise data management have worked well from an application point of view, but have been ineffective for enterprise data. [Read more]

Healthcare is one of the last industries where you hear the term MDM (Master Data Management) mentioned. Most IT industry analysts, software firms and consulting organizations are geared towards your typical company that sells products to people or businesses. MDM examples are always getting a master list of products or cleansing your way to a consistent list of customers, which is not exactly the mindset of healthcare organizations. But lack of MDM is precisely what is adding untold costs on healthcare organizations (and ultimately on all of us) and inhibiting these organizations from improving the quality of health care services at an affordable cost.

Let's divide the healthcare industry (simplistically) into insurers and providers (we will position pharmaceuticals, biotechs and medical device companies as life sciences). Many of the large insurers have invested in data warehousing and data integration, but smaller insurers, i.e. regionally based HMOs (healthcare maintenance organizations) and healthcare providers, such as hospitals and physician groups, have fledgling efforts or have been bogged down in many of the issues below.

Healthcare organizations have significant data consistency issues regarding the following data subjects:

• Patients
• Physicians
• Procedures
• Diagnosis codes
• Service Rates
• Pay for Performance (P4P) measures

Each insurer and each healthcare provider tracks this data differently. The problems are magnified because healthcare is regulated on a state by state basis along with federal and industry regulations. Throw in privacy and security concerns to exacerbate what healthcare groups need to deal with.

Most healthcare organizations, even large ones, are an affiliation of generally small physician groups. These groups may be your local doctor's office, i.e. primary care physicians (PCP), specialists or emergency room (ER) providers. Often these groups do not have a lot of IT resources at their disposal.

Data flow is often flat file transfers between insurers and healthcare provider organizations, as well as from the individual physician groups and the larger provider organization. These flat files are generally not standardized and change each year when contracts are renegotiated between insurers and providers. This is an industry where you typically are not in control of your source data. It is thrown at you and you have to deal with it.

The need for an MDM is significant at healthcare organizations. The benefits from MDM are

• Data consistency
• Productivity
• Enabling more cost effect patient care

It is remarkable when one looks at the amount of resources devoted to manually dealing with inconsistent master data throughout health care. People in this industry do an amazing job of dealing with it, but it is often a time-consuming manual effort with much reconciliation. Having an MDM program would improve overall productivity and enable organizations to process and react more quickly to patients, insurers, employers and physicians.

A hidden jewel of an MDM effort though is enabling health care organizations to provide more proactive care. I have seen healthcare providers develop data solutions oriented to specific populations of patients who have diseases, chronic conditions or are at specific risks. These solutions may be for diabetes or asthma, for instance.

By bringing in historical clinical or demographic data related to patients tied together through consistent master data and taking advantage of predictive analysis, health care providers can proactively help their patients rather than waiting for the next episode when the patient's health has worsened. Many insurers are linking Pay for Performance (P4P) programs with these kinds of efforts because a healthier patient is a great goal by itself but better health also means lower health care costs.

The MDM silver bullet product has not been invented for healthcare industry but these organizations should not despair. There are concrete steps these organizations need to take.

• First, healthcare organizations need to examine where they are spending their resources on handling inconsistent master data and focus their efforts on those areas.
• Second, the efforts need to be in collaboration with business operations, physicians, insurers and IT, and need to involve defining master data and performance metrics.
• Finally, such organizations need to leverage their data warehousing and data integration efforts.

The most recent TDWI Boston Chapter meeting focused on how companies should approach and implement Master Data Management (MDM). Although the meeting had a keynote presenter and panelists with strong industry expertise and experience, the key to the discussions were the questions and insights of the audience attending the meeting.

The Boston chapter, with industries representing financial services, insurance, high tech, medical devices, biotech, retail, professional services and consumer products goods companies offers a diversity of perspectives about the challenges and benefits of addressing MDM.

Two key insights kept being reinforced during the discussions:

1. Leverage, Leverage, Leverage

Any company that will benefit from tackling MDM has most likely already been attacking the problem but not in the focused manner that MDM needs to truly be successful. But the biggest mistake that people saw with their peers and early adopters was the belief that MDM was something different than before.

Too often MDM is pitched as a "green field" opportunity with some solution as the "silver bullet" to one's problems. This approach fails to leverage past efforts from a business and technical perspective, thereby creating the potential for yet another application and data silo.

And, more importantly, failing to realistically assess the shortcomings and successes of existing efforts in making master data consistent is a sure fire way to plan to fail, i.e. repeating the failures of history is inevitable if you fail to learn from them.

Participants suggested looking at existing data warehousing and data integration efforts to leverage data, technical and people resources. Learn from the past.  Turn your joint IT and business efforts to define and manage data into a full-fledged  data governance program. If you have already started that type of program, expand  it and tie it into business successes (below).

2. Business Value

There needs to be specific business value derived from your MDM efforts. Catch phrases like "360 degrees of the customer" or "single version of the truth" are great marketing slogans but are esoteric and will become the brunt of jokes if they don't help you achieve real business value that can be measured. You can use these slogans to rally the troops and to get funding for the MDM efforts, but don't fall into that trap of believing your own sales pitch.

Your MDM will undoubtedly provide business ROI. Participants stressed that you should seek out those business opportunities and target your MDM towards them. Focused MDM efforts are more likely to get business participation, a critical success factor, to help you be on track to building your MDM program. Trying  to boil the ocean, i.e. trying to solve everyone's problems all at once,  generally fails to solve anyone's problems.

There are countless business processes or analytics that can be and are improved  by implementing MDM in your company. Find them, document them and determine  the business ROI that you can quantify or qualify. Get the business people involved  to be your customer references to sell the MDM program.

Success breeds success.

SOA has nothing to do with technology. It has everything to do with defining and managing the business as a collection of service functions and information exchanges. A business may be viewed as an internal organizational unit within a large company that provides services to other other units and consumes services from yet other groups. Or if you view the business as the company overall at the macro level, then you could define and manage the business as a collection of services consumed from its supply chain and provided to its customers. [Read more]

The most recent TDWI Boston Chapter meeting focused on MDM (Master Data Management). The keynote presentation titled “MDM: Data Salvation or the Next Round of Silos?” was followed by a panel discussion. The panelists included representatives from IBM, Oracle (Hyperion), SAP (Business Objects) and Kalido.

In keeping with TDWI principles, the panel discussion concentrated on the how’s and why’s of MDM in customer situations and avoided sales pitches. I moderated the panel discussion and was very impressed with the panelists’ knowledge and insights on their customers’ experiences in approaching and implementing MDM.

The panel discussed the most common characteristics that customers who are experiencing success in their MDM efforts have:

“Educated consumers”

There is a men’s clothing store that has an ad that says an educated consumer is our best customer. That is absolutely true in MDM. Companies that have been involved in data warehousing and enterprise data integration understand the complex and conflicting world of trying to get consistent, comprehensive and current master data or conformed dimensions in DW terminology.

Significant business participation

MDM is not a program that can be undertaken with IT alone. In fact, if you do not have business participation in the beginning there is little chance of success. IT has to sell the MDM program to the business and get real resource and time commitments.

Data governance program

Data governance is more important than what products you use to implement MDM. The old saying about garbage in, garbage out (GIGO) is absolutely true in MDM. You can’t cleanse data nor make it consistent unless you can define the data, business rules/transformations and performance measures you wish to use across your enterprise.

Enterprise Data Integration

Companies that have been implementing enterprise data integration efforts can leverage those efforts in their MDM program. They have gained an understanding of the complexities that a formal MDM effort will entail. For these companies MDM is not a new thing but simply taking their EDI efforts to the next level.

MDM is a journey and companies that have been in the trenches trying to address the problems of inconsistent data are well-positioned to take the next step.

Many thanks to Loraine Lawson who wrote an insightful article on her blog Classifying Integration Initiatives as Tactical or Strategic Her posting got me thinking. 20 years ago we didn’t need Integration Competency Centers – now we do. What changed?

From my perspective, there are three key drivers:

First, organizations have gotten bigger. The Fortune 500 companies today are much larger in terms of revenue, number of employees and global operations than Fortune 500 companies in the 1980’s. This conclusion is based on personal experience and anecdotal evidence. If readers of this column have seen any studies or statistics on the growth of large enterprises, please speak up.

Second, data diversity and volume has been growing at an exponential rate. We now have data in multiple formats such as video, images, audio, and text to name a few – and in multiple protocol formats and multiple languages. On the volume front, I recall an article from a few years ago that stated that manufacturers of storage devices would ship more than 22 exabytes (22 million trillion bytes) of hard disk capacity in 2005 - which is four times the space needed to store every word ever spoken by every human who has ever lived. In a relatively short number of years we’ve seen data storage capacities move from megabytes, to gigabytes and terabytes – now we’re talking in petabytes and exabytes. And coming soon are zettabytes and yottabytes.

Third, information technology has become more complex. One of the first principles of computer science is that re-use and decoupling of components is enabled by adding an abstraction layer. When I started my career (roughly 30 years ago), I was loading programs into computers using paper tape readers and toggle switches (I was pretty good at reading octal and hexadecimal code in my prime). But with increasing layers of abstraction – macro code to structured programming languages to 4^th generation languages to graphical integrated development tools to cloud computing in a web 2.0 environment – we have increasingly distanced the user from the underlying hardware and micro-code. Computers basically still work the same as they did 30 years (binary logic gates), but they are much more powerful, easy to use, and interchangeable which has been enabled by layer upon layer of abstractions.

The common elements across these drivers is constant change and growing incompatibility and complexity. Business processes are constantly changing with broader global scope adding many complications that are not transparent. Data definitions are constantly morphing and volumes are increasing to massive levels as everything becomes digitized and more regulated. Technology waves are constantly adding new variations and complexities without retiring prior generations.

Ergo, we have seen the emergence of integration as a new discipline which didn’t exist 20 years ago because it wasn’t needed then. In the 1980’s, integration was viewed as a one-time project discipline, not as an ongoing capability that an organization needed to sustain efficient end-to-end operations in a complex pattern of incompatible components that are constantly changing.

We could say that projects require micro-integration techniques while the complex systems-of-systems that emerge over time at the enterprise and supply chain scale require macro-integration best practices such as Integration Competency Centers. The ICC disciplines are a powerful capability, but they are non-trivial and hence those organizations that do it better than others will have a competitive advantage.

It seems in recent years pundits and vendors alike have been applying the 2.0 label to everything and anything emerging across the technology plain. In some cases, the new label has stuck - witness the widespread adoption of the terms 'Web 2.0' and its business sibling, 'Enterprise 2.0.'

In some cases, it’s a case of marketecture, but yet, the 2.0 identifier does convey a certain sense of maturity – that a technology is moving to a new stage of sophistication, of engagement with the business and its end users.

There have been moves afoot to identify the next generation of data warehousing as "Data Warehouse 2.0." However, there are differences of opinion as to what exactly will constitute DW 2.0, and thus no clear standard sense of direction in the market. [Read more]

This Wednesday, June 25, I have the privilege of hosting a Webinar featuring Ash Parikh, Informatica’s Principal Product Marketing Manager and a well-known author and speaker on enterprise data integration issues. Ash will be joined by David J. Ramos, Director of Business Intelligence and Analytics at LinkShare, an Informatica customer that provides online marketing services.

The Webinar, entitled Data Services - Maximizing Business Value Through Right-Time Information, is sponsored by Informatica and will be available live via ebizQ at 12:00 pm Eastern Time.

UPDATE: Archived replays of the Webcast are now available on demand.

Ash Parikh will discuss why many of the current approaches to integration - such as enterprise application integration (EAI), enterprise information integration (EII), and many manual processes still in use – are not giving organizations the agility they need to move to truly real-time, customer-focused enterprises. He will discuss an emerging approach - called data services - that creates a data abstraction layer that opens up all these formerly unreachable data stores across the organization.

David Ramos will explain how LinkShare, which handles 40 GBs of data across 300 million transactions a day, is employing Informatica technology to deliver grid-based data integration and meet the growing real-time data demands of its customers.

This promises to be a very informative and engaging session. Again, the live presentation will take place this Wednesday, June 25, at Noon Eastern Time.

Archived, on demand replay available here.

Integration is a discipline. That is, the task of making independent and autonomous entities work together in a synergistic fashion requires unique skills and practices within the context of an enterprise strategy. You may well ask "isn't integration just the same thing as management?" Not really. Management disciplines are clearly one aspect of an integration discipline, but they are insufficient because of two key characteristics: incompatibility and independence. The variations in data and process definitions across software packages and across organizational groups in combination with the autonomous nature of each link in the flow of data drives the need for a specialized management practice.

The autonomous nature of each link in the chain means that we have limited control of all the elements that make up an end-to-end operation. Clearly we have limited control when we partner with another business in the supply chain and we certainly can't control all the regulatory demands, but surely we have control within our own organization so management practices alone should suffice - but do they? While this may have been true years ago in vertically integrated organizations with more autocratic cultures, times have changed. For starters, many large organizations today operate more like a federation than a dictatorship. And the people that work in the organizations have a mind of their own, come from all over the world (and even work all over the world) and often aren't permanent employees of the company. Traditional management practices are no longer sufficient in this new environment.

Integration is a distinct discipline that seeks to find the optimal balance between efficiency in a complex chain of responsibility while remaining adaptable to change and without stifling innovation at each link in the chain. Integration demands unique practices that build on traditional management practices. For example, integration techniques are often more of an agreement process rather than an analytical process.

Integration as a process can be done manually or with the assistance of technology. The internet is a good example of highly automated integration. A person can connect to the internet, discover an unused website name, register it, and create the website; within a short time search engines will index the new site so that others can find it. Voila - you are in business. All this is done totally automatically without human intervention because of common standards, protocols and languages.

What are implications for individual enterprises? Effective business integration can drive bottom line results and if done well, integration will:

• Lower ongoing costs and one-time initiative costs by reducingduplication and redundancy in systems, data and business processes.
• Reduce time to market for new product introductions by developing standardized business services layer that decouples agile business processes from the underlying technical implementation.
• Drive revenue by allowing individual functional units to innovatewhile still leveraging common shared functions.
• Improve customer satisfaction by providing an integrated view of the customer and increased reliability resulting from a more simplified technical infrastructure.
• Improve the employee's quality of life by reducing frustrations of "swivel chair integration" and providing systems that provide the information they need, when they need it and where they need it to do their work.

While these benefits sound great, there are in fact significant implications for achieving them. At a business operational level achieving these benefits requires:

• New organizational structures and relationships such ascross-functional teams with authority to make trade-off decisions. For example, providing product bundles where individual product managers may see lower margins, but the enterprise overall benefits from increased wallet share.
• Changes to metrics, goals and rewards. For example, when just about any function can be performed in just about any channel, you may need to change incentives to encourage store sales staff to steer customers to lower-cost channels. And you may need to add metrics that track cross-functional sharing and reuse and reward managers accordingly.
• Revamped governance processes for funding of cross-functionalmulti-year initiatives. Single-project and single-iteration budget decisions need to be supplemented with planning and governance that takes a two-to-three-year view of application portfolio upgrades.
• More effective communications between functional groups and across geographic regions. And just as important, a closer ongoing strategic working relationship between business functions and IT functions.
• Common vocabulary for data and business processes to enable one version of the truth. The key to process integration is data standardization. As standardized data are made available, business owners can effectively integrate their processes.
• A 360 degree view of customers. Simplified and personalized presentation is a powerful combination that leads to greater customer loyalty and retention.

For the IT organization the implications are no less significant.

• An integration group is needed which is a permanent part of theorganization.
• A defined application portfolio is required with clear accountability across business and IT functions.
• Strict standards for process and data interactions across applications. While you can continue to allow significant autonomy of decisions within an application domain, the critical nature of how all systems are connected with each other requires tight controls over their interactions.
• The need for an integration methodology in addition to change management and software implementation processes. Integration initiatives must start with a definition of the end-to-end business processes and be based on a common business language.
• Implementation of new technology and tools to enable end-to-end dataintegration.
• Development of staff skills in formal modeling techniques and tools.The skills required to abstract functionality and generalize business concepts for enterprise-wide use are not always common in technical staff and must be developed.
• Standards compliance as a matter of principle rather than a matter of convenience. Standards that support the principle of loose coupling between systems are enablers for an agile business.

These capabilities are enabled by a fundamental understanding that integration is a distinct discipline. Organizations that view integration as a "strategy" and that focus their people, policies and investments around the strategy will have a clear competitive advantage. They will create an agile business where each link in the chain can change and adapt to meet local needs while the end-to-end chain remains strongly aligned with the overall operating model.