Informatica Enterprise Data Management

One of the dirtiest unspoken secrets in business intelligence (BI) and performance management (PM) is that most business people, despite the sizable investment in BI, PM and data warehousing, are not getting the information they need to make decisions. (See this Accenture study.)

And the reason for that, yet another unspoken truth, is that most business people are performing their analysis via data shadow systems or spreadmarts.

A quick recap: a data shadow system is an application built by the business. It gathers data from many sources, and business people often augment that data manually, using Microsoft Office for reporting and analysis. Ask business people how they really get their information and their answer is almost sure to include a data shadow system.

Too often the “problem” of data shadow systems is that people are using Microsoft Excel as the front-end BI tool. That is really not the problem. The real problem causing inconsistent and error-prone data is that people are using both Microsoft Excel and Microsoft Access to perform ETL! A typical data shadow system has:

• between six to three dozen steps (and I have seen systems with hundreds!) of Microsoft Office (Microsoft Access and/or Microsoft Excel) queries or imports gathering data,
• a series of steps to “integrate” the data, and finally,
• a number of worksheets to create the reports.

These data shadow systems are where data is truly transformed from data into business information. This is because there is often a gap between what is in an Enterprise Data Warehouse and what the business person needs to do analysis. If IT fails to close that gap with data marts or cubes then the business will fill that gap with data shadow systems.

Many of your IT initiatives for next year are centered on delivering more comprehensive, consistent and current information to support projects such as BI, PM or customer data integration (CDI). The secret isn’t necessarily to pump up your EDW, but to transform the data that is in that EDW into usable business information for these initiatives.

What should you do? Step back, take an inventory of your data shadow systems, prioritize which ones are currently supporting key business processes and then replace/renovate those that will produce the top business ROI.

When you are replacing or renovating a data shadow system your main concern should not be whether you should use Microsoft Excel as a BI front-end. Rather, you should focus on using your data integration tool to replace the ETL cobbled together in these systems. The BI or PM initiative is only as good as the underlying data. That is what you should concentrate on.

Your data integration suite should have a portfolio of options to replace the ETL processes of your data shadow systems. These options include your standard ETL batch process, pushdown or ELT processes, real-time EII (enterprise information integration) or even data integration services (implemented as part of your SOA strategy).

No matter what data integration choice you make it surely will be a significant improvement from stringing together Microsoft Office products to perform ETL.