Informatica Enterprise Data Management

No longer do companies keep increasing tech spending “like it’s 1999.” Companies have become more selective in what they invest in. The key word is “investment.” Most companies don’t just go out and buy the latest and shiniest technology, no matter how much hype there is, unless there is a solid business ROI (return on investment.) No discretionary project gets funding without the ROI.

I am an avid reader of the Wall Street Journal (WSJ) Business Technology blog written Ben Worthen. In his post “Oracle and Accenture: Not a Rising Tide”, December 20, 2007, he postulates that although Oracle and Accenture just had terrific quarters and are forecasting good times in 2008, tech spending will be flat and other tech vendors will not experience the same solid growth as these two companies. Unlike in 1999, there will not be a rising tide for all tech companies. I couldn’t agree more.

I don’t know (nor does anyone else, really) whether tech spending will be flat or will grow. (It could grow because we will avoid a recession or because people’s estimates are too conservative, as they have been the last few years.)
[Read more]

I am seeing a disconnect between many clients' 2008 plans for service-oriented architecture (SOA) and the expectations they are setting with the business groups that are funding them.

The good news is that people have been pervasive in getting sponsorship and funding for SOA projects based on anticipated business benefits. The bad news is that despite good technology, many IT projects fail when business group don't get what they think they were promised. It’s like the old saying “you can talk the talk, but can you walk the walk?”

The promise of SOA is to establish reusable services that can be deployed throughout the enterprise to deliver accelerated application implementation at lower risk and effort. Reusable business services reduce redundant or overlapping development efforts and increase IT productivity. They also enable applications that were previously siloed to exchange data and interoperate.
[Read more]

Do you have projects lined up for next year that will deal with reference data? It's a good idea, especially if you want to better understand your customers, determine the profitability of your products or work more effectively with your suppliers.

If your projects are formal they may be identified as master data management (MDM), customer data integration (CDI) or product information management (PIM). All of these are examples of reference data.

Getting your reference data in order – reflecting what is and was – helps to establish data consistently. This, in turn, enables historical analysis such as trending, and helps you plan for the future. Some of you will not have projects with the MDM, CDI or PIM labels, but if you look beneath the covers you almost inevitably will see efforts to improve your reference situation.

The reasons the MDM, CDI and PIM acronyms have emerged in our industry is because most businesses have not done a great job with reference data yet and because it is difficult. But, as with many things in life, if it was easy it probably wouldn’t be worth doing.

Don’t let a vendor convince you that all you need is their shiny new MDM, CDI or PIM product. Software alone won’t solve the problem; otherwise it would have been solved a long time ago.
[Read more]

One of the dirtiest unspoken secrets in business intelligence (BI) and performance management (PM) is that most business people, despite the sizable investment in BI, PM and data warehousing, are not getting the information they need to make decisions. (See this Accenture study.)

And the reason for that, yet another unspoken truth, is that most business people are performing their analysis via data shadow systems or spreadmarts.

A quick recap: a data shadow system is an application built by the business. It gathers data from many sources, and business people often augment that data manually, using Microsoft Office for reporting and analysis. Ask business people how they really get their information and their answer is almost sure to include a data shadow system.

Too often the “problem” of data shadow systems is that people are using Microsoft Excel as the front-end BI tool. That is really not the problem. The real problem causing inconsistent and error-prone data is that people are using both Microsoft Excel and Microsoft Access to perform ETL! A typical data shadow system has:

• between six to three dozen steps (and I have seen systems with hundreds!) of Microsoft Office (Microsoft Access and/or Microsoft Excel) queries or imports gathering data,
• a series of steps to “integrate” the data, and finally,
• a number of worksheets to create the reports.

[Read more]

Your 2008 strategy and budget are probably all set. Now it is just a matter of tactics to implement your business and IT objectives for next year. Have you examined the top business initiatives and IT projects in the context of integration?

Unfortunately, companies historically design and deploy business initiatives and their supporting IT projects in business and application silos. They justify these projects with a solid business return on investment (ROI) based on very high expectations of what the business will get. The expectation may involve great looking dashboards, terrific visualization, real-time access and widely distributed reports.

But many of these projects will fall into what Gartner Research describes as the “trough of disillusionment.” Too many times these projects will be labeled failures, not because the technology was not terrific, but because the data was not correct or consistent. A beautiful visual display or enabling pervasive data access is only of value to the business if that data is right. It’s all about the data. And that’s all about the integration.
[Read more]

The recent Informatica Release 8.5 launch highlighted Real-time Integration Competency Centers (ICCs) as the optimal model for successful data integration. I’d like to review the concept of the Real-time ICC and why Release 8.5 supports this advanced operational, organizational and technology model.

As data integration moves beyond the realm of data warehousing into operational integration, real-time and data services use cases have exploded in importance to the business and necessitated stronger, unified infrastructure for IT to meet the challenge. Philip Russom, Senior Manager, TDWI Research captures this trend specifically in his quote on Release 8.5.

"The movement toward real-time data access and delivery has been the most influential trend in data integration this decade. The trend has enabled user organizations to initiate a variety of valuable real-time practices, including operational BI, real-time data warehousing, on-demand computing, performance monitoring, just-in-time inventory, and so on. And the trend has led vendors to extend their data integration products, so that many functions operate in real-time, not just batch. Informatica 8.5 is a great example of this trend, because it’s re-architected to support more real-time and on-demand functions for data integration, changed data capture, and data quality." [Read more]

I teach data warehousing and business intelligence courses at Northeastern University's Graduate School of Engineering. It provides me a fresh perspective on what we are doing in the industry. This is because unlike when I am talking with customers, analysts and vendors, my graduate students are always asking "WHY."

They ask because they do not have our industry experience or background. Yes, I do need to take the time to explain a lot of things that I otherwise would not have to, but that is actually good news. The graduate students also have not been locked into a particular way of doing things because they see everything as a new experience. They have open minds.

It would be a good idea for anyone planning projects for next year to have an open mind, too.
[Read more]

One technical challenge not often discussed in data integration circles is the impact of real-time data to performance and scalability. I attribute this to a lack of real-world experience in handling real-time data, or a lack of recognition by IT that data integration software can effectively manage real-time data. Many architects and IT developers that I meet lump real-time into the EAI domain. This was a logical assumption 5 years ago, due to the fact that the data integration market was then primarily known for tackling “large batch volume” workloads (or as I like to refer to them “big batch problems”)

Informatica has spent 10 years focused to a good degree on solving that “big batch” problem. The inherent division between design time and run time in the underlying platform architecture enabled the introduction of parallelization/partitioning techniques, 64 bit processing, support for RDBMS vendor supplied batch utilities/APIs and improved data conversion/transformation without impacting the business logic design. This has proven invaluable to our customers in meeting their increasing volume, and in shrinking load window requirements.
[Read more]

Recently I moderated a panel at the Boston TDWI chapter (I am a chapter officer) on emerging trends in business intelligence (BI). I framed the discussion by having the panelists position technology in the five stages of the Gartner Hype Cycle.

It was a lot of fun and provided some good insights. The panel agreed that ETL was on the productivity plateau — meaning it was mainstream and commonplace. Everyone assumes everyone is doing it, but I challenged whether it was truly pervasive.

To support my claim I did an informal survey of the audience and asked some questions on their use of ETL. Sure enough, everyone was using it — that’s great news. And everyone was using it to load their data warehouse — again terrific.

But here is where the fun and eye-opening insight begins. When asked if they used their ETL tool to load their data marts it turns out most did not. And how many loaded their OLAP cubes with their ETL tool? Almost nobody.

This is consistent with what I see time and time again at my clients and what I hear from fellow consultants and IT folks. Recent surveys indicate that approximately 45% of ETL work is done by hand-coding.
[Read more]

Data integration gets the priority treatment when there's been a merger or acquisition (M&A). In his recent post (M&A and Divestitures Need Effective Data Integration), Don Tirsell captures some of the key data-integration issues that need to be addressed after the M&A dust settles. After all the press releases and analysts' reviews of an acquisition, the companies involved have to roll up their sleeves, settle in, and work towards a successful and profitable M&A.

But I'd like to pose a more fundamental question: Why limit the data-integration focus to M&A activities? Yes, it's obvious that when companies are involved in M&A they need to integrate their application and data silos to increase the top line (revenue), decrease the bottom line (costs), and ultimately increase profits. But, knowing this, and knowing that most (all?) companies have application/data silos, why isn't it obvious that they need to integrate that data to improve business?

The M&A data-integration activities are undertaken with a sense of urgency because it makes business and technical sense to do them – plus, M&A shines a new spotlight on both companies. But meanwhile, other companies sit back and accept their silos, as if that is simply the way it should be.

Do they need the kick-in-the-pants of an M&A to get their data-integration efforts in gear?
[Read more]