Monthly Archives: October 2007
ETL is not pervasive yet!
Recently I moderated a panel at the Boston TDWI chapter (I am a chapter officer) on emerging trends in business intelligence (BI). I framed the discussion by having the panelists position technology in the five stages of the Gartner Hype Cycle.
It was a lot of fun and provided some good insights. The panel agreed that ETL was on the productivity plateau — meaning it was mainstream and commonplace. Everyone assumes everyone is doing it, but I challenged whether it was truly pervasive.
To support my claim I did an informal survey of the audience and asked some questions on their use of ETL. Sure enough, everyone was using it — that’s great news. And everyone was using it to load their data warehouse — again terrific.
But here is where the fun and eye-opening insight begins. When asked if they used their ETL tool to load their data marts it turns out most did not. And how many loaded their OLAP cubes with their ETL tool? Almost nobody.
This is consistent with what I see time and time again at my clients and what I hear from fellow consultants and IT folks. Recent surveys indicate that approximately 45% of ETL work is done by hand-coding.
(more…)
Real-time Data – A Data Integration Challenge
One technical challenge not often discussed in data integration circles is the impact of real-time data to performance and scalability. I attribute this to a lack of real-world experience in handling real-time data, or a lack of recognition by IT that data integration software can effectively manage real-time data. Many architects and IT developers that I meet lump real-time into the EAI domain. This was a logical assumption 5 years ago, due to the fact that the data integration market was then primarily known for tackling “large batch volume” workloads (or as I like to refer to them “big batch problems”)
Informatica has spent 10 years focused to a good degree on solving that “big batch” problem. The inherent division between design time and run time in the underlying platform architecture enabled the introduction of parallelization/partitioning techniques, 64 bit processing, support for RDBMS vendor supplied batch utilities/APIs and improved data conversion/transformation without impacting the business logic design. This has proven invaluable to our customers in meeting their increasing volume, and in shrinking load window requirements.
(more…)
You Don’t Need M&A to Justify Data Integration
Data integration gets the priority treatment when there’s been a merger or acquisition (M&A). In his recent post (M&A and Divestitures Need Effective Data Integration), Don Tirsell captures some of the key data-integration issues that need to be addressed after the M&A dust settles. After all the press releases and analysts’ reviews of an acquisition, the companies involved have to roll up their sleeves, settle in, and work towards a successful and profitable M&A.
But I’d like to pose a more fundamental question: Why limit the data-integration focus to M&A activities? Yes, it’s obvious that when companies are involved in M&A they need to integrate their application and data silos to increase the top line (revenue), decrease the bottom line (costs), and ultimately increase profits. But, knowing this, and knowing that most (all?) companies have application/data silos, why isn’t it obvious that they need to integrate that data to improve business?
The M&A data-integration activities are undertaken with a sense of urgency because it makes business and technical sense to do them – plus, M&A shines a new spotlight on both companies. But meanwhile, other companies sit back and accept their silos, as if that is simply the way it should be.
Do they need the kick-in-the-pants of an M&A to get their data-integration efforts in gear?
(more…)
Loading ...