I was recently reading James Markarian’s entry on “What is ELT”, or Pushdown Optimization and wanted to relate his overview to some of my customer experiences in the past year. I’ll start from his joint CTO session with Stephen Brobst, Teradata’s CTO, delivered at Informatica World 2007 outlined in my earlier Thoughts on Informatica World 2007 post. The room was overflowing and attendees paid close attention to learn how to leverage their database in conjunction with Informatica’s data integration server through “hybrid architecture”. I think I know why our customers are so excited about this new innovation in data integration technology.

Many organizations have invested millions in large databases that have windows of available capacity when transaction or query loads subside. This phenomenon is prevalent in many of the large organizations I’ve visited while working to quantify the impact of data integration performance to real-world business results. I’m particularly interested in this area having spent 7 years of my career working on database performance. In the last 7 years, IT has increasingly examined usage patterns of infrastructure to optimize costs and increase leverage of existing investments. Database usage has come under increased scrutiny due to more specific pattern analysis and the need to hold costs down. Initial reviews early in the decade factored in total CPU usage as a way to measure server utilization. Now, with more in-depth analysis based on time and geography, patterns have emerged where windows of idle resource have been identified. These windows of opportunity are what ELT or “ Pushdown Optimization” seeks to exploit.

One financial services customer that I recently visited relies on a multi-terabyte enterprise data warehouse to analyze portfolio coverage across their client-base. Although enterprise in scope and international in use, a significant amount of CPU cycles were available for the *processing* of data. I’m differentiating processing from loading of data, as the architecture with Informatica at the time, was taxing disks but not using CPU to aggregate and transform data. Our initial analysis showed that by moving to hybrid pushdown/data integration server architecture, a significant load bottleneck could be addressed with minimal impact to existing design and no new investment in hardware.

Performance has traditionally been one of the biggest hurdles in achieving business results. It’s just one of the factors outlined in the whitepaper titled “Redefining Enterprise Data Warehousing”. Now with the advent of a “hybrid architecture” that allows organizations to balance data integration load across a wider variety of processing resources, without having to make changes to data flow design, Informatica has delivered a win-win for organizations looking to achieve high performance and increase leverage of existing investments.

Do you have excess capacity available to use in your IT environment?

Have you measured utilization of databases on a time and geographic basis?

I’ll bet you’ll find ways to optimize performance across your data integration environment that involve Pushdown Optimization.