Tag Archives: Pushdown Optimization
Flipping around traditional ETL (Extract-Transform-Load) on its head is not a new practice. ELT (Extract-Load-Transform), where processing is handled in the database, instead of the ETL server, has been proven to enhance performance in many types of data warehousing deployments.
For example, Oi, a leading telecom provider in Brazil, implemented an enterprise data warehouse (EDW) consolidating information on 36 million customers, speeding response time to customer requests. The right-time EDW also enabled Oi to rapidly launch a successful new service offering, which made it easier for customers to recharge their pre-paid accounts for telecom service.
By implementing ELT with Informatica’s pushdown optimization capabilities for this Teradata data warehouse, Oi accelerated its data warehousing loading process two-fold. This has led to even more timely updates of Oi’s customer information, while lowering costs.
ETL (Extract-Transform-Load) technology has been around for over a decade, and while it rocked the world in the 90’s, it’s considered a bit of a relic nowadays. Data warehousing, the original driver for ETL technology, isn’t considered as sexy anymore. That’s in part why vendors have used different names to broaden this software category and added new capabilities to keep it relevant.
Informatica is no exception. We’re “the Data Integration Company“, where data integration consists of many different capabilities, only one of which is ETL (granted, the ETL piece is the cornerstone for data warehousing and other data integration projects).
And the letters E-T-L themselves have been put in the blender to be reconfigured into newer, fresher concepts. ELT or ETLT incorporates the concept of pushdown optimization, where processing is handled in the database, instead of the ETL server. (For more detail, Rajan Chandras has a good post discussing ETL vs. ELT.) ETQL pulls data quality into the ETL workflow. And I’m sure the permutations will continue.
So, is classic ETL just not relevant anymore? (more…)
I was recently reading James Markarian’s entry on “What is ELT”, or Pushdown Optimization and wanted to relate his overview to some of my customer experiences in the past year. I’ll start from his joint CTO session with Stephen Brobst, Teradata’s CTO, delivered at Informatica World 2007 outlined in my earlier Thoughts on Informatica World 2007 post. The room was overflowing and attendees paid close attention to learn how to leverage their database in conjunction with Informatica’s data integration server through “hybrid architecture”. I think I know why our customers are so excited about this new innovation in data integration technology.
Many organizations have invested millions in large databases that have windows of available capacity when transaction or query loads subside. This phenomenon is prevalent in many of the large organizations I’ve visited while working to quantify the impact of data integration performance to real-world business results. I’m particularly interested in this area having spent 7 years of my career working on database performance. (more…)