Informatica Perspectives

As I discussed in my last posting, ELT or pushdown optimization can significantly improve data warehousing performance, while reducing costs.  I also mentioned it’s important to implement a data integration platform that supports both traditional ETL (Extract-Transform-Load) and ELT (Extract-Load-Transform) methods, because different situations call for different methods.

Taking this thought a step further, metadata is the critical binding agent that should cut across all data integration approaches, be they ELT or ETL or some other combination such as ETLT.  If the actual transformation logic and business rules are defined as metadata, the choice of where the processing actually occurs, be it the ETL server or the database/data warehouse, becomes a matter of configuration rather than of coding.  A truly metadata-driven data integration platform enables you to design and reuse the same transformation rules, regardless of whether you choose ELT or ETL for data warehousing.  [Read more]

Flipping around traditional ETL (Extract-Transform-Load) on its head is not a new practice.  ELT (Extract-Load-Transform), where processing is handled in the database, instead of the ETL server, has been proven to enhance performance in many types of data warehousing deployments.

For example, Oi, a leading telecom provider in Brazil, implemented an enterprise data warehouse (EDW) consolidating information on 36 million customers, speeding response time to customer requests.  The right-time EDW also enabled Oi to rapidly launch a successful new service offering, which made it easier for customers to recharge their pre-paid accounts for telecom service.

By implementing ELT with Informatica’s pushdown optimization capabilities for this Teradata data warehouse, Oi accelerated its data warehousing loading process two-fold.  This has led to even more timely updates of Oi’s customer information, while lowering costs.
[Read more]

ETL (Extract-Transform-Load) technology has been around for over a decade, and while it rocked the world in the 90's, it's considered a bit of a relic nowadays. Data warehousing, the original driver for ETL technology, isn’t considered as sexy anymore. That’s in part why vendors have used different names to broaden this software category and added new capabilities to keep it relevant.

Informatica is no exception. We're “the Data Integration Company", where data integration consists of many different capabilities, only one of which is ETL (granted, the ETL piece is the cornerstone for data warehousing and other data integration projects).

And the letters E-T-L themselves have been put in the blender to be reconfigured into newer, fresher concepts. ELT or ETLT incorporates the concept of pushdown optimization, where processing is handled in the database, instead of the ETL server. (For more detail, Rajan Chandras has a good post discussing ETL vs. ELT.) ETQL pulls data quality into the ETL workflow. And I'm sure the permutations will continue.

So, is classic ETL just not relevant anymore? [Read more]

Maintaining product lists is often cited as a great example of Master Data Management (MDM). Many companies that manufacture or sell products need to get a consistent list of products for a variety of business reasons. The business value includes tracking what you sell to customers and also how you manage your supply chain. Product firms create products organically (internally) and through acquisitions. In both cases, each product line has, at least for part of its life, been operated as a separate business. At some point in the product life and sales cycle, business conditions dictate a transition into the company's product portfolio. Although managing product lists is not always a simple task, it is only the tip of the iceberg for companies that design or engineer products. These companies have a need for a more complete PIM (Product Information Management) solution that extends far beyond simple product lists.

If you design or engineer products, then you need to track product designs and configurations that evolve and change over time. This applies to many manufacturing industries from high tech, consumer products, defense, and automobile to even farm machinery. These designs and configurations are likely scattered across many databases and often unstructured data sources. This data is not stored in your classic data warehouse (DW) or integrated through your run-in-of-the-mill ETL tool. You need to think outside your typical DW effort and determine how to get that data integrated into your PIM solution.

I did work for a farm machinery company many years ago. Their product data issues involved configuring combines, harvesters and other machinery that cost six and sometimes seven figures. This data challenge was not a trivial endeavor. These farming machines are highly customizable by the agricultural organizations purchasing them. The vendor needed to track what was available, what was sold and to whom. The customers needed to know what was available for their agricultural needs, have those machines built and then be able to service them for years.

What have these companies done and what should you do if you are just starting to design and implement a PIM solution?

• Get your data warehousing and data integration in place to support the classic product, sales and customer data stored in your various source systems
• Work with your engineering and product design groups to understand what they have in place to manage engineering drawings, specifications, configurations and all the associated versioning
• Get your sales and customer support organizations to define the PIM requirements
• Implement a data governance effort to get a handle on both your structured and unstructured data
• Leverage data integration capabilities that can handle the variety of data supporting PIM

The business benefits are to both the top line (increased sales) and bottom line (managing and reducing costs) when engineering and product design firms implement a PIM solution.