The Right Way To Do Data Integration For Applications

Informatica 9 On the 1st October 2009, I participated in a webinar “The Right Way to Do Data Integration for Applications,” hosted by David S. Linthicum, a recognized expert in SOA, Cloud computing and Enterprise Application Integration. It was an event that was very well attended and generated a lot of interest from the attendees judging by the large number and quality of questions that were submitted. I recommend that you listen to the replay and download the associated white paper he wrote on that subject.

David covered some of the limitations he has encountered over the years with the way Enterprise Integration Application (EAI) and Enterprise Service Bus (ESB) technologies deal with the integration of data, deployed in a SOA initiative.

I found the content covered by Dave to be very interesting as it fully related to some of the real-world experience I had gained over the years working with customers. Based on my experience, EAI and ESB technologies typically manage process integration and transactional event-based message movements in a publish/subscribe environment. But when it comes to the manipulation of the data itself, these technologies quickly reach their limits leading to a hand coded approach or at best, abusing the capabilities of the EAI/ESB.

Fortunately enough, some of these companies were able to leverage the data integration capabilities of the Informatica Platform to enable a number of key capabilities that were missing in EAI/ESB.

Here are the key capabilities that the Informatica Platform can seamlessly address with respect to the limitations of EAI/ESB:Bulk data movement scalability

  • Rich metadata capabilities
  • Set-oriented data transformations
  • Codeless environment
  • Non-intrusive database transaction detections
  • Ease of deployment and modification

These valuable capabilities, from an IT perspective, provide immediate return on investment to the SOA initiative by providing the right way to enable data integration development and reduce the costs of maintenance.

But it is the second round of capabilities that brings the most value. EAI or ESB technologies are meant to propagate transactions over queues in the quickest way possible. In other words, if the data associated with a transaction is corrupt, then this same corrupt data will get propagated to all the applications, having significant impact to the business process.

The Informatica Platform can provide a data quality firewall to the company, ensuring that corrupted information does not enter the business transactions. Or, it can provide data quality checks as a service to the applications (also known as Data Quality point of entry) so that the information is verified in the earliest step of the process.

Finally, all these data integration patterns such as bulk processing, transaction capture, set-oriented transformations, or data quality certification can all be delivered as a standards-based service to the EAI/ESB. This is a great starting point for implementing a data services foundation in an SOA strategy.

I strongly recommend you take a look at this webinar.

This entry was posted in Data Governance, Data Integration, Data Integration Platform, Data Quality, Data Services, Data Synchronization, Enterprise Data Management, Pervasive Data Quality, SOA and tagged , , , , , , , . Bookmark the permalink.

2 Responses to The Right Way To Do Data Integration For Applications

  1. william says:

    Hi Bertrand

    I know very well what you are all talking about.

    The main issue here is not only technical, it is also an organizational maturity issue. From years, Production (Ops guys) were in charge of making batch transfer from legacy system, then ERP, then B2B, etc.

    Then came ETL. Then EAI/ESAB managed more by an integration team. But guess what, data movement was still managed by ops guys and using a scheduler.

    Data stewardship and data governance should be number one priority for the business, and the Informatica platform should be used accordingly. Not only for physical data migration (aka ETL), but also for data quality and logical data management.

    Raise the maturity of your company and use the integration platforms you bought accordingly.

  2. Hi William

    Yes, I would agree with you that IT organization maturity has been a limitation to industrialization of data integration between systems. In fact, IT organizations are often very application-oriented, so as siloed as applications are. Each application group would have its dedicated team that would only support its own applications tools (Cobol, ABAP, PL*SQL, Script, java, etc.).

    SOA is changing these rules. IT organizations adopting an SOA approach undergo a mutation that leads them to reconsider this old approach.

    The raise of SOA with the definition of common services shared across business processes is shifting the balance of power from application management to process management. SOA process management should by definition be cross enterprise and independent from applications. But it won’t remove the need for point to point interfaces. It will remain because business processes mandate both real-time transaction management as well as batch consolidation and reconciliation of information. What Informatica brings, is a common platform that can manage all these complex data interfaces and deliver it as a service as part of the SOA strategy. It leads to the promise of SOA to providing higher agility at lower cost.

    Another way to accelerate this mutation is as you mention a data governance approach. But before the business can adhere to that strategy, they must see the economical impact of inaccurate data to their processes and then change their processes accordingly for better operational efficiency. It is at this stage that they can partner with the IT organization to bring this value. Again, the Informatica platform provides data quality services that can be hooked to the process, embedded in the application, as part of the ESB mediation, or shared and reused in many other ways. It also ensures correct collaboration between the business and IT to make these services efficient.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>