Informatica Perspectives

I was asked to participate in the new “SOA Data Integration Architect Community,” or SDIAC, which launches today. As most of you know, I don’t like to work with vendor-focused standards bodies or on-line communities, but I found out quickly that this was very different.

“The SOA Data Integration Architect Community is the world's first open and free community that focuses on the value of data integration and data services in agile architectures such as SOA. This one-of-a-kind community will bring together software professionals who will contribute to and be involved in software architecture, to raise awareness about the foundational value of data in enterprise architecture. This includes enterprise architects, application architects, data architects, solution architects, IT architects and IT managers responsible for or contributing to enterprise architecture, across all industries and verticals.” [Read more]

Darren Cunningham, in his recent blog post How to Migrate To The Cloud, made some great points around the use of staging for data integration for cloud computing. The reasons he would leverage a staging area for cloud computing include:

• It enables better business control before the data is pushed from one system to the other.
• It enables tracking and reconciliation of a business process.
• It enables the addition of new sources or targets with reuse instead of building the spaghetti plate of point to point direct interfaces. It responds to the SOA paradigm.
• It breaks the dependencies between the two systems enabling asynchronous synchronization or synchronous with different size of data set (single message or bulk). [Read more]

According to J.J. Green, WFED's National Security Correspondent, the front line on fighting terrorism could be a simple matter of data integration that will allow agencies to better share information. He might be onto something:

“Intelligence sharing is not the problem for the US government when it comes to counter terrorism failures. A former Senior intelligence official said the problem is data integration.”

After looking at the facts of the case, I have to agree. Indeed, September 11^th and other attacks could be avoided if there was a common understanding of information between agencies. We seem to be missing patterns among the larger amounts of information collected, patterns not understood because we lack a well understood data sharing and data integration policy and practice between those who own the relevant data. [Read more]

Years ago when I taught database design at the local college I remember defining the data within a business as one of the most valuable assets a business has. I’m not sure most of my students bought that, but I still stand by that assertion and it’s much more apparent today. Indeed, today data is a huge business asset that we’re learning to finally exploit, understanding relationships and the context of other meaningful data sets, inside and outside of the enterprise.

The best example I can provide around the power of data is the number of flight tracking and estimating systems out there on the Web these days. These systems mashup the flight status data from the airlines, which are often wrong, with statistics around the make and model of the plane, the capabilities of the pilots, the weather, and other factors that will affect the early or late arrival of the flight. As a result, you can check a flight and pretty much determine the time the flight will actually arrive. This is very handy to me as a frequent flyer, and spot on most of the time. [Read more]

Lately I’ve been getting a few more head hunter calls, and many of them talk about jobs related to data integration. Trends always intrigue me and I wasn’t sure what brought this one about, so I looked into it. It seems that the Global 2000 and the US Government (The Global 1), have slowly but steadily been growing their in-house data integration skills. Thus, while cloud computing and SOA grab the headlines, pure data integration has been quietly keeping pace with that market.

According to Indeed.com which tracks job postings (see Figure), data integration job posting have increased exponentially since 2005. This is not a scientific methodology, but it’s a good metric to follow. I see similar metrics in my practice, typically in the context of big enterprise SOA and cloud computing job growth. [Read more]

Data quality issues are like back pains…everyone has to deal with them at one point or another, and in many cases it requires some major surgery. The strategic use of data is critical, and is one of those things you have to put into hard dollars before people begin to take data quality seriously. Understanding the true cost of data quality issues provides a solid foundation of understanding around the business case for putting a good data quality program and core enabling technology in place to solve this problem.

There are two costs to consider: Operational inefficiencies and long-term consequences. [Read more]

As we discussed in my last blog post, when building a SOA, data abstraction is the single most important approach and enabling technology when it comes to managing data within a SOA.

“Data abstraction is the key. It allows you to fix issues with the existing physical databases within the data service itself. Moreover, you can combine many different databases, and even unstructured information, into a single unified view of the data that is more representative of the business.” [Read more]

Those moving toward SOA seem a bit confused by the use of data within a SOA. While most consider data as…well, data, those in the know understand that data needs to be a strategic part of the SOA for SOA to succeed as a project, or as an overall architectural strategy.

The trouble comes in when attention is centered on the “S” in SOA, which stands for services. Those charged with building architectures and systems, who focus on the notion of a service as delivering functional behavior, neglect the need to manage the underlying data. In many cases, data quality and consistency issues quickly arise, and the agility that SOA should provide is limited by the need to alter services directly after the underlying data has changed. [Read more]

The rise of cloud computing naturally leads to the rising need for data integration from cloud-to-enterprise, and even cloud-to-cloud. However, most data integration solutions are on-premise. Thus, there is a clear need for cloud-delivered data integration, such as Informatica On Demand, which is an example of data integration-as-a-service. Let’s look into this a bit more.

Data integration-as-a-service is the ability to deliver a complete data integration stack from the cloud, including interfacing with applications, semantic mediation, flow control, logging, and integration design. In essence, data integration-as-a-service includes most of the features and functions found within traditional on-premise data integration technology, but it is delivered as-a-service over the Internet. [Read more]

When I wrote the EAI book years ago, I never really focused on the business case for data integration, other than looking at the money being wasted in hand coding data flows between systems. Indeed, it was in the single digit billions then, worldwide. I’m sure it’s in the double digits now, despite the fact that great data integration technology exists these days.

So, if the benefit of data integration is so obvious, why do you need to create a business case, defining the ROI to justify the cost of the effort and the technology? The fact is that most major projects require them these days, considering that IT costs are much more controlled than they were a few years ago. Indeed, a CIO friend of mine has his staff create business cases for any project over $10,000, thus they are creating a lot of business cases there. [Read more]