David Linthicum
Data Integration Moves Up The Food Chain
There was an interesting discussion out on the SOA Data Integration LinkedIn Group around the creation of a new role that focuses on data integration. Whether you call them Chief Integration Officers, or Data Integration Leaders, the idea is that data integration is more strategic to enterprises these days, and it needs focus at the highest levels in the company.
The key reasons for a reemergence of interest in data integration these days include:
The volume of the data. The amount of data that has to move from place to place within enterprises has grown a great deal in recent years, typically around the number of critical systems that exist, compliance reporting, and the cloud. The movement of data from place to place is easy. However, the ability to assure that the data arrives at its destination in a clean and useful state is difficult. Thus data integration has to provide both the role of transport and data assurance. The larger the data set, the more data integration becomes a key enabling technology. (more…)
More Selling Points Of Data Services and Data Abstraction
As we discussed in my last blog post, when building a SOA, data abstraction is the single most important approach and enabling technology when it comes to managing data within a SOA.
“Data abstraction is the key. It allows you to fix issues with the existing physical databases within the data service itself. Moreover, you can combine many different databases, and even unstructured information, into a single unified view of the data that is more representative of the business.”
Let’s walk further down this road. When using the approach of data abstraction, and data abstraction using data services, you’re able to emulate the desired data and data structure without having to recreate and restructure the physical databases, nor having to statically bind the services to the physical data. The core value of this is agility, but there are many other advantages as well, including: (more…)
If You Have A Data Integration Round Hole, Don’t Buy Square Pegs
I’m often taken aback by the focus on technology as “the solution,” and not as an approach to the solution. One of the reasons I blog for Informatica is because they do focus on the solution and not the tool. Believe me, vendors will sell plenty of tools if they provide those tools in the context of the solution.
So, how do you find the right data integration tool? It’s really a matter of understanding your own data integration requirements, and creating baseline models for what the existing “as is” state is. This means you must understand your data at the structure and model levels, or, more simply put, you must understand what you have, where you have it, and what it’s doing. (more…)
Rethinking Data Virtualization
I’m looking forward to doing a Webinar on data virtualization this Thursday, April 22nd. Why? Because this is the single most beneficial concept of architecture, including SOA, and it’s often overlooked by the rank-and-file developers and architects out there. I’m constantly evangelizing the benefits of data virtualization, including integrating data from many and different data sources in real-time, and enabling query-based applications to get data from multiple systems.
The idea is pretty simple, really. Considering that there are many physical database schemas within most enterprises, and typically no common view of the data, data virtualization allows you to map many physical schemas to virtual schemas that are a better representation of the business. For example, a single view of customer data, sales data, and other data that has the same logical meaning, but may be scattered amongst many different physical database systems, using any number of implementation models. (more…)
Best Practices In Data Abstraction
Data abstraction is a core mechanism required for SOA to be successful. The idea behind data abstraction is that you can address some very dysfunctional and federated database schemas and tame them into something that’s much more usable by all applications and services. Thus you have the ability to address data as something that is related to the core business entities, and not data that’s scattered throughout the enterprise. For example, how many places are you storing customer data?
While the benefit of data abstraction is clear, the approaches you take to drive to data abstraction are also important to achieving the desired results. Indeed, you need to consider existing database structures and the target virtual schemas, as well as governance, security, and performance. So, what are the best practices? Let’s look at three of the most popular. (more…)
The Achilles Heel Of Cloud Computing – Data Integration
Loraine Lawson did a great job covering the topic of the integration challenges around the cloud and virtualization. She reports that “…a recent Internet Evolution column [by David Vellante] looks more broadly at the cloud integration question and concludes that insufficient integration is holding up both cloud computing and virtualization.”
In fact, what currently limits the number of cloud deployments is the lack of a clear understanding of data integration in the context of cloud computing. This is a rather easy problem to solve, but it’s often an afterthought.
The core issue is that cloud computing providers, other than Salesforce.com, don’t consider integration. Perhaps they are thinking, “If you use our cloud, then there is no reason to sync your data back to your enterprise. After all, we’re the final destination for your enterprise data, right?” Wrong. (more…)
The New SOA Data Integration Architect Community
I was asked to participate in the new “SOA Data Integration Architect Community,” or SDIAC, which launches today. As most of you know, I don’t like to work with vendor-focused standards bodies or on-line communities, but I found out quickly that this was very different.
“The SOA Data Integration Architect Community is the world’s first open and free community that focuses on the value of data integration and data services in agile architectures such as SOA. This one-of-a-kind community will bring together software professionals who will contribute to and be involved in software architecture, to raise awareness about the foundational value of data in enterprise architecture. This includes enterprise architects, application architects, data architects, solution architects, IT architects and IT managers responsible for or contributing to enterprise architecture, across all industries and verticals.” (more…)
Making The Case For Staging And Cloud Computing Integration
Darren Cunningham, in his recent blog post How to Migrate To The Cloud, made some great points around the use of staging for data integration for cloud computing. The reasons he would leverage a staging area for cloud computing include:
- It enables better business control before the data is pushed from one system to the other.
- It enables tracking and reconciliation of a business process.
- It enables the addition of new sources or targets with reuse instead of building the spaghetti plate of point to point direct interfaces. It responds to the SOA paradigm.
- It breaks the dependencies between the two systems enabling asynchronous synchronization or synchronous with different size of data set (single message or bulk). (more…)
Is Fighting Terrorism A Data Integration Problem?
According to J.J. Green, WFED’s National Security Correspondent, the front line on fighting terrorism could be a simple matter of data integration that will allow agencies to better share information. He might be onto something:
“Intelligence sharing is not the problem for the US government when it comes to counter terrorism failures. A former Senior intelligence official said the problem is data integration.”
After looking at the facts of the case, I have to agree. Indeed, September 11th and other attacks could be avoided if there was a common understanding of information between agencies. We seem to be missing patterns among the larger amounts of information collected, patterns not understood because we lack a well understood data sharing and data integration policy and practice between those who own the relevant data. (more…)
2010 Will Be The Year Of The Power Of Data
Years ago when I taught database design at the local college I remember defining the data within a business as one of the most valuable assets a business has. I’m not sure most of my students bought that, but I still stand by that assertion and it’s much more apparent today. Indeed, today data is a huge business asset that we’re learning to finally exploit, understanding relationships and the context of other meaningful data sets, inside and outside of the enterprise.
The best example I can provide around the power of data is the number of flight tracking and estimating systems out there on the Web these days. These systems mashup the flight status data from the airlines, which are often wrong, with statistics around the make and model of the plane, the capabilities of the pilots, the weather, and other factors that will affect the early or late arrival of the flight. As a result, you can check a flight and pretty much determine the time the flight will actually arrive. This is very handy to me as a frequent flyer, and spot on most of the time. (more…)
Loading ...