For years, organizations have been relying on strategies such as enterprise application integration to streamline and automate business processes from disparate silos and systems. However, there’s a large gaping hole in the capabilities EAI – and its successor, enterprise service buses – can deliver. Current middleware strategies fall short in addressing data integration and data quality issues – and this is costing organizations.

I recently had the opportunity to moderate an Informatica Webinar, hosted by ebizQ, featuring Dave Linthicum, CEO of the Linthicum Group and one of the world’s leading authorities on SOA, Cloud, and application and data integration, and Ash Parikh, Informatica’s SOA and data services evangelist. (And a contributor here at the Perspectives site.)

Dave, who literally wrote the book on “Enterprise Application Integration” back in the 1990s, says EAI  and ESB approaches are not suited for today’s high-transaction data integration needs, and have great limitations. But still, many organizations persist in attempting to plug in these types of solutions into vexing business problems that require a more holistic architectural approach. “There are a lot of people trying to put square pegs into round holes,” he says. “I’m seeing almost an epidemic out there in the world of service oriented architecture and enterprise architecture and architecture modernization.”
(Dave also talks about the challenges in introducing data integration into SOA efforts in a post here at the Perspectives site.)

Dave says that while the square-peg-in-a-round-hole approach will work for a while, it’s far more costly in the long run. “With EAI and ESB technology, there are certain instances and problem domains where they’re a fit,” he explains. “But you need to understand there are certain limitations that are part of that technology that should be considered. Ultimately, if you don’t consider them and pick those technology approaches anyway, you’re going to start running into walls that are very difficult to back up and get around as you move the architecture forward.”

“It’s very easy to write a check for this stuff. It’s very easy to put it on a server, and it’s easy to start coding to it, and putting adapters around it. But down the line there’s so many dependencies that are put into the technology – it’s very difficult to uncouple them out of the problem domain. After two or three or five years of inefficiencies, and issues with the technology, these approaches can cost you millions of dollars, and even cost you the success or failure of the architecture.”

Along with the issues of data integration, EAI is not equipped to manage data quality, and thus may taint the performance of SOA. “EAI will enhance overall efficiency of the business process,” Ash pointed out. “EAI will only propagate corrupt information in real time across your information systems. Sure, EAI can process messages as fast as possible across applications. But what if the information is incorrect?  Your entire process will be fed with corrupt information, such as wrong mailing address, wrong person, wrong quantity.”

Dave says that he’s finding clients “in droves” that are missing these requirements. “They typically drive with technology, so they have the technology in mind before they even look at the requirements,” he says. “Even going as far as calling something an ‘ESB project,’ which is silly. It’s a data integration project.” Data needs to be available to the enterprise, and it must be traceable and auditable, Dave says.

Data integration is beyond the reach of EAI and ESB solutions. Ash advocates putting data integration into a data services layer abstracted from physical databases. “It would be very tough for EAI to create a holistic view of data across heterogeneous data sources,” he says. “What would solve the problem is a centralized data abstraction layer.” Such an interface can “deliver a single view, regardless of the type or location of the data source.”

This data services layer in what can be loosely termed as a SOA stack, Ash observes. “Data services can be this compelling technology that sits inside this stack to enable SOA to finally realize its true potential. A single centralized layer that would eliminate all the hand coding, and solve the complexity.”

“Start with the architecture,” Dave advises. “You need to understand your own issues, and look at your own business requirements. And ultimately that’s going to lead you do to the right technology.”

“At the end of the day, there are things you need to consider, as you go through data integration architecture,” he says. “Understanding how information is going from system to system. You need to look at your requirements in detail.”