The most recent “big thing” in the integration arena was SOA. That is, it was the promised silver bullet solution that would enable efficient, fast, and seamless integration. As good as SOA is, it didn’t quite live up to all the hype and as a result leaves many of the fundamental integration challenges unresolved.
To be clear, the kinds of integration challenges I am referring to in this blog article are at the enterprise level – the “system of systems”. Integration at this level must deal with systems and data that were independently developed, are based on incompatible technologies and data models, remain independently managed, evolve at their own pace, and yet need to work together seamlessly.
SOA was not the first promised silver bullet to the enterprise integration challenge. If we go back further in time we can find several prior waves of industry movements to solve the challenges. For example, the OOD (Object Oriented Development) movement from the early 90′s introduced the idea of encapsulation and distribution of objects that were mediated through an independent broker.
CORBA, or Common Object Request Broker Architecture, was a major development and is still considered by many IT professionals as the first generation service oriented architecture. OO wasn’t the silver bullet that some expected, but it did make some essential and lasting contributions to the industry.
The next wave that appeared, around the mid-90’s, included publish/subscribe message brokers, extract-transform-load engines and process management tools to name a few. These spawned an entirely new class of system called middleware which over the past 15 years has gone through a tremendous explosion of variety, followed by a wave of consolidation, and now moving to the stage of mature broad-based integration platforms. Middleware didn’t solve all the integration issues, but this new class of technology has clearly become a permanent part of the landscape and hence is an essential enabler.
And as already mentioned, the next big silver bullet wave was SOA early in the new millennium. It had many parallels with the original OO wave, but this time with new web-based protocols and able to leverage enterprise service bus (ESB) style infrastructures. Once again a useful and essential contribution, but still leaving many of the original challenges undressed and therefore with many users back in the trough of disillusionment.
By the way, I maintain that the enterprise integration problem as we know it today didn’t exist much before 1990. It’s only been over the last 20 years or so that the complexity of application interconnectivity has reached the point where simple hand-coded point-to-point integrations were not effective solutions. Prior to then, the biggest systems that the industry built were essentially operating systems (like OS-360) which were built as massive multi-year engineering efforts.
Unfortunately, after 20 years of trying, it seems the industry has only scratched the surface of solving the problem. According to at least one industry study (1), only 32% of the respondents said they automated data and application integration with tools – the vast majority still use brute force methods such as custom hand coding or custom SQL.
The study is a bit ambiguous and depending on how you interpret the data, the amount of hand-coding could be even higher than 68%. I suppose one silver lining in this (at least from Informatica’s perspective) is that this presents a huge opportunity for growth for our business for many years to come.
The question remains, why haven’t these prior waves of technology a) solved the problem, and b) what will be the “next big thing” that we hope will be our salvation? What do you think? Please comment on this article to let me know your thoughts, and then tune in next week for my answer to the question (or subscribe to my RSS feed for automatic notification) http://blogs.informatica.com/perspectives/index.php/author/john-schmidt/feed.
(1) Forrester Research, Data Integration and Management Study, December, 2006
4 Comments
IMHO solutions to date haven’t tackled very well (or possibly at all) what i believe to be a major cause of the complexity. It seems that the IT systems reflect the way the business operates ( or has operated in the past) more or less well. Making the IT systems work together is always going to be at least as hard as making the business units work together seamlessly. I think with SOA we are now at the point where the ‘business driven’ complexity is the major factor. For the next thing to be BIG it would need to tackle this issue. Perhaps business service modelling going right back the operational strategy ? It doesn’t have to be all at once and offers a way to bite at the problem incrementally.
I would agree with Darryl. We are trying to address a business driven complexity with a technical solution. The issue is that even seemingly simple sounding terms like “Inventory” can mean different things to different departments/companies – eg. Stock on hand, usable stock only, including/excluding reserved stock etc. SOA and all the other big things that came in the past would work well if the various teams developing applications to be integrated are aligned in terms of business needs. Unfortunately this is easier said than done when different teams/vendors develop huge applications that needs to be integrated later. SOA is just a medium – like a telephone. A telephone is not helpful unless both of us speak the same language.
I would agree with both of the previous comments – it is fundamentally an issue surrounding the semantics of business integration but with the additional problem of technology change.
My answer to a) is that most of us working in IT are technologists first and information architects last (except those few who really are IAs). We know that we can ‘program’ technology to fit the semantics of integration. However, that tends to narrow the focus on the solution at hand and not on the business process.
For question b) what I would like to see in the future is an ‘integration cloud’ that allows me to define semantic data groups, a pattern of integration and business end-points without worrying about files, queues, brokers, and services. What if you could pick a business endpoint – get the list of semantic relationships it owns (say, Product, Product-Order, Product-Replenishment, etc.), pick your integration pattern (Async – Message Groups), your SLAs, pre-defined mappings and never have to create infrastructure work orders for queues, queue managers, broker instances, endpoint monitors as well as development of message schemas. These would be hidden in the cloud – yes, I dream, but hope this is the future.
Thanks all for your comments.
Darryl, if I can paraphrase your comment, the next big thing needs to address business integration needs including the non-technology dimensions (such as incentives, cross-functional investments, matrix team structures, etc.). In other words, a more formal architecture for the organization and not just IT.
Ratheesh, you reinforced the point by suggesting that much of the IT complexity is really business complexity. I know of one organization that tried to implement an enterprise rules engine so that they could quickly develop new on-line products. The problem was that each business unit had different policies, processes and structures for defining their products so they were missing a “product architecture”; in other words, a consistent way to define their product. The resultant solution ended up with a bunch of hard-coded integrations to adapt the generic rules engine into the unique rules and processes demanded by each LOB. In the end they spent a bunch of money implementing a rules engine which didn’t give them any additional agility and in fact just added to the complexity.
Don, it sounds to me like you are suggesting that IT staff need to become somewhat more “business savvy” (which I would agree with) and then you go on to make a number of interesting suggestions based on the cloud computing paradigm and a distinct semantic layer.
Based on these three comments, it seems we’re all in agreement that the NEXT BIG THING in integration needs to be holistic and engage the business much more explicitly. For my answer to the question, check out my May 11 posting. http://blogs.informatica.com/perspectives/index.php/2009/05/11/the-next-big-thing-is-lean-integration/