Since the advent of middleware technology in the mid-1990’s, data integration has been primarily an IT-lead technical problem. Business leaders had their hands full focusing on their individual silos and were happy to delegate the complex task of integrating enterprise data and creating one version of the truth to IT. The problem is that there is now too much data that is highly fragmented across myriad internal systems, customer/supplier systems, cloud applications, mobile devices and automatic sensors. Traditional IT-lead approaches whereby a project is launched involving dozens (or hundreds) of staff to address every new opportunity are just too slow.
The good news is that data integration challenges have become so large, and the opportunities for competitive advantage from leveraging data are so compelling, that business leaders are stepping out of their silos to take charge of the enterprise integration task. This is good news because data integration is largely an agreement problem that requires business leadership; technical solutions alone can’t fully solve the problem. It also shifts the emphasis for financial justification of integration initiatives from IT cost-saving activities to revenue-generating and business process improvement initiatives.
Lean Integration is also part of the solution. A recent Forbes article by Haydn Shaughnessy talks about The Rise of Lean And Why It Matters. He goes on show how Lean is helping organizations innovate and become “elastic” and about a new book The Lean Entrepreneur which is coming out in February.
This is not a “new” phenomenon. There are indeed a number of organizations that are leading the charge and have been demonstrating for years the value of being a “data-driven enterprise” – one where their effective management and governance of data across the enterprise is used strategically to create business value and marketplace advantage. If you want an example, listen to HealthNow New York talk about next generation data integration and self-service. This has been a growing trend and I believe 2013 is at an inflection point where we will begin to see broad-based adoption of data governance programs and self-service integration solutions.
Why self service? I am reminded of a quote from a seminal paper on IT architecture Big Ball of Mud by Brian Foote and Joseph Yoder.
“Metadata allow systems to adapt more quickly to changing requirements by pushing power into the data, and out onto users.”
The direction we are headed as an industry is toward standard integration processes, automated tools, simple and intuitive user interfaces, and aggressive use of metadata. The cloud movement, and solutions like Informatica’s Cloud Data Integration platform, has helped to fuel this vision and demonstrate that it is indeed possible. The end result is to push traditional integration work into the hands of business users such that it happens at the speed of business.
Furthermore, the primary driver (i.e. the reason business leaders are getting involved) is speed. As I wrote in 2010, business leaders simply cannot wait months or years for an integrated solution – they are demanding integrated solutions at the speed of business.
Why do I believe we can create self-service data integration solutions? Because we’ve done it with other just-as-complex processes. For example, 50 years ago the order-to-cash cycle for a mail-order catalog business was measured in months. Today it is virtually instantaneous as epitomized by Amazon’s “one click” purchase process. How is it possible to reduce the order-to-cash cycle from months to seconds? By standardizing the processes, automated tools, simple and intuitive user interfaces, and aggressive use of metadata. And as I wrote in 2009, it is not that hard to make a business case for data integration – we just need the business leaders engaged and partnering with IT.
I for one am looking forward to exciting times in 2013 and beyond!
 Brian Foote and Joseph Yoder, Big Ball of Mud, Addison-Wesley Software Patterns Series, 2000