Integration technologies have been around for 20 years (as long as Informatica has been in business) and have proliferated in corporate IT. We are now at an inflection point in the business needs and maturity of integration best practices which we can call Next Generation Data Integration (DI). If we’re going to talk about the next generation, then first we need to put a stake in the ground to describe the current, or prior generation. Furthermore, for it to be a “generational” change, it needs to be a significant step-function improvement in how the work is done and in the business value generated by data assets. Or as Jim Collins said in Built to Last: Successful Habits of Visionary Companies, we need a Big Hairy Audacious Goal.
In our book Lean Integration, we describe four levels of integration maturity: Project, Program, Sustaining and Lean. The Next Generation DI is an orthogonal view to the maturity levels, but in general I would say that an organization needs to be at the Sustaining or Lean level of maturity to adequately leverage the next generation. So let’s start by contrasting the current/prior generation with the next generation.
|Current/Prior Generation Data Integration||Next Generation Data Integration|
|Data integration is a technical activity led by IT. When business users, frustrated by innefficiencies and lack of involvement, do their own data integration, it is often done in secret or referred to as “shadow IT.”||Data integration is a business process just like sales, marketing, order fulfillment, invoicing, etc. Data Integration encourages business user self-service, where IT is an enabler and provides technical support and validation just like for other business processes.|
|New integration points are implemented as discrete, dis-associated projects, adding to the hairball.||New integration points are implemented as routine business-as-usual (BAU) activities. Business users (e.g. analysts, administrators, data scientists, etc.) routinely create new integration points in the form of reuseable services, via self-service portals.|
|The development life-cycle methodology is largely manual. Communications, coordination and multiple handoffs between team members are via email, MS-Office tools and in-person meetings.||The development life-cycle methodology is largely automated. All relevant information about the integration is maintained in a metadata repository which is accessed by a role-based user interface (different users see a different view) and an integrated workflow system which controls the handoffs between workers and provides visibility about the end-to-end process.|
|Integration technologies are treated as tools.||Integration technologies are embedded components of a comprehensive platform and are treated as first-class business systems. Newer technologies such as data virtualization, cloud integration and browser based analytical/discovery tools enable end-user self-service capabilities.|
|Enterprise-wide data governance is non-existent or ad-hoc.||Enterprise-wide data governance is an integral aspect of how the business operates day-by-day, just like finance, accounting and HR governance are built into management systems at all levels of the organization.|
|Data being integrated is typically structured application data.||Data being integrated includes traditional transaction data as well as unstructured data, images, social media, sensor, mobile device and other big data sources.|
The Next Generation DI is not just my view. As a leading analyst firm suggests, Next Generation DI must be more agile, architected, collaborative, operational, real-time, and scalable. An independent survey by this firm indicates that most users prefer a single data integration solution that supports a rich variety of integrated tools and techniques.
Further, in a Business2Community article published at the end of last year called The Future Of Enterprise IT: 30 Executives Share Their 2013 Predictions, a number of the forecasts mentioned the increasing importance of integration – both in terms of mobile device integration, social media and location data, and integration of all the data fragmented across cloud applications. Several of them also mentioned the continued trend of business-led initiatives done outside of IT. The trend is clear partly driven by the growing adoption of cloud, the need for speed and self-service BI. Here are a few of the comments that caught my attention:
- Devanshi Garg: There is no such thing as an IT project; there are only business projects that are enabled by IT. IT professionals are becoming more attentive than ever to business and the role end-users play in the ultimate success of the software. 2013 will involve businesses setting aside more time and budget, towards intangibles around IT; communication plans, continuous user training and implementation support.
- Oliver Bussmann: The new Era is here; in order for IT organizations to stay relevant, they will have to adapt by building organizational competencies that exude speed, flexibility, security and closer ties to the business.
- Tiemo Winterkamp: The delays associated with IT will be brushed aside in favor of the speed, control, and rapid access that come along with self-service business intelligence (BI). BI users will become more self-sufficient so they can optimize and accelerate their decision making processes.
- Lou Guercia: Although customer-facing systems such as CRM are increasingly migrating to the cloud, ERP systems, housing sensitive record information, will remain mostly on-premises. Hence, nimble data integration between cloud and on-premises systems will be a key IT trend in 2013.
- Deepak Kumar: Because of the outside, consumer-based influence on enterprises today, 2013 will bring the need for lightning quick deployment cycles in the IT world, and the demand from workers for simple integrated solutions and technologies that will make their lives easier.
- Joel Bomgar: With cloud solutions that can be purchased with the swipe of a credit card, departments outside of IT will continue to select and purchase their own technology solutions with little or no involvement from IT.
The key themes from these comments (integration and business ownership/involvement) are, in my opinion, two key factors that are changing the IT landscape in 2013 and setting the stage for Next Generation DI.
The big hairy audacious idea is that the integration platform is a “business” system – not a technology tool. SAP is a business system as are PeopleSoft, Siebel, Hogan, Retek, etc. Next Generation DI is also a business system—one where the business (not IT) drives the investment, is the direct user of the system (not IT), and assumes responsibility for realizing business value (IT is the support mechanism just like IT supports SAP, Retek,…) while adhering to compliance. And like any other business system, the integration system manages the complex collaboration and workflows across a variety of business and IT stakeholders to deliver an optimal outcome.
The challenges in realizing this vision are not technology. All of the raw ingredients (i.e. agile, architected, collaborative, operational, real-time, and scalable) of the Next Generation Data Integration are available today so it is technically feasible (check out the Informatica Platform). The real challenges are cultural and behavioral based on the status-quo and pre-conceived ideas. And the way to change perceptions and behavior starts with a new vision – even a big hairy audacious one. What do you think?