A Big Hairy Audacious Vision

Integration technologies have been around for 20 years (as long as Informatica has been in business) and have proliferated in corporate IT. We are now at an inflection point in the business needs and maturity of integration best practices which we can call Next Generation Data Integration (DI). If we’re going to talk about the next generation, then first we need to put a stake in the ground to describe the current, or prior generation. Furthermore, for it to be a “generational” change, it needs to be a significant step-function improvement in how the work is done and in the business value generated by data assets. Or as Jim Collins said in Built to Last: Successful Habits of Visionary Companies, we need a Big Hairy Audacious Goal.

In our book Lean Integration[1], we describe four levels of integration maturity: Project, Program, Sustaining and Lean. The Next Generation DI is an orthogonal view to the maturity levels, but in general I would say that an organization needs to be at the Sustaining or Lean level of maturity to adequately leverage the next generation. So let’s start by contrasting the current/prior generation with the next generation.

Current/Prior Generation Data Integration Next Generation Data Integration
Data integration is a technical activity led by IT. When business users, frustrated by innefficiencies and lack of involvement, do their own data integration, it is often done in secret or referred to as “shadow IT.” Data integration is a business process just like sales, marketing, order fulfillment, invoicing, etc. Data Integration encourages business user self-service, where IT is an enabler and provides technical support and validation just like for other business processes.
New integration points are implemented as discrete, dis-associated projects, adding to the hairball. New integration points are implemented as routine business-as-usual (BAU) activities. Business users (e.g. analysts, administrators, data scientists, etc.) routinely create new integration points in the form of reuseable services, via self-service portals.
The development life-cycle methodology is largely manual. Communications, coordination and multiple handoffs between team members are via email, MS-Office tools and in-person meetings. The development life-cycle methodology is largely automated. All relevant information about the integration is maintained in a metadata repository which is accessed by a role-based user interface (different users see a different view) and an integrated workflow system which controls the handoffs between workers and provides visibility about the end-to-end process.
Integration technologies are treated as tools. Integration technologies are embedded components of a comprehensive platform and are treated as first-class business systems. Newer technologies such as data virtualization, cloud integration and browser based analytical/discovery tools enable end-user self-service capabilities.
Enterprise-wide data governance is non-existent or ad-hoc. Enterprise-wide data governance is an integral aspect of how the business operates day-by-day, just like finance, accounting and HR governance are built into management systems at all levels of the organization.
Data being integrated is typically structured application data. Data being integrated includes traditional transaction data as well as unstructured data, images, social media, sensor, mobile device and other big data sources.

The Next Generation DI is not just my view. As a leading analyst firm suggests, Next Generation DI must be more agile, architected, collaborative, operational, real-time, and scalable. An independent survey by this firm indicates that most users prefer a single data integration solution that supports a rich variety of integrated tools and techniques.

Further, in a Business2Community article published at the end of last year called The Future Of Enterprise IT: 30 Executives Share Their 2013 Predictions, a number of the forecasts mentioned the increasing importance of integration – both in terms of mobile device integration, social media and location data, and integration of all the data fragmented across cloud applications. Several of them also mentioned the continued trend of business-led initiatives done outside of IT. The trend is clear partly driven by the growing adoption of cloud, the need for speed and self-service BI.  Here are a few of the comments that caught my attention:

  • Devanshi Garg: There is no such thing as an IT project; there are only business projects that are enabled by IT. IT professionals are becoming more attentive than ever to business and the role end-users play in the ultimate success of the software. 2013 will involve businesses setting aside more time and budget, towards intangibles around IT; communication plans, continuous user training and implementation support.
  • Oliver Bussmann: The new Era is here; in order for IT organizations to stay relevant, they will have to adapt by building organizational competencies that exude speed, flexibility, security and closer ties to the business.
  • Tiemo Winterkamp: The delays associated with IT will be brushed aside in favor of the speed, control, and rapid access that come along with self-service business intelligence (BI). BI users will become more self-sufficient so they can optimize and accelerate their decision making processes.
  • Lou Guercia: Although customer-facing systems such as CRM are increasingly migrating to the cloud, ERP systems, housing sensitive record information, will remain mostly on-premises. Hence, nimble data integration between cloud and on-premises systems will be a key IT trend in 2013.
  • Deepak Kumar: Because of the outside, consumer-based influence on enterprises today, 2013 will bring the need for lightning quick deployment cycles in the IT world, and the demand from workers for simple integrated solutions and technologies that will make their lives easier.
  • Joel Bomgar: With cloud solutions that can be purchased with the swipe of a credit card, departments outside of IT will continue to select and purchase their own technology solutions with little or no involvement from IT.

The key themes from these comments (integration and business ownership/involvement) are, in my opinion, two key factors that are changing the IT landscape in 2013 and setting the stage for Next Generation DI.

The big hairy audacious idea is that the integration platform is a “business” system – not a technology tool. SAP is a business system as are PeopleSoft, Siebel, Hogan, Retek, etc. Next Generation DI is also a business system—one where the business (not IT) drives the investment, is the direct user of the system (not IT), and assumes responsibility for realizing business value (IT is the support mechanism just like IT supports SAP, Retek,…) while adhering to compliance. And like any other business system, the integration system manages the complex collaboration and workflows across a variety of business and IT stakeholders to deliver an optimal outcome.

The challenges in realizing this vision are not technology. All of the raw ingredients (i.e. agile, architected, collaborative, operational, real-time, and scalable) of the Next Generation Data Integration are available today so it is technically feasible (check out the Informatica Platform). The real challenges are cultural and behavioral based on the status-quo and pre-conceived ideas. And the way to change perceptions and behavior starts with a new vision – even a big hairy audacious one. What do you think?

[1] John G. Schmidt and David Lyle, Lean Integration, An Integration Factory Approach to Business Agility, 2010, Addison-Wesley
This entry was posted in Big Data, Business Impact / Benefits, Business/IT Collaboration, CIO, Data Governance, Data Integration Platform, Enterprise Data Management, Integration Competency Centers and tagged , , , , , , , . Bookmark the permalink.

9 Responses to A Big Hairy Audacious Vision

  1. Brad Sheridan says:

    Great perspective John…thanks for sharing. To take it a step further, DI is merely one component of an overall EIM strategy to include (DG, MDM, DI, etc…)…just imagine the cultural and behavior challenges of this larger, overall goal!!! Mind boggling. Start small and grow it. :)

  2. Hi John… I am looking forward to wherever NGDI goes. It is always a pleasure to keep in touch with other thought leaders.

    Reading the comments above leads me to think that the whole world is going self-service with a swipe of the credit card in a cloud-based solution, and I don’t disagree with it except in a few cases.

    I’ve been in banking and insurance for 20+ years, and I don’t agree that banks, insurance companies, and medical institutions will be moving their customer data to the cloud. Under the watchful eye of OCC or HIPAA regulations, they would be abdicating their responsibility to protect customer information and hoping their cloud-provided security is good enough. Generally okay to use it for Sales information (SalesForce.com), so long as it is not the master customer database.

    A few years ago, the Sony Playstation network was hacked and the thieves got away with 12,000 credit card numbers and information for 77 million users. The first lawsuit arrived less than 2 weeks later saying Sony did not take reasonable precautions to protect the customer data. It cost them millions to deal with the problem. If I was a president of a bank, then in my opinion, I would keep my customer data in my data center and not dare to move it to the cloud because if it gets stolen, the bank is still responsible no matter where it resides.

    Other than banks, insurance companies, and medical institutions, I do believe there is a gigantic movement towards cloud solutions. All of my consumer-oriented friends seem to be storing their data in the cloud. They store their photographs and family tree info, but I am personally reluctant to put my customer trade secret information in the cloud because I am not comfortable with how it can be protected. I can be sued for breach of confidentiality if it gets into the wrong hands.

    I also think that cloud based web services are a big part of the current revolution. Speed to market and rapid development will be achieved by piecing together services and components that are all over the place.

  3. John Schmidt says:

    Jeff, thanks for your comment. I was at a bank a few years ago that had a “no cloud” policy, until one day IT discovered that there were over 1,000 business users with SalesForce.com accounts. It may be that core customer, deposit and lending systems in banks never move to the cloud for the reasons you suggest (this could be the topic of another blog article), but the cloud has provided several foundations for NGDI. First, it has demonstrated that browser-based self-service solutions really can work. Second, it has demonstrated an IT engagement/delivery model that breaks the “project” paradigm. Banks, insurance, and healthcare organizations may never fully embrace the public cloud, but whether they call a private cloud or give it some other label, business-led self-service integration is here to stay.

  4. Very much an evolutionary approach compare to a revolutionary approach. I think most of us are somewhere between first gen and Next gen a the moment. Good to see that a lot of thinking is going into this subject, but to go all the way there must be a way to train / teach company management the advantages of moving all the way. While that process is still not there, I guess all will just slowly migrate from first to next …. as with all previous waves.

  5. Jerry W. Jetelina says:

    No matter how you look at it, the 21st Century DI systems start with the intellectual organization of informations and design of the RDB in the 6th, 7th and 8th NF.

    Well it looks that I am cursing.



    good article John, Earlier there is no cloud policy but now cloud has providing several foundations for NGDI. so Banks, Insurance and Medical Institutions will move to NGDI.

  7. Marina Kerbel says:

    Very interesting blog and comments.

    It may be just a matter of time until all kinds of data will be in the cloud. Somebody said that it’s like money in the banks – it took awhile for people to trust banks, but we do not keep money under our mattresses anymore.

  8. Marina Kerbel says:

    Some of the listed capabilities look very futuristic, but others may be already happening. Do you know to what degree different companies are already using the listed capabilities?

  9. John Schmidt says:

    Marina, good question. I have two answers. If you look at the individual aspects of NGDI, roughly 80% of large corporations are doing at least one of the capabilities that make up the next generation. Maybe they have an ICC that has standardized and automated some aspects of the development cycle. Maybe they have started to use some level of self-service cloud integration. Maybe they have a data governance program that leverages a metadata management capability. Maybe they are doing something with big data or with data virtualization. The point is that I can find examples somewhere in the real world where every single one, 100%, of the NGDI capabilities is in existence.

    But if we ask the question “who is doing all of these things with a reasonable level of maturity in the same organization?” The answer is probably in the low single digit percentage range. We should turn this around and ask the readers of this blog. Do you know of any organizations that are at, or very close to, the Next Generation Data Integration level of competence?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>