Data Management for a Multimodal World

TDWI_end-to-end-agility-cover
Data Management for a Multimodal World

My most recent post looked at the increasing complexity in both the data integration and metadata management spheres, and the underlying theme was data complexity . TDWI has just released a checklist report called End to End Data Management Agility, in which research director Philip Russom investigates the problems and solutions around this growing complexity.

In the report, Russom addresses these challenges in valuable detail, but his ultimate conclusion is short and sweet:

“To seize data-driven opportunities with agility, organizations need a data management architecture (or infrastructure) that is comprehensive, flexible, and highly productive.”

To some analysts, rising IT complexity, greater business demands and the variety and velocity of data demands a world of two-speed or “bimodal” IT. But I’d say our needs are actually “multimodal.” The complexity is greater than any binary split of “new/fast” and “old/slow” could address. Consider just a partial list of the many types of data today’s enterprise is struggling to manage:

  • On-premise and cloud data
  • “Regular data” and big data
  • Batch data and streaming data
  • Structured and unstructured data
  • Unconnected data and related data
  • IT-managed and self-service data
  • Batch and streaming data
  • Centrally managed and distributed data

But multimodal needs don’t mean creating a half-dozen or more divisions of IT, each moving at its own speed. It means a single data platform that is, as Russom writes, comprehensible and flexible—and able to work with any data, for any purpose, at any speed the business needs.

This is a lofty goal in a world where too often, each of those data challenges above is tackled within its own remote silo.

Islands in the Data Stream

We know the data challenges IT is facing, and they’re not either/or situations. It would be a rare organization that works only with traditional, structured data, or just with sprawling, unstructured big data. We’re all working with both extremes, and everything in between, as data gets bigger, faster, and more complex. On top of which, there are two other challenges:

  • Application complexity: The vision of a single, integrated ERP suite is giving way to a grab bag of best-of-breed cloud applications that are replacing more and more components of the integrated application suite.
  • The analytics explosion: The data warehouse and basic BI tools will be with us for some time, but organizations with more advanced analytics maturity will also employ Hadoop, NoSQL, columnar, machine learning, and predictive tools—and the list grows daily.

These trends conspire to isolate new islands of data, slowing our ability to support business and analytics initiatives. For the business, that means bad data, late data, or data it just can’t trust to drive better healthcare outcomes, improved customer loyalty (and spending), and faster and better fraud detection. Such initiatives are meant to provide a competitive advantage for the business, but to do so, they need to draw on more sources of data both inside and outside the organization—and faster than ever.

We Know What We Need

The old tactics won’t scale, aren’t reusable, are very hard to manage and change, and simply don’t provide the intelligence and automation required of an advanced data management platform. Modern business requires a different approach to data management. One that doesn’t rely on hand coding and takes advantage of automation and reuse. This means no more letting each developer use his or her tool of choice, and no more chasing individual best-of-breed tools that cannot be integrated.

Organizations know this. TDWI’s new checklist report shows that data professionals want to move to integrated data management platforms like Informatica to address the problem. While the largest group of TDWI’s respondents are today using multiple data integration tools from a variety of vendors, nearly half of them want a single, integrated tool suite, and a quarter more want “just one DI tool.” In total, 66 percent are looking to trade a best-in-class hodgepodge for a more singular solution.

Getting to Agile Data Management

The TDWI checklist is a great way to start wrapping your arms around your data challenges. It offers eight specific ways to be more agile in your data management practice. The first points to “an integrated tool platform for end-to-end data management.” The report goes on to discuss “connectivity as the golden thread” linking multiplatform hybrid data ecosystems, the steps toward delivering data in “multiple right-time speeds,” and more.

This is great reading for everybody responsible for data management, and anybody looking to better align data strategy with business needs. It will inform the challenges you’re dealing with right now, and those not quite on your horizon yet. Send it to your manager—you’ll look like a hero.

Download the TDWI Checklist Report, End to End Data Management Agility, and start building a better data strategy.

Link to Gartner MQ post.

Comments

  • Roger, you’re making good points about how increasingly complex IT portfolios for tools and platforms have led many user organizations into a whole new level of complexity. And I agree that Gartner analysts are on the right track to talk about “bimodal IT”, although the complex situation many users face today is actually “multi-modal IT.” That’s all good, but I’d like to extend the discussion by drilling into the critical success factors for multi-modal data architectures.

    I’ve talked to a lot of people about these issues. And I regularly hear from data management professionals, who say that they are succeeding with multi-modal data management — or whatever their unique term for it is. However, success is on the micro level, not so much on the macro level.

    Let’s look at micro level success. Data management professionals (especially those with data warehouse and enterprise data integration experience) are really good at deploying yet another instance of a data management database or platform, ranging from relational database types to Hadoop. And they’re good at creating new and adjusting old integration and interoperability solutions to assimilate the new instance into its complex multi-platform ecosystem. This applies to multi-platform data environments across an enterprise, include those for data warehousing, CRM/SFA, ERP, and so on.

    Now, let’s look at macro level challenges. That level is about the “big picture” data architecture and system infrastructure. With the proliferation of data platform types (not to mention related tools for analytics), technical users find it harder and harder to maintain a consistent architecture. When you lose sight of your architecture, you are more prone to problems with maintenance, optimization, scalability, governance, stewardship, and standards for data and its management.

    What can users do to get a better grip on the architecture of complex multi-platform ecosystems – or multi-modal data management? One approach is the end-to-end data management practice that Roger referenced in his blog, which is also discussed in my TDWI Checklist End-to-End Data Management Agility. Since it’s the integration infrastructure that stitches together the many platforms and tools of a complex ecosystem, a single platform consisting of multiple integrated data management functions (for integration, quality, MDM, services, metadata, etc.) can provide a fairly comprehensive view of the local data ecosystem (sometimes whole enterprises). When that platform becomes the system of record for ecosystem metadata, the scope and usefulness of the view is even greater.

    By the way, there are other approaches, too, which are less about tools and more about people and process. For example, the grand plan for an ecosystem’s data architecture and system infrastructure is more and more designed and enforced by data warehouse architects and enterprise data architects, with guidance from data stewards, governors, and technical teams.

    So folks, what do you think about multi-modal IT as it relates to data management? Are you also challenged by macro level architectures? I’d love to hear your thoughts. Cheers!

    • Roger Nolan

      Thanks for your input on this, Philip. I totally agree: Strategy, people and processes come first, then comes the choice of technology and tools that will meet the requirements that have been defined. And, it is critically important for the architects to also look beyond the immediate needs and allow for technology change and growth, particularly in the multi-modal areas discussed.
      I would be interested to hear how organizations are approaching these challenges. The issues is always speed of delivery for the current initiative versus building an architecture that will scale and grow with the organizations needs in the medium to long term.