Data Management for a Multimodal World
My most recent post looked at the increasing complexity in both the data integration and metadata management spheres, and the underlying theme was data complexity . TDWI has just released a checklist report called End to End Data Management Agility, in which research director Philip Russom investigates the problems and solutions around this growing complexity.
In the report, Russom addresses these challenges in valuable detail, but his ultimate conclusion is short and sweet:
“To seize data-driven opportunities with agility, organizations need a data management architecture (or infrastructure) that is comprehensive, flexible, and highly productive.”
To some analysts, rising IT complexity, greater business demands and the variety and velocity of data demands a world of two-speed or “bimodal” IT. But I’d say our needs are actually “multimodal.” The complexity is greater than any binary split of “new/fast” and “old/slow” could address. Consider just a partial list of the many types of data today’s enterprise is struggling to manage:
- On-premise and cloud data
- “Regular data” and big data
- Batch data and streaming data
- Structured and unstructured data
- Unconnected data and related data
- IT-managed and self-service data
- Batch and streaming data
- Centrally managed and distributed data
But multimodal needs don’t mean creating a half-dozen or more divisions of IT, each moving at its own speed. It means a single data platform that is, as Russom writes, comprehensible and flexible—and able to work with any data, for any purpose, at any speed the business needs.
This is a lofty goal in a world where too often, each of those data challenges above is tackled within its own remote silo.
Islands in the Data Stream
We know the data challenges IT is facing, and they’re not either/or situations. It would be a rare organization that works only with traditional, structured data, or just with sprawling, unstructured big data. We’re all working with both extremes, and everything in between, as data gets bigger, faster, and more complex. On top of which, there are two other challenges:
- Application complexity: The vision of a single, integrated ERP suite is giving way to a grab bag of best-of-breed cloud applications that are replacing more and more components of the integrated application suite.
- The analytics explosion: The data warehouse and basic BI tools will be with us for some time, but organizations with more advanced analytics maturity will also employ Hadoop, NoSQL, columnar, machine learning, and predictive tools—and the list grows daily.
These trends conspire to isolate new islands of data, slowing our ability to support business and analytics initiatives. For the business, that means bad data, late data, or data it just can’t trust to drive better healthcare outcomes, improved customer loyalty (and spending), and faster and better fraud detection. Such initiatives are meant to provide a competitive advantage for the business, but to do so, they need to draw on more sources of data both inside and outside the organization—and faster than ever.
We Know What We Need
The old tactics won’t scale, aren’t reusable, are very hard to manage and change, and simply don’t provide the intelligence and automation required of an advanced data management platform. Modern business requires a different approach to data management. One that doesn’t rely on hand coding and takes advantage of automation and reuse. This means no more letting each developer use his or her tool of choice, and no more chasing individual best-of-breed tools that cannot be integrated.
Organizations know this. TDWI’s new checklist report shows that data professionals want to move to integrated data management platforms like Informatica to address the problem. While the largest group of TDWI’s respondents are today using multiple data integration tools from a variety of vendors, nearly half of them want a single, integrated tool suite, and a quarter more want “just one DI tool.” In total, 66 percent are looking to trade a best-in-class hodgepodge for a more singular solution.
Getting to Agile Data Management
The TDWI checklist is a great way to start wrapping your arms around your data challenges. It offers eight specific ways to be more agile in your data management practice. The first points to “an integrated tool platform for end-to-end data management.” The report goes on to discuss “connectivity as the golden thread” linking multiplatform hybrid data ecosystems, the steps toward delivering data in “multiple right-time speeds,” and more.
This is great reading for everybody responsible for data management, and anybody looking to better align data strategy with business needs. It will inform the challenges you’re dealing with right now, and those not quite on your horizon yet. Send it to your manager—you’ll look like a hero.
Download the TDWI Checklist Report, End to End Data Management Agility, and start building a better data strategy.
Link to Gartner MQ post.