This is the third and last post in my myth-busting series. Myth #1 was “Apps and Data Live and Die Together“. Myth #2 was “You Only Need to Focus on the Application in Any Modernization Initiative.” Now for #3.
Myth: “Big data” is just another type of data, and IT will figure out how to get it into our enterprise applications.
Fact: There is a whole new world of data out there that is waiting to be tapped for the business—data from web activity, social media interactions, mobile devices, sensors, tags—you name it. You can monitor Facebook and Twitter to understand how customers feel about your company, and link it to the products they have purchased from you. You can utilize sensor data from manufacturing equipment to improve quality control. You can track shipments on trucks and trains to dynamically optimize logistics and scheduling. (For some good research on the business opportunity of big data, check out what the folks at McKinsey & Co. have written.)
However, to take advantage of these new business opportunities, you’re going to have to make some changes in your approach. Unless your business application was developed in the last three years or so, it probably can’t handle this type of data. Enterprise applications have traditionally been designed to handle structured relational data of a certain volume. These volumes could be very large—terabytes—but now with the advent of big data we’re seeing petabytes and exabytes of data. And a lot of the data is in hard-to-handle forms—either unstructured text from social media, or complex call detail records, or binary data from sensors and meters. The vast majority of existing applications simply can’t process this data.
A telling example is from the utilities sector. Much has been made in recent years of the shift to smart meters, which gather electric and gas meter data several times a minute and send it to the utility company. Many utility companies world-wide have already deployed smart meters to their customers, and many more are in the process of doing so. With the data from smart meters, utilities were meant to be able to do new innovative things like variable pricing and dynamic load control.
However, the dirty secret is that in many cases, all that smart meter data is going nowhere. Existing utility pricing, billing and operational systems were designed to process a single monthly meter read for each customer, not one every 15 seconds. They simply can’t handle the huge influx of data. So utilities are scrambling to implement new applications, or overhaul their existing ones, to actually achieve the promised potential of smart meters.
Processing and managing big data requires a new approach. You can’t just assume that the IT folks running your applications will be able to simply add all these new data types into the mix. To realize the potential of big data, you need to be able to extend your existing infrastructure and resources to access, parse, process and manage the data in new ways. And you want to be able to relate it to the existing data you already have in your enterprise (e.g. match Facebook comments about your product back to a specific customer in your CRM system.) The application investments you will need to make will be specific to your business, but no matter what, you will need a data infrastructure that can handle big data and integrate it with both your existing and new applications. Otherwise, the big data opportunity will pass you by, and you’ll be watching your competitors leapfrog you from the sidelines.
There will be a lot of sessions on the topic of Big Data at Informatica World– hope you can join us there to continue the conversation.