Internet of Things (IoT) Changes the Data Integration Game in 2015

Data Integration
Internet of Things (IoT) Changes the Data Integration Game in 2015

As reported by the Economic Times, “In the coming years, enormous volumes of machine-generated data from the Internet of Things (IoT) will emerge. If exploited properly, this data – often dubbed machine or sensor data, and often seen as the next evolution in Big Data – can fuel a wide range of data-driven business process improvements across numerous industries.”

We can all see this happening in our personal lives.  Our thermostats are connected now, our cars have been for years, even my toothbrush has a Bluetooth connection with my phone.  On the industrial sides, devices have also been connected for years, tossing off megabytes of data per day that have been typically used for monitoring, with the data tossed away as quickly as it appears.

So, what changed?  With the advent of big data, cheap cloud, and on-premise storage, we now have the ability to store machine or sensor data spinning out of industrial machines, airliners, health diagnostic devices, etc., and leverage that data for new and valuable uses.

For example, the ability determine the likelihood that a jet engine will fail, based upon the sensor data gathered, and how that data compared with existing known patterns of failure.  Instead of getting an engine failure light on the flight deck, the pilots can see that the engine has a 20 percent likelihood of failure, and get the engine serviced before it fails completely.

The problem with all of this very cool stuff is that we need to once again rethink data integration.  Indeed, if the data can’t get from the machine sensors to a persistent data store for analysis, then none of this has a chance of working.

That’s why those who are moving to IoT-based systems need to do two things.  First, they must create a strategy for extracting data from devices, such as industrial robots or ann  Audi A8.  Second, they need a strategy to take  all of this disparate data that’s firing out of devices at megabytes per second, and put it where it needs to go, and in the right native structure (or in an unstructured data lake), so it can be leveraged in useful ways, and in real time.

The challenge is that machines and devices are not traditional IT systems.  I’ve built connectors for industrial applications in my career.  The fact is, you need to adapt to the way that the machines and devices produce data, and not the other way around.  Data integration technology needs to adapt as well, making sure that it can deal with streaming and unstructured data, including many instances where the data needs to be processed in flight as it moves from the device, to the database.

This becomes a huge opportunity for data integration providers who understand the special needs of IoT, as well as the technology that those who build IoT-based systems can leverage.  However, the larger value is for those businesses that learn how to leverage IoT to provide better services to their customers by offering insights that have previously been impossible.  Be it jet engine reliability, the fuel efficiency of my car, or feedback to my physician from sensors on my body, this is game changing stuff.  At the heart of its ability to succeed is the ability to move data from place-to-place.


  • Excellent article David! It’s time for more thought leaders with real IoT expertise to step forward and lead the architectural discussion. It’s not just another hashtag, IoT is the next big challenge for digital society. Look forward to more posts on this from you and Informatica!

  • In IoT context, the premise of BigData is to shrink the data. We need data refineries sitting very close to data exhaust so that wire loads can be made manageable. To make the best out of data, it has to sit next to compute and that is the crux of the problem (how do we reduce proximity of data from compute). As you mentioned, both data and processing sides have to make some adjustments.

  • Aditya

    Very Futuristic article.
    I think we are 20 years away right now to implement this level of integration.

    Integrating appliances would be very helpful.also it would help us track the perforance of our appliancs.

  • Jim O’Reilly

    IoT data will be like the Amazon – Many streams will form a huge river.
    The key is to pre-process data near to the source, so that the streams coming into the data center shrink to manageable levels.

  • Vinod Damodaran

    The IoT revolution has begun with early adopters already tinkering with it. For it to mature and for mass adoption to take place will take some time, but I think it will be more in the 5 – 10 years range (not 20 years).
    The sensors that are generating data will become a lot smarter, with chip manufacturers putting some of the code (eg: security, data aggregation, pattern matching etc) into the chips, so that most of the real time alerts / actions can be taken by the sensors. These smart sensors will also become programmable / upgradable, so at the beginning of the IoT revolution, the hardware companies will see the boom ahead of the others.
    The enormous data generated will also be passed onto the data integration frameworks to be stored into the hadoop / bigdata cloud, where machine learning / analytics will be applied to discover / fine tune more patterns / intelligence. So as David mentioned, every sector of the data value chain will have to make improvements for the IoT revolution.