Internet of Things Means Plenty of Work on the Way for Data Gurus

Internet of Things Means Plenty of Work

The so-called Internet of Things is all about data, all of the time. While there’s plenty of talk about of the wondrous applications and business potential with the ability to monitor, in real time, the location and state of machinery, devices, sensors, and the people associated with them, that means a lot of work on the back end for data specialists, who will be charged with identifying and validating the information that is moving through the enterprise.

That’s the prediction by experts who have been watching the IoT rumble through the digital economy. My colleague Mike Kavis, who has a lot of experience in enterprise and data architecture, cautions that the IoT so-called Internet of Things requires a lot of work on the back end.

There are three areas that need to be concentrated on in seeing IoT through, Mike points out: data ingestion (harvesting data), data storage, and analytics. Analytics is where IoT data delivers value to the business; whereas data ingestion and data storage are back-end necessities to make it all work. However, it’s in these back-end processes that IoT may get tripped up. “Experts estimate that over half of all big data projects fail and most of those failures are due to projects never getting past the data ingestion phase,” Mike says. There are a range of technical solutions available to manage the storage side of things – such as Hadoop – but this requires an abundance of skills to manage.

I’m going to add a fourth set of challenges to making the most out of the IoT opportunity – data integration. Namely, being able to pull data out of silos and bring together IoT equipment and technologies – which currently take different forms, vendor by vendor. This challenge was underscored in a recent report published by James Manyika and a team of McKinsey consultants from the McKinsey Global Institute. The challenge – and potential for economic gain – is seen in the complexity and interconnectedness of worksite equipment. “The effort required to capture the added benefits available from interoperability is not trivial,” Manyika and his co-authors point out. “It requires integration across multiple systems and vendors, sometimes across different industries.”

Often, equipment, sensors, software and people are being brought together from different industries and platforms. Such lack of standards may be reducing the potential value of IoT efforts by at least 40 percent, the McKinsey report estimates. There is a great deal of complexity that needs to be overcome at every level. “For example, there can be 30,000 sensors on an offshore oil rig,” Manyika and his co-authors state. Most of this data is not used. “Only one percent of the data are examined. That’s because this information is used mostly to detect and control anomalies—not for optimization and prediction, which provide the greatest value.” Greater interoperability between these systems would “provide decision makers with an integrated view of performance across an entire factory or oil rig.”

Achieving integration with existing data management and analytics systems, such as data warehouses, is also an area that needs to be addressed for IoT to deliver its full potential value.

“This can be extremely challenging when the underlying database technologies of the data warehouse are different than what is used for the IoT data,” Mike states. Add to that the costs and effort associated with maintaining and provisioning “enough infrastructure to keep up with the incoming flow of data” – “an arduous task that continues to keep risks high throughout the life of the IoT investment.”

Add to this challenge the fact that “it is also highly likely that the demand for real time analytics, coupled with storing many petabytes of data require different server, disk, and network infrastructure than what exists in most data centers today,” Mike says. This will lead to “even larger infrastructure costs and the consumption of additional floor space,” he adds.

There are ways to overcome some aspects of these challenges, Mike continues. “The first and most popular model is to leverage one of the many database as a service (DBaaS) offerings in the market place,” he suggests, noting that such services free companies from having to “install, manage, and operate the underlying technologies required to make large NoSQL databases scale. DBaaS solutions abstract away much of the underlying complexities so that engineers can focus on the data as opposed to the collection of technologies that make up the underlying database.”

The second option is to leverage managed service providers, which typically “own the responsibility of data ingestion and database management as well as provide capabilities for performing analytics and extracting datasets,” without large upfront investments.

These approaches are not without their challenges, of course, the main one being that it increases reliance on single DBaaS technologies. At the same time, it frees organizations of the headaches involved in attempting to assemble actionable information for decision makers. Until then, much of the data that is captured and stored by enterprises may remain sored, but unused.

The data emanating from manufacturing automation systems on factory floors also tends to not end up anywhere, instead being limited to real-time control or anomaly detection. While an increasing number of machines are “wired,” this instrumentation “is used primarily to control the tools or to send alarms when it detects something out of tolerance,” the McKinsey team states. “The data from these tools are often not analyzed, or even collected in a place where they could be analyzed, even though the data could be used to optimize processes and head off disruptions.”

The McKinsey report also calls for both standardization across industry systems, as well as a wider array of middleware that can provide for brokering and translating services. “The development and adoption of IoT standards is still in its early stages,” they point out.

The IoT economy promises great potential for today’s enterprises. At the same time, it promises great employment and career potential to those data specialists and technology leaders who think outside their industries, since all industries are becoming interconnected.