Data Exhaust: From Wasted Bits to Hot Commodity

“Data exhaust” is a cute phrase for all the stuff enterprises have been casting away since the beginning of time – and describes the potential future of enterprise intelligence.

Data ExhaustData exhaust could be log data, sensor data, and all the other unstructured stuff streaming inside, through and out of enterprises. The Internet of Things (IoT) is perhaps the most prominent source of data exhaust. Until recently, it was too expensive to get at this data. Now, with open-source tools and platforms such as Hadoop and Spark, the cost of capturing and leveraging this data has become negligible.

The value of data exhaust is increasingly being recognized, as explained by Wharton Senior Fellow Scott Snyder, and Alex Castrounis, vice president of product and advanced analytics for Rocket Wagon, an IoT, digital and AI company in a recent Knowledge@Wharton report.

Data exhaust represents a huge volume of data – perhaps exponentially larger than traditional data employed for insights. “With the rapid improvements in longer range, lower power IoT connectivity, smaller and lower cost sensors, and cloud and edge computing with artificial intelligence, more of this data exhaust not only can be analyzed for new insights, but also turned into real-time actions,” Snyder and Castrounis state. “Self-driving cars are a great example of harvesting large amounts of sensor data to learn safe and efficient driving behaviors based on each new scenario and environment.”

To successfully leverage data exhaust, Snyder and Castrounis urge the following actions:

Instrument for outcomes. Identify “the potential opportunities for deriving value from IoT data and determining the most seamless and economical means to collect the data to enable these. For example, if an athlete is already wearing cleats or a jersey, perhaps we can just embed the sensor there instead of having to create a new device and calculate the cost.”

Acquire the right skills. Snyder and Castrounis say having the right brainpower and tools to turn IoT data into insights and decisions is crucial. “This means having a team of data scientists, data engineers, and architects that can both answer known questions to optimize performance, and answer new questions by discovering hidden patterns in the data,” they state. In addition, they caution that “finding this breed of critical thinking data scientists is not easy and may require integrating employees with different skills (Python programmers, modelers, statisticians and business analysts) while building up a pipeline through other channels — like open data competitions on Kaggle or sponsoring student projects to identify new talent.”

Rethink your business model. “Fully unlocking the value of data often requires rethinking your overall offerings and business model with data and the ability to leverage it as a core advantage,” Snyder and Castrounis point out. “Given the ability of IoT to scale to incredible volumes of data around your current business operations and customer experience, companies have a unique opportunity to shift to a data-centric business model and offer their products as an on-going service.”