Big Data in Automotive Accelerates from 0-60 in 3 seconds with Big Data Management

In preparing for a Big Data presentation to one of the top 3 global automotive manufacturers I discovered many interesting benefits that Big Data can provide through connected vehicle and IoT programs.  By collecting and analyzing big data from sensors, machines, databases, and social media, automakers can increase customer loyalty, improve predictive maintenance, reduce warranty costs, improve dealer performance, improve vehicle performance, optimize supply chains, improve driver safety, and even potentially monetize vehicle data for third parties such as insurance companies (e.g. Pay As You Drive, Pay How You Drive).  The potential benefits of Big Data in the automotive industry seem almost endless.

However, in order to realize these benefits a number of important decisions need to be made in terms of architecture and technology investments.  This particular automotive manufacturer started off by hand-coding a lot of the components necessary for a big data platform.  Their goal was to provide a central data lake of vehicle related data for 360 customer analytics so they could answer questions about a consumers’ propensity to buy a new vehicle, which one, and when or whether there is a propensity for one car to have the same issue as another car.  They were also looking at a number of “sexy” startups as point solutions for data blending, data governance, and mastering data.  I explained that tackling Big Data with hand-coding and point solutions is a recipe for disaster and that you need to establish a flexible architecture that delivers scalable, repeatable, consistent, and trusted information across the enterprise.  The reasons which even their chief architect admitted to was that a big data platform that is stitched together becomes unwieldy, difficult, and expensive to maintain over time.

I then suggested they consolidate and rationalize the technologies in their Big Data platform into three layers:  A scalable persistent storage and processing layer, a big data management layer, and an analytic/visualization layer.  While there has been a lot of market hype around Hadoop and analytics that form the bottom and top layers, there has been less attention drawn to the middle layer of the Big Data technology stack which is Big Data Management.  Big Data Management forms the foundation for successful Big Data projects with three pillars:  Big Data Integration, Big Data Governance, and Big Data Security.  So my advice to all of you readers is step back a moment from all the Big Data hype and consider the requirements necessary to be successful for Big Data projects.  You basically need to store, process, integrate, govern, secure, and analyze the data at scale.  Therefore, invest in platform technologies that are designed to work together.  To learn more about Big Data Management I encourage you to read the white paper, “How Big Data Management Turns Petabytes into Profits”.