Data and Analytics: Report Card and Future Direction

I have just read a new report from McKinsey, The Age of Analytics: Competing in a Data-Driven World (Dec 2016).  It weighs in at a mammoth 120+ pages, but there are some very interesting nuggets in here.

What Has Been Achieved with Data and Analytics in Five Years?

Data and AnalyticsFirst, McKinsey went back to a previous study they did on data and analytics in 2011 and compared the economic value that they estimated could be realized by the use of data and analytics with the actual results.  To get to the point, the results to date are not exactly awe-inspiring.
Top 5 Opportunities Identified & Results 5 years later

Opportunity Value Captured by 2016
Location based data 50-60%
US Retail 30-40%
Manufacturing 20-30%
EU Public Sector 10-20%
US Health Care 10-20%


Let’s not forget that data and analytics have been a top priority for many organizations for 5 years now according to many industry analysts. So, why did organizations fail to realize the potential in these areas?

Reasons given in the McKinsey report are:

  1. Data and Analytics Strategy: What will data an analytics be use for? How will insights drive value?
  2. Building a Data Architecture: The architecture should include data management, data collection, data generation.
  3. Acquiring Analytics Capabilities: These include skilled people, processes and technology, including a potentially huge shortage of available data scientists.

#1, an Analytics Strategy, is obviously critical, but there are a lot of “science experiment” types of analytics projects out there that involve a lot of very smart people doing a lot of hand coding, but all too often with little connection with the business strategy and objectives and little thought toward how the insights delivered will be automated and operationalized

#2 is obviously my favorite topic. Organizations will need to build a distinct competence in data management to solve the data challenges while at the same time solving the problem of rapid technology change in analytics and applications.  The data warehouse is a useful tool and a great place to build experience in data management and analytics, but it is rapidly becoming table stakes in the race to different products and customers experiences with data.
Top Challenges in the Pursuit of Data and Analytics Objectives
These were the top three challenges found by McKinsey. There are no surprises here. This is very consistent with other surveys they have done.

  • 45%. Designing an appropriate organizational structure to support data and analytics objectives
  • 42%. Ensuring senior management involvement.
  • 36%. Designing effective data architecture and technology infrastructure.

We know that there are new sources of data emerging every day from data aggregators, data sets published by governments, social media, and the increasing instrumentation of everything from mobile phones to buildings, cars, and sports equipment.  The balance has quickly shifted from internal data (such as what we know about our customers or patients) to external data that can be combined with that internal data (such as genomic information, location based data, sentiment and actions outside of the sphere of your organization).  The pharmaceutical business has experienced a complete data inversion in the past 10 years, with the vast majority of data being internal and proprietary to the vast majority of data coming from external research, clinical studies, and data aggregators.  Many organizations are still struggling with that change.

What Is the Potential for Data and Analytics to Provide Disruption?

It’s time to disrupt or be disrupted.  The McKinsey report goes on to talk about future models for disruption using data and analytics.  Here is my take on some of the data management capabilities that would be required for each of the disruptive model models that McKinsey identified.


Future Models for Disruption  Example Data Management Capabilities Required (not a complete list)
Business models enabled by orthogonal data


  • Connect, ingest, transform data.
  • Then, match that data with existing internal data.
Hyperscale, real time matching
  • Filter and match data in real time.
  • Then, apply a rule based on the actual data.
  • Present data on match and other relevant context.
Radical personalization


  • Rapid matching of personal data with orthogonal data, real time events and activities.
  • Rapidly match preferences, events and activities with an actual customer, patient, or voter.
  • May also require relationship management to understand social graphs, hierarchies, family structures.
Massive data integration capabilities


  • Fast ingestion of batch or streaming data.
  • Fast integration of never-before-seen data.
  • You may be looking at a data lake to hold large data sets from multiple internal and external sources.
  • After you identify the data that is valuable you will want to ensure the quality of the data and ensure that it has business meaning and context.
  • Once ingested, you will need to integrate and match this data with data from all of your internal data silos in order to deliver insights that drive value.
Data-driven discovery and innovation


  • Here we are using data for optimizing, predicting and recommending.
  • All of these efforts benefit from a wide range of data from disparate sources.
  • The particular challenge of discovery and innovation is enabling self-service so that teams can innovate rapidly, fail, and iterate until they find something useful.
  • The other challenge is to provide very timely but “good enough” data to enable this effort. It probably does not have to be perfect data for innovation.
Enhanced decision making
  • Early work in this area focused on operational or process optimization, integrating and combing data from multiple data sources never matched before.
  • Enabling fast decisions will involve real time data management and matching.
  • Future-focused decisions may involve prediction which will depend on just the right types of trustworthy data, from a wide variety of sources, delivered in a very timely manner.

Machine Learning Emerges

The report also goes into a good bit on the growth of machine learning and deep learning.  The initial focus here will be on providing very large “experience” data sets to “train” rather than program the algorithm.  But, it will also be very important to understand the data deeply in order to fully understand the results the algorithm delivers.  There will also be a challenge to deliver the vast amounts of data required for both learning and operationalization.

There is clearly enormous business potential here.  We are seeing organizations like Google quite literally “bet the ranch” on machine learning.  There is a very interesting New York Time article on how they used machine learning to transform Google Translate.


The good news is that there is tremendous potential for business disruption by building a competence in data and analytics.  The bad news is that almost everybody sees the same thing and everybody is trying to do it.  Even worse, data availability and technology changes are making it possible for new entrants to leapfrog the traditional barriers to entry.  The one thing nobody else has is your data and the ability to combine that with internal data to delivery better services, better experiences, better patient outcomes etc.   Start with your strategy. Make sure that you have a distinct competence in data management.