Top 5 Big Data Challenges

Top 5 Big Data Challenges
Top 5 Big Data Challenges
In the recent past, we were constrained by many limitations around data. Now, we are only limited by our imagination. By using more data and more types of data, we can fundamentally transform our organizations, and our world. These transformations bring great opportunity, but also come with challenges. To that end, here is my take on the top 5 big data challenges organizations face today:

1) It’s difficult to find and retain resource skills to staff big data projects

The biggest challenge by far that we see with Big Data is that it is difficult to find and retain the resources skills to staff Big Data projects.  The fact that “Its expertise is scarce and expensive” is the #1 concern about using Big Data according to an Information Week survey of 541 business technology professionals (1).  And according to Gartner by 2015, only a third of the 4.4 million big data related jobs will be filled (5)

2) It takes too long to deploy Big Data projects from ‘proof-of-concept’ to production

At Hadoop Summit in June 2014, one of the largest Big Data conferences in the world, Gartner stated in their keynote that only about 30% of Hadoop implementations are in production (4).  This observation highlights the second challenge which is that it takes too long to deploy Big Data projects from the ‘proof-of-concept’ phase into production.

3) Big data technologies are evolving too quickly to adapt

With the related market projected to grow from $28.5 billion in 2014 to $50.1 billion in 2015 according to Wikibon (6), Big Data technologies are emerging and evolving extremely fast. This in turn becomes a barrier to innovation since these technologies evolve much too quickly for most organizations to adopt before the next big thing comes along.

4) Big Data projects fail to deliver the expected value

Too many Big Data projects start off as science experiments and fail to deliver the expected value primarily because of inaccurate scope.  They underestimate what it takes to integrate, operationalize, and deliver actionable information at production scale.  According to an InfoChimp survey of 300 IT professionals “55% of big data projects don’t get completed and many others fall short of their objectives” (3)

4) It’s difficult to make Big Data fit-for-purpose, assess trust, and ensure security

Uncertainty is inherent to Big Data when dealing with a wide variety of large data sets coming from external data sources such as social, mobile, and sensor devices.  Therefore, organizations often struggle to make their data fit-for-purpose, assessing the level of trust, and ensuring data level security.  According to Gartner, “Business leaders recognize that big data can help deliver better business results through valuable insights. Without an understanding of the trust implicit in the big data (and applying information trust models), organizations maybe be taking risks that undermine the value they seek.” (2)

For more information on “How Informatica Tackles the Top 5 Big Data Challenges,” see the blog post here.


  1. InformationWeek 2013 Analytics, Business Intelligence and Information Management Survey of 541 business technology professionals
  2. Big Data Governance From Truth to Trust, Gartner Research Note, July 2013
  3. “CIOs & Big Data: What Your IT Team Wants You to Know,“ – Infochimps conducted its survey of 300 IT staffers with assistance from enterprise software community site SSWUG.ORG.
  4. Gartner presentation, Hadoop Summit 2014
  5. Predicts 2013: Big Data and Information Infrastructure, Gartner, November 2012
  6. Wikibon Big Data Vendor Revenue and Market Forecast 2013-2017