The Big Data Hype is Over: Now the Real Work Begins!

The Big Data Hype is Over
The Big Data Hype is Over
Every year, as Fall approaches, I look forward to the Strata + Hadoop World conference in New York. As a technology buff, I am constantly watching for the earliest signs of an industry inflection point. With Big Data, I believe we are at one such critical juncture. My deduction, while not too scientific, has some method to the madness.

Every new technology wave starts with an ample dose of curiosity, followed by a flurry of fans lining up to learn more. Next, there comes a period of early adoption, followed by a longer period of disillusionment. Then, there is a period of soul-searching. Finally, when the dust settles, there is wide adoption. From what I can see, Big Data is in the “period of soul searching.”

How can I be sure? As I warned you, there is no science to this. But I do see it as a sign when, on the very last day of the Strata + Hadoop World, I still have a box full of stuffed elephants ready to be opened. I am standing at the booth at waiting to give them away. Instead, I get constantly pulled into “serious discussions” with customers – with senior IT executives and architects.

It is said that the first signs of a market crash are when news-stand vendors on corners near Wall Street begin confidently dishing out serious investment advice. Similarly, and I admit it makes for good press, I believe there is some truth to the fact customers are starting to ask some serious questions of Big Data. That to me, my friends, is an inflection point.

When I said “serious discussions,” I meant deep soul searching – specifically, questions like these:

  • How do we get beyond experimentation?
  • How do we find the skills?
  • How do we demonstrate business value?
  • Why is it so difficult to gain the insights we were promised?
  • Why it is so hard to get good quality information out of the data lake?

As I boarded the plane back home, I was excited to blog about by my findings from the conference and to share my first-hand experience with you. But why would you take my word for it? So I did some more research – in fact, let’s do the research together. Open your favorite browser and do a search for “big data hype,” and then in a separate tab do a search for “big data challenges 2015.”

Here is what I see – big data is no longer an emerging technology, it is beyond the hype. I am also seeing many articles pointing to the lack of big data skills. But most interesting to me are the results that indicate that without the rigor of data management, big data will be a promise that will remain unfulfilled. Unless something is done about it.

Informatica has known this for a long time – that just because it is “big” data, the fundamental guiding principles of data management don’t become irrelevant. Which is why we have built a big data management platform from the ground-up, to specifically address the serious challenges of “big” data integration, “big” data governance and quality, and “big” data security.

Interested in learning more? Join me today for our online launch event.

Comments