The Big Secret to Fulfilling the Big Data Promise Unveiled

Big Data Promise At first, a lot of people were writing about the fact that big data is inevitable. Then, even more people wrote about the big promise that big data holds for the business. And now, a lot more is being written about the big failures of big data. However, the topic that is the least discussed – and what I believe should be top of mind – is the big secret to delivering big value from big data.

As they say – it’s about the journey and not the end. This is exactly right! Let’s look at the typical big data journey. Meet Joe, a big data enthusiast, the hero of my story. Everyone was talking about Hadoop. So, Joe wanted to see for himself what all the noise was about. Commodity hardware – check! Credit card – check! Hadoop downloaded – check! Self-help tutorials – check! Ready to go – check!

Joe was excited –he had taken that first step into the big world of big data. He quickly built himself a cool new sandbox to play in. He landed new types of data into it, and the system simply sucked it in without any extra work with what they called schema on read – cool stuff! He then cranked things up and lo and behold, tons and tons of data swept in, without even breaking a sweat. It was alive!

Knees shaking, Joe ran to get Samantha, his data scientist friend, to show her his handiwork. Samantha took one look at the sandbox and told Joe that this was just the beginning of what she felt would be a game changer for the business. The company had been looking for ways to improve the effectiveness of their marketing campaigns – maybe they would be the ones that would show the way.

Samantha told Joe about this thing she had heard about – a data lake. Maybe he could help her build it. They could simply land all their marketing data into this data lake, quickly prep it, write some code, and before they knew it, they would uncover some amazing ways to target their very best customers. No more of that complicated data warehousing stuff. It was the brave new world after all!

They imagined the moment when they would present Fred, their CMO, with some amazing new ways to deliver next best offers to their customers. They imagined his grin when he saw the marketing pipeline going through the roof. They imagined walking up the ramp to receive their Employee of the Year awards. They imagined a promotion and that vacation in Bora Bora. The possibilities were endless!

But here’s where our story falls flat on its face. Six months into starting to build a data lake, Joe and Samantha were exactly where they were when they started – no meaningful results. The data they needed was not just in Hadoop. It could not be instantly trusted. There was a new technology to master every day.  And, it was painful, manual, hard labor. It wasn’t supposed to be this way.

Unfortunately it was. According to Gartner1, “through 2018, 70 percent of Hadoop deployments will fail to meet cost savings and revenue generation objectives due to skills and integration challenges. “And, “through 2018, 80 percent of data lakes will not include effective metadata management capabilities, making them inefficient.” The big secret to delivering big value with big data lies in those very words.


A typical big data journey


Big data is no different from small data. Fine, there is a lot more variety of data, and much more data to be handled in general. But, does that mean that the very fundamental principles of managing data should be ignored? Definitely not. With any data – big or small – you still have to acquire, ingest, transform, secure, master, govern, and blend it…before consuming it. That’s the big secret – and now you know it!

And, if you want to learn more about how to become big data ready and deliver big value with big data  – go here.

1Predicts 2016: Evolving Information Infrastructure Technologies and Approaches Bring New Challenges, December 2015, Ted Friedman, Roxane Edjlali, Guido De Simoni, Adam M. Ronthal, Nick Heudecker, Merv Adrian, Bill O’Kane, Mark A. Beyer, Donald Feinberg