Has big data entered the “trough of disillusionment?” That’s what I’ve heard recently. Like many hyped up technology trends the trough can be deep and long as project failures accumulate, or for ‘hot’ trends that evolve and mature quickly the trough can be shallow and short, leading to broader and rapid adoption. Is the big data hype failing to deliver on its promise of increased revenue and competitive advantage for companies that leverage big data to introduce new products and services and improve business operations? Why is it that some big data projects fail to deliver on their promise? Svetlana Sicular, Research Director, Gartner points out in her blog Big Data is Falling into the Trough of Disillusionment that, “These [advanced client] organizations have fascinating ideas, but they are disappointed with a difficulty of figuring out reliable solutions.” There are several reasons why big data projects may fail to deliver on their promise:
- Data volume quickly consumes information management infrastructure capacity resulting in costly HW/SW upgrades. Business impact: cost overruns
- Big data projects often start as POCs that include open source software. IT teams underestimate the work involved moving from the POC stage to implementing an enterprise deployment. Business impact: project delays
- Big data projects often revert to hand-coding which needs to constantly be re-factored as data volume grows, new data sources are added, and emerging technologies evolve. Business impact: project delays, cost overruns and lower than expected results
- Unable to address the security and governance standards of the Chief Risk Officer. Business impact: project canceled or delayed
- The specialized skills required to staff projects are difficult to find and very expensive. For example, data scientists and Hadoop developers are in short supply and as a result expensive to hire and retain. Business impact: project delays and cost overruns
- Big data projects can be difficult to maintain as developers move onto other projects. Business impact: SLAs not met resulting in low user acceptance
So how can the IT industry pull itself out of the big data trough of disillusionment? It will take a combination of best practices, productivity tools, proven architectures and efficient project team structures.
Organizations need to also recognize that data integration is often 80% of the work in big data projects and the tendency to hand-code on Hadoop is too costly, inefficient and slows down time-to-value. Informatica has already seen customers lower costs as much as 2x, minimize risk, and innovate faster with big data. To avoid the trough of disillusionment and deliver on the promise of big data:
- Companies will lower costs by offloading data storage and processing to low cost commodity HW/SW (e.g. Hadoop) and increasing productivity by up to 5x with a no-code development environment (like the one offered by Informatica).
- Companies will minimize the risk of big data projects by quickly staffing projects with more productive and trained data integration experts by leveraging a single data integration platform (like Informatica) across their existing infrastructure and new technologies like Hadoop.
- Companies will innovate faster with big data by quickly onboarding any type of data, discovering insights faster through rapid prototyping and collaboration, and operationalizing insights for deployment with enterprise scalability, security and flexibility as data volumes continue to grow. Once again, they can do this thanks to automation offered by companies like Informatica.
There are already big data successes proving companies are using big data to introduce new innovative products and services, improve business efficiency, and make better decisions. Companies cannot afford to ignore the potential big data has to change their business. As Svetlana points out in her blog, the “…plateau of productivity will be reached when tools and product suites saturate the market.” I’m confident we will see business pass over the trough like a Lamborghini at 200mph as more companies adopt best practices like those recommended by Gartner (e.g. “Understanding the Logical Data Warehouse: The Emerging Practice”) and Ralph Kimball (e.g. “Newly Emerging Best Practices for Big Data”) and as organizations tackle the 80% of big data integration work by implementing an enterprise-ready data integration platform (like Informatica).