0

Avoiding Big Data, and Big Data Integration Confusion

We discussed Big Data and Big Data integration last month, but the rise of Big Data and the systemic use of data integration approaches and technology continues to be a source of confusion.  As with any evolution of technology, assumptions are being made that could get many enterprises into a great deal of trouble as they move to Big Data.

Case in point: The rise of big data gave many people the impression that data integration is not needed when implementing big data technology.  The notion is, if we consolidate all of the data into a single cluster of servers, than the integration is systemic to the solution.  Not the case.

As you may recall, we made many of the same mistakes around the rise of service oriented architecture (SOA).  Don’t let history repeat itself with the rise of cloud computing.  Data integration, if anything, becomes more important as new technology is layered within the enterprise.

Hadoop’s storage approach leverages a distributed file system that maps data wherever it sits in a cluster.  This means that massive amounts of data reside in these clusters, and you can map and remap the data to any number of structures.  Moreover, you’re able to work with both structured and unstructured data.

As covered in a recent Read Write article, the movement to Big Data does indeed come with built-in business value.  “Hadoop, then, allows companies to store data much more cheaply. How much more cheaply? In 2012, Rainstor estimated that running a 75-node, 300TB Hadoop cluster would cost $1.05 million over three years. In 2008, Oracle sold a database with a little over half the storage (168TB) for $2.33 million – and that’s not including operating costs. Throw in the salary of an Oracle admin at around $95,000 per year, and you’re talking an operational cost of $2.62 million over three years – 2.5 times the cost, for just over half of the storage capacity.”

Thus, if these data points are indeed correct, Hadoop clearly enables companies to hold all of their data on a single cluster of servers.  Moreover, this data really has no fixed structure.  “Fixed assumptions don’t need to be made in advance. All data becomes equal and equally available, so business scenarios can be run with raw data at any time as needed, without limitation or assumption.”

While this process may look like data integration to some, the heavy lifting around supplying these clusters with data is always a data integration solution, leveraging the right enabling technology.  Indeed, consider what’s required around the movement to Big Data systems additional stress and you’ll realize why strain is placed upon the data integration solution.  A Big Data strategy that leverages Big Data technology increases, not decreases, the need for a solid data integration strategy and a sound data integration technology solution.

Big Data is a killer application that most enterprises should at least consider.  The business strategic benefits are crystal clear, and the movement around finally being able to see and analyze all of your business data in real time is underway for most of the Global 2000 and the government.  However, you won’t achieve these objectives without a sound approach to data integration, and a solid plan to leverage the right data integration technology.

FacebookTwitterLinkedInEmailPrintShare
This entry was posted in Big Data, Data Integration, SOA, Uncategorized and tagged , , , , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>