Big Data, Tipping Points and Disposable Data Integration

At our recent Informatica World Conference I had a revelation. These don’t come often to me, and it was made all the clearer to me when I was watching EMC World unfold across the social networks recently as well.

My revelation relates to the impact Big Data is going to have on enterprises and their approach to integrating data. I believe that Big Data will be the tipping point that will force enterprises to move away from hand-coding and other disposable data integration (or “DDI”) techniques. Just as we did 30 years ago when enterprises threw out hand coding and embraced the relational database; just as we did 15 years ago when enterprises threw out hand coding and embraced storage management software, now so too must enterprises throw out hand coding that delivers disposable data integration and embrace commercially available data integration software for true enterprise data integration. There is simply no other way of managing the enormous volumes, variety and velocity that will hit all enterprises in the coming years.

Informatica has been the leader in delivering enterprise capabilities that enable any data to be integrated, cleaned, transformed, virtualized, consolidated and synchronized. We have continued to innovate and deliver solutions that track heritage and lineage and deliver data timelines to ensure changes are understood over time. Through enhanced meta-data and business-IT collaboration we foster a strong relationship between the business and IT teams looking to enhance the value of data to drive business imperatives.

Yet, when we promote our technology into enterprises we often come across developers who believe they can hand-code data integration rather than using commercially available software. It’s always an interesting viewpoint that simplistically makes sense when dealing with a single process deep within the world of IT. The problem is that these apparently simple tasks mount up and suddenly enterprises find that integration happens everywhere and repeatedly.  Understanding what is connected to what and how data flows through the ecosystem becomes impossible – but still many developers seem obsessed with hand-coding and continue to deliver what can be be summed up as “Disposable Data Integration”.

Big Data will be the tipping point.

The tsunami of volume, variety and velocity that is beginning to hit enterprises cannot be managed through armies of hand coders repeatedly creating disposable data integrations across their enterprise. It simply CANNOT.  Enterprises are soon going to be drowning in data and devices, the like of which makes the current computing environment look puny in comparison. A reliance on disposable data integration (aka: disparate hand coding) will drive fragmentation, errors and result in a continuation of the seemingly never-ending spaghetti of IT integrations across the enterprise. This will only get worse as enterprises look to integrate an explosion of external cloud applications.

It is time to rethink hand coding once and for all and recognize it for what it is – disposable data integration that is disconnected and drives future maintenance issues across the enterprise.  It just makes more sense for an organization to spend their time and effort using enterprise data integration solutions to gain the greatest competitive advantage from their data rather than writing disposable data integration that cannot grow and scale as their business does.


This entry was posted in Big Data, Data Integration, Operational Efficiency. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>