What is big data? Simply put, it’s data that is big, it’s your data when it gets big. And for most of you, that’s already happened or inevitable. Regardless of how you define big data, a more important question is “Are you ready for big data?”. Without some careful consideration of your data architecture, when faced with big data challenges you may find yourself writing some big checks as you scramble to address the new demands on your business. Fortunately not all is lost. As many Informatica customers have already learned, next generation data integration can help arm you to handle big data without ripping and replacing your existing data integration architecture.
Among other things, next generation data integration will make sure you’re ready to handle both the volume and variety of big data. Transaction data volumes are reported to be growing on average 50-60% annually. ‘Other data’ like social, unstructured or streaming data inside and outside the firewall is growing 2-3x annually and on average already makes up roughly 80% of a company’s data. Just to keep up with this growth, companies are spending a lot of money on hardware. In one survey over 50% of respondents said they were already using an appliance, while another 34% were planning on deploying one in the next 12 months. Customers have told Informatica the total cost of ownership for their appliances is as much as $100,000 per terabyte. Many of our customers have hundreds of terabytes of data in their warehouse; some have petabytes. Instead of spending the money upgrading, they’re changing their architecture, offloading processing from source systems and appliances and archiving data to get twice the scale out of what they have today. That’s saving millions and giving them a few more years of room for growth, without having to change their architecture.
Other organizations are turning to Hadoop to address some of these challenges.In the same survey mentioned above, 6% said they were using Hadoop today while 28% were planning on deploying in the next year. These companies are mostly using Hadoop to offload processing or handle new types of data. And while many today are hand coding on Hadoop, using MapReduce, Pig or Hive, they shouldn’t have to. In a perfect world, data integration tools should just work with Hadoop. With Informatica, they do.
History has shown us that there’s a lot to be gained from using data integration tools over hand coding. It’s easy to find examples of productivity gains from the use of best practices such as templates or other techniques to increase reuse, data quality, self-service, testing automation, operational monitoring. The good news for Informatica customers is these gains can be applied to Hadoop as well. Informatica allows you to design once, deploy everwhere, enabling you to reuse existing work and deploy to existing PowerCenter deployments or Hadoop. Some of our leading customers are seeing 3x productivity gains or more using these principles.
So ask yourself: “Is my organization ready for big data?”. When it comes to big data if you still find yourself with more questions than answers, I encourage you to join us for a January 31st webinar on next generation data integration where we’ll dig into these topics in detail.