Data Integration Still a Big Data Obstacle
A Forbes Insights survey found that 90% of organizations have made medium to high levels of investment in big data, with a third viewing their investment as “very significant”.
“While businesses may be piling more money into getting value out of their data, many challenges remain. The survey found that more than half found it difficult to adopt a data-driven culture. Other challenges that are typically raised as an obstacle are difficulties with data integration, the skills gap and cost of investment.”
The truth is that data integration is very much an afterthought during the move to big data. The tactical solutions of the past are no longer capable of supporting the requirements of big data, including the volume of data and the need for real time data integration.
This has many enterprises scrambling for data integration strategies and technologies to get the necessary plumbing in place. As such, there is a huge inflection in my world of those enterprises who reach out to get their data integration act together as fast as possible. I’m doing my best to provide some guidance, which can be summarized as the following:
First, get your requirements in order. I know I sound like a broken record, but you need to understand what your core technology and business requirements are before you begin to create any sort of a strategy. This means understanding the volume and types of data you’ll be integrating, and governance, security, and performance requirements.
Second, understand the data. Again, the basics. However, you need to understand where the data is, how it’s defined, and what is the “single source of truth.” MDM tools are available to make this process easier, or you can define your own enterprise data model, working from department to department, or domain to domain.
Third, create a strategy to work from the biggest to the smallest. This means a holistic or conceptual definition of the problem. From there you can define macro solutions, and break those down to more tactical solutions. For instance, conceptually define the way that information needs to flow from operational to new big data systems. This is the “what.” Then, you define the tactical solutions that actually solve the problem, the “how.”
Finally, select a standard technology set that will provide solutions to most of your data integration needs. This means looking at the market with everything you know about your own data integration challenges. You’ll note that this is not about picking one piece of technology, but rather picking out several pieces that can be configured and reconfigured into solutions that will address the majority of your requirements, as defined above.
Of course, there is implementation and testing, as well as integration with development processes and the culture. Those can be even more challenging, depending upon the organization you work for.
The journey to big data needs to be lined with data integration approaches, strategies, and technology that work. Otherwise, we’ll see more surveys that show enterprises coming up short and not finding the huge value that’s in its data.