There is a recurring theme about Big Data projects. It’s about getting business excited about gaining deeper insight into how your organization is running ─ from all sources of data that is not traditionally available, at least cost-effectively. It involves social, mobile and cloud data and of course, includes historical insight from the traditional relational data sources stored that are clean and integrated (well, in an ideal world). Understanding what your customers and partners are feeling about your company can be extracted from text-based data and mining of relationships. The insight will be more powerful if you know the relationship history with them. Analyzing against search indexes and Web page views is also a prime example – again the key is to test those “behavioral patterns” that can translate into a better engagement model. There is a tremendous spirit of experimentation and curiosity in the process. Hadoop is also fueling this trend as IT can now store data that they used to throw away after a certain period and use its cost-effective, data processing power against unstructured data, the processing of which may not have been the strong suit for traditional databases.
Experimentation is happening at the levels of how to make Big Data projects relevant and meaningful to business. This “analytic’ laboratory is uniquely enabling people across the traditional business and IT boundaries to share current dashboards and business rules that show behavioral patterns. Some companies created new organizational designs so that business analysts and data warehousing professionals can truly collaborate in a spirit of partnership under one executive owner. Many see this as an irreversible shift that has the potential of finally resolving the known gulf of IT and business.
I was speaking with an IT executive of a major transportation and logistics company a while ago. The company is adding a Hadoop environment into the existing infrastructure where they have a fairly mature real-time business intelligence practice. So I asked, “Why do you need Hadoop?” He said, “Well, Hadoop is optimized for business content questions – like on-time delivery of freight for specific customers including names of drivers and what they are doing. We want to try different models for proactively engaging with customers and contacting the right departments. We are taking middle men out of it whether it is business or IT. By that, I mean, allowing business to ask questions and get answers without intermediaries directly and interactively. The textual integration piece is crucial to our information management practice and it is best augmented with a distributed environment like Hadoop. Behavioral data and predictions are key to making this successful. Of course, this will be integrated with management dashboards and operational reporting that we already provide in our information management practice.”
He is empowering the business to think out of the box about how to engage better with their customers and partners while eliminating unnecessary overhead. Another key aspect of it is that he is forming a list of “business questions” that architects can ask to link the business value into technology architecture about labor, idle time, compensation policies, etc. Through the process of experimentation, IT and business are collaborating more effectively.
In the next blog, I will talk more about the needs and trends in information management for organizations engaged in Big Data projects.