I just came back from MicroStrategy World. There were many conversations about social, mobile, cloud and big data. There was strong interest in cloud, clear adoption of mobile, and some big data adoption. eHarmony had a great presentation about how they handle big data with Informatica, and how they’re starting to use Hadoop with Informatica HParser running on Hadoop for processing JSON.
But that wasn’t the number one conversation. The one topic that everyone was interested in – and I talked to nearly 100 customers and partners over four days – was creating new reports faster, or Agile BI.
I’m not talking about agile as in scrum, but agile as in greater business agility. A little over a year ago, Forrester found that 66% of BI requirements change each month, but 77% of respondents said it takes days to months to get their requests fulfilled. Most of the people I talked to at the conference said the majority of new reports required data outside the warehouse and typical turnaround times were one month or more.
To some, one month may seem OK, but not to most in the business. The business wants the results in days. For eHarmony, that often meant in one day. If you think one month is OK, ask yourself why there are so many ‘spreadmarts’ (spreadsheets and data marts) created by the business. It’s because the way you increase profitability or add new revenue streams is with new insights, not the old ones. If you wait a month, you lose a month of benefits or worse, the entire opportunity.
So why does it take so long? If you look at BI as a business process, there are two reasons. The first is that business and IT have to talk. A business analyst has to explain to a developer what they need and that can often take weeks. The second reason is that the BI team often takes all their data from the data warehouse. If new data is needed that is not in the warehouse, it may take a month or more to add that data, to change and test the warehouse. So IT operates in batch. They gather many requirements and make many changes at once in the warehouse to be efficient. That’s efficient for IT, but not necessarily what the business needs.
The good news is that companies are now creating reports in days, not months. How did they do it? They eliminated the conversation with IT and accessed data directly from the sources. IT analysts at companies like HealthNow are using a Web-based tool called Informatica Analyst to directly access, profile, transform and cleanse data live. A BI developer adds a report on top of the view, which looks just like it came from a data warehouse – except it didn’t. It’s a live or federated query across many source systems and the warehouse. This is called Data Virtualization. It’s implemented with Informatica Data Services, which along with Informatica Data Explorer, Informatica Analyst and Informatica PowerCenter are all an integral part of the Informatica Platform, and part of the new PowerCenter Data Virtualization Edition. So the business gets their report in days. A developer can take this work, captured in metadata and decide whether to keep it virtual or move the data into the warehouse as an ETL movement with PowerCenter. The report doesn’t change.
So, have that conversation. If you’re in the data warehousing team, go ask your BI team, your analysts, whether they need to get new reports in days, rather than months. Go find the people with spreadmarts and ask why they created them. Show them they could get reports in days and you’ll hear what I heard from nearly 100 people, that it’s critical to the business and it’s time for Agile BI.