Last week I was in both Bangalore and Mumbai providing the keynote address at the Informatica World Tour events. I also had the opportunity to visit with customers and speak with media. Much like the rest of the world, big data is all the buzz. I was particularly taken by a set of questions asked by local media. I thought I’d share them with you along with some of my perspectives.
Who needs big data? What kind of business problems should an enterprise be facing to consider big data?
In my opinion, everyone who wants to create a sustainable competitive advantage for their business needs big data. However, every industry and company is different, so the application of big data is different. I like what Kitty Fok, IDC Country Manager for China says…”If you’re not figuring out big data, your competition is and you will be out of business.” I like to say, “if you’re not figuring it out, your replacement will.’’
Where does it fit into the maturity curve of an enterprise using BI and Analytics?
All of us are at different stages of maturity with BI and analytics. I think the big data challenge is like the proverbial analogy of changing tires on a moving car. Perhaps you don’t have a data governance program. Perhaps data quality isn’t part of your vernacular. Maybe you have a menagerie of BI tools and data marts throughout your business. Does this mean you get it right before you embark on your big data initiatives. In my opinion, you need to start now, regardless of your lack of hygiene. I believe it’s our leadership challenge to resolve the organizational and maturity issues. Like most things, you start small and with a well-defined problem. The operative word is “start.”
How does IT partner with the business who is still swamped with reports and dashboards?
First, I don’t believe there is a separation of IT and the business. IT needs to be the business and business analysts need to know how to do the business’ job as well as their own. No, they don’t get two salaries, it’s an aspect of our chosen profession. Expanding on the prior question, I would start with a specialized team that includes members from both IT and business functions trying to solve a defined problem. In some circles, these are called skunk works, R&D teams, think tanks…
How can actionable insights from big data be ensured?
When Edison was inventing the light bulb, he failed the vast majority of the time before he found success. I think expectations need to be managed. We can’t simply implement technology and hope for a solution. Knowing the problems you want to solve, it will take several iterations before you realize success.
When does a data warehouse become inadequate and big data necessary?
I liked this question because it inherently assumes a technology stance, not a business problem stance. Let’s not forget that we are trying to solve business problems, not technology problems. Like in most cases, it all depends and it’s not a case of switching, i.e. a data warehouse or big data. The answer is both. They are used in concert with each other to gain actionable insights for an organization.
Is it about unstructured data or volume of data or the complexity of analysis?
I was sitting on a panel with a global partner from Accenture. When it comes to the three Vs of big data: volume, velocity and variety (and arguably a fourth veracity), he said he’s never seen a client with all three Vs. At most he’s seen two and in most cases he sees one. What we agree upon is the need for architecture because we know that demands will change over time. The architecture needs to be flexible to scale with the changing business climate.
Does analysis of social media or text become big data even when volume is low?
As with the previous question, it’s not likely to have all Vs present in a big data effort.
Data analytics has always been in the realm of data warehousing. How does this change with the launch of new tools and third-party cloud options for analytics?
Technology typically follows a natural evolution. One of the biggest challenges today, and in my opinion the biggest challenge with big data, is the lack of talent available. These are specialized people that you just can’t send to a class and they qualify. As with technology evolution, we often start with hand coding and realize the need to automate and this is absolutely the case with big data.
To conclude, I liked how the head of IDG in India calls big data – the new currency. I don’t think this is new for many of us as we realize that we can establish a sustainable competitive advantage with data. For him the main difference is now the ability to combine large external sources of data with on-premise information to create new value.