The greatest challenge to big data management and analysis isn’t necessarily the technical underpinnings, but rather, lingering executive confusion and uncertainty about what it is and what it can do for their organizations.
The main issue – and root of executive befuddlement – is the abject and ongoing confusion about what, exactly, is meant by “big data.” It’s certainly a hyped-up term for something that has been around for a long time. If you had a one-terabyte database around the turn of the century, you had big data, that’s for sure. If you had a 500-megabyte database in 1990, that would have been big data.
So the “volume” has always been there, and has always been a relative measure. The same goes for the “variety” aspect of big data. Unstructured data – such as word documents or machine log data – has been floating around organizations for decades now. How about the “velocity”? Real-time processing has been on corporate radar screens for well over a decade.
So, what’s changed that we suddenly see this data as an enabler, a game-changer, opening up the gates to a brave new world of analtyics-driven purpose? The rise of relatively cheap open-source tools and platforms for one. Capturing and analyzing large volumes of fast-moving data of various structures required very expensive equipment and consulting assistance. The expensive consultants may still be needed, but the technology is within reach for many organizations.
With this in mind, it is interesting to see that business leaders are warming up to the possibilities of big data, as a new industry survey shows. But what it is exactly they think they’re warming up to is still a big question mark. The survey, conducted among 500 business and IT executives by CompTIA, shows the big data phenomenon has caught the eyes of executives. The vast majority of organizations, 78%, say they feel more positive about big data as a business initiative this year compared to a similar survey conducted a year ago. And, remarkably, 57% feel they’ve made progress in moving in the right direction with data-driven programs, compared with 37% the year before.
To its credit, the CompTIA study’s authors question how accurately these findings actually translate to progress on the big data front: They note that while this years’ survey finds 42% of respondents claiming to be engaged in some of big data initiative – more than double from a year ago (19%) – such initiatives may be “big data” in name only. “This may stem from confusion or reflect the possibility of different users interpreting the concept of big data in different ways,” they observe.
So what we have is a lot of organizations diving into what they see as “big data” projects because that’s what everybody tells them they should be doing. But how much of this is simply the same types of data management and analytics projects that may have been engaged five, 10 years ago?
To really be making the most of big data as we understand it today, organizations should be addressing the following questions:
How much unstructured data is coursing through the organization, and how much of it is worth harvesting? It’s usually easy to measure the amount of structured data, such as that stored in relational databases or data warehouses, but unstructured data is a huge question mark. In many cases, management is clueless about what types of unstructured assets (user-generated files, machine-generated data) are actually available. It’s going to take a lot of research and discovery to uncover the unstructured data assets that are truly meaningful for the business.
Does the current data architecture support the introduction and integration of data sources? Most traditional data architectures are fairly rigid, built to support the inputs and outputs of relational data. Efforts involving other forms of data tend to be one-off projects, in which connectors or interfaces are hand-built built for a single purpose and then forgotten. Reaching out and exploring new and varied types of data require an architecture in which new sources can be rapidly and seamlessly introduced, without the usual silos.
Is the organization moving to an analytics culture? Big data analytics will never be “big” if it only is available to a few select decision makers or analysts. Big data will pack its punch when it enables decision makers at all levels of the organization – from customer care centers to production floors to the executive suite – to access analytics from various data sources. Even more helpful would be a way in which decision-makers can access analytical tools and back-end data sources through self-service approaches.