Big Data Management: How to Turn Big Data into Big Value

Value and maturity have now supplanted the ‘shiny new object’ fascination around big data. That’s according to a roundup of 2016 big data predictions from ZDNet. [1] And that’s heartening to folks like me who’ve been working with customers that are seeing some great successes with their big data projects.

But what we’ve also learned from these early adopters is that to get business value from big data, these projects must be supported by four big data management building blocks and repeatable processes.

Through our work with customers around the world, we’ve amassed a great deal of experience helping businesses ensure their big data helps their business analysts and business leaders make better decisions and drive operational efficiencies. So much so that in partnership with Wiley, we’ve created “Big Data Management For Dummies,” an eBook that explains the building blocks, including the three core pillars: big data integration, big data governance, and big data security.

Guidance for getting big data right, first time

Big Data Management
More than just an introduction to the topic of big data management, the eBook explains what big data project failures have in common, and why a reliance on new and unmanaged point solutions and manual processes are unreliable and unsustainable.

So no matter whether you’re just dipping your toes into big data, or you have several projects under your belt, the eBook will provide you with valuable guidance.

In this blog, I’ll describe the building blocks, but the eBook goes into a lot more detail. (And the eBook is free for you to download, now.)

Building Block #1: Understand the Big Data “Lab” vs. Big Data “Factory”

Even before you think about applying technology, successful big data management begins with understanding the skills and talent you need to make big data work. This group includes data scientists, data engineers, architects, and subject matter experts. The first group of talent is more scientific, primarily focused on analyzing data and discovering insights. This is the “big data laboratory.” The second group is more engineering and IT-centric—focused on architecting, building, and maintaining consistent, reliable, and trusted data pipelines that deliver actionable insights to the business. They work in what we’ll call the “big data factory.”

Laboratories experiment until they have a solution that will provide business value. They pass that solution on to the factory for production. Factories implement the solution provided by the laboratories to generate real business value. You absolutely need both groups.

Building Block #2: Apply the Three Pillars: Data Integration, Governance, and Security

For big data management to truly be effective, you need to start with a platform that delivers three key elements:

  1. Dynamic and optimized big data integration
  2. End-to-end big data governance and quality
  3. Risk-centric big data security

Big data integration should deliver high-throughput data ingestion and at-scale processing so business analysts can make better decisions using next-generation analytics tools and data engineers can rapidly deploy and easily maintain data pipelines as things change.

End-to-end Big data governance means business and IT users can be confident with the data they’re using. Look for comprehensive data governance that includes: data quality assessments, pre-built data quality rules, universal metadata catalog, entity matching and linking, collaborative self-service data discovery and preparation tools, and end-to-end data lineage.

Big data security analyzes all data to quickly detect and act upon risks and vulnerabilities. This requires a 360-degree view of sensitive data, supported by risk analytics, and policy-based protection of data at risk.

Building Block #3 – Clearly Define Management Processes

Much of the heavy lifting of big data management occurs during integration—where data ingestion, cleansing, preparation, and processing occur. But security and governance also have processes. Understanding these will enhance your ability to manage big data more efficiently.

Key big data management processes include:

  • Accessing, integrating, and cleansing data
  • Mastering and organizing data
  • Securing and protecting sensitive data
  • Exploring and analyzing data
  • Operationalizing insights into business value

These processes aren’t necessarily linear—rather they run as a cycle as data is brought into the system, processed, tested, and then implemented for the business. Then the next data project or test is started.

Building Block #4 – Empower Your Team with Intelligent Resource Management

Taking steps to empower your big data staff isn’t just right for them as employees—it delivers key benefits for the company as well. Your people are an investment, your strongest asset. To empower the team:

  • First, understand the role and needs of each team member or category of member. There will be a mix of data scientists, modelers, analysts, stewards, engineers, and business users, all with different perspectives, skill levels, and needs.
  • Next, incorporate the three pillars of big data management into the team members’ operating principles and environment. One of the biggest favors you can do for the team is to use a disciplined approach, ensuring that in particular governance and security processes are followed.
  • Then get help for your team in terms of training, effective technology, outside experts, and vendor experience. Odds are your team is already overworked—do don’t deny them the tools and expertise to increase their effectiveness.
  • Finally, consider what you can do with what you already have by creating repeatable, automated processes and standardized technologies.

Applying these building blocks will go a long way to improve the effectiveness of big data projects that turn big data into big value. To learn more, download our new eBook, Big Data Management For Dummies.

[1] ZDNet, “Big Data Predictions for 2016.” December 30, 2015