No matter how well integrated and powerful your back-end resources may be for managing Big Data, it’s all for naught if information can’t be effectively delivered and presented over that last 100 feet to decision-makers. It’s kind of like having a sophisticated power grid supporting the generation and transmission of electricity, but the consumer at home can’t figure out where the switch is to turn on the lamp.
That’s where data visualization can make all the difference. Yes, graphical displays of data have been around for more than a couple of decades now. I remember back in earliest days of the PC revolution using a package called Harvard Graphics, which did a nice job of converting rows and columns of data into nice, snazzy bar charts or pie charts. The spreadsheet makers recognized the power of visual representations and data, and also incorporated graphical capabilities into their products.
Now, there is an emerging class of front-end visualization tools that convert data points into visual displays – often stunning – that enable users to spot anomalies or trends in seconds. They are also referred to as 3D visualizations, but there is also the fourth dimension involved as well – time. Interfaces can be moved back in time – or forward if predictive analytics is available – to show how selected scenarios will change within a specified timeline.
If you want an illustration of what visualization can look like for enterprises, let’s broaden our horizons for a moment – really broaden our horizon. The Google Data Arts Team recently designed an interactive 3D map of the universe called “100,000 Stars.”
The 100,000 Stars interface enables you to zoom in on our own planet, then zoom out to the solar system, with our Sun at the center, then zoom over to the closest adjoining star and its solar system. Click on specific stars and planets, and you will get a brief description. Zoom out further, and you see we’re actually in one of the arms of the pinwheel of the Milky Way galaxy.
Imagine similar visualizations for business problem and opportunity areas, and you get what I mean – it’s out of this world. You can plot your data points, as well as even plot time to see how trends unfold. You already see this with those weather maps that move two, three days into the future. It turns the data into a physical object that you can view from different angles or timespans. It really brings data alive, and drives home any points that need to be made.
And we’re not just talking about “spacy” visualizations either. You may have seen, on some websites, the use of word “clouds,” for example. These terms getting the most usage are in the largest fonts, so at a glance, a view can see what the hottest topic may be.
In his latest book, Data Points: Visualization That Means Something, Nathan Yau makes the case for applying visualization against the toughest business and societal problems, as well as to uncover new opportunities that could not be considered previously in our flatter, 2D world. Ultimately, with data visualization, one can’t help but spot the trend or anomaly almost instantaneously:
“When you look at visualization for the first time, your eyes dart around trying to find a point of interest. Actually, when you look at anything, you tend to spot things that stand out, such as bright colors, shapes that are bigger than the rest, or people who are on the long tail of the height curve.”
In report published in 2011, Tony DeSantis, Mathew Gentile and Rich Simon, all of Deloitte, provide down-to-earth, on-the-ground example of how visualization can deliver business value: spotting potentially fraudulent invoices within an enterprise accounts payable department. “A traditional detection technique would be to list the invoice or purchase order numbers on a spreadsheet and sort them to identify numbers that are repeated, occur out of sequence, or increase by unusually small amounts over time, which such that the vendor has few or no other customers,” they point out. A visual graphic, on the other hand, will quickly make such anomalies blindingly obvious.
As DeSantis and his team put it: “Visual analytics builds on humans’ natural ability to absorb a greater volume of information in visual than in numeric form, and to perceive certain patterns, shapes and shades more easily than others. Using mathematical techniques to evaluate patterns and outliers, effective visuals can translate multidimensional data such as frequency, time and relationships into an intuitive picture.”
Big Data isn’t a technology or solution set that gets dropped into organization, ready to deliver compelling insights that will put the business on an upward trajectory of intelligence and prosperity. Rather, it is a gradually building wave that organization’s leaders will need to learn to ride, or else get swamped on the sidelines. Understanding and working effectively with big data will take a lot of practice.
That’s the theme of a new book co-authored by Michael Minelli, vice president of information services for MasterCard Advisors, along with Michele Chambers, formerly general manager and VP of Big Data analytics at IBM, and Ambiga Dhiraj, head of client delivery for Mu Sigma.
In the book, Big Data, Big Analytics: Emerging Business Intelligence and Analytic Trends for Today’s Businesses, Minelli, Chambers and Dhiraj lay out the ways organizations can prepare to consume big data analytics.
1) Consider who is handling the “last mile” in data analysis: You need people who can look at the big picture with big data, and be able to explain its implications to the business. The authors quote Dr. Usama Fayyad, who talks about the crucial last mile in data analytics – the people “who are basically there to deliver the results of the analysis and put them in terms the business can understand. This last-mile group is made up of data analysts who know enough about the business to present to the CMO or the CEO.” At issue is the ability to find and hire these people, which is not an easy task. Also, a mistake many organizations make is putting these people to work on tactical assignments. “That’s a mistake, because these are people who can help develop and guide strategy, move the needle, and grapple with big issues,” Fayyad is quoted as saying.
2) Introduce the power of “geospatial intelligence”: Geospatial intelligence involves the gathering and analysis of data to form more of a 3D view of what’s happening around the organization. It’s about “using data about space and time to improve the quality of predictive analysis.” Minelli and his co-authors quote IBM’s Jeff Jonas: “It’s going to come from weaving together data that has traditionally not been woven together.” This means location data generated from sensors and smartphones, as well as social media data.
3) Separate the signal from the noise: With so much data and extremely large datasets, there’s going to be a lot of noise, with a lot of conflicting signals. “As data gets larger, it becomes increasingly difficult to fully grasp the meaning and magnitude of the data through exploratory analysis.” the authors state. The best way to help analysts decipher the nuggets of information needed is through visualization tools. For example, a “word cloud” of relevant terms plucked from a site or journal – and the most mentions, the larger the font – will provide, at a glance, the topics mentioned most often.
4) Collaborate: “successful analytics is a collaborative endeavor,” Minelli and his co-authors state. The first step in the process is to take your analytics intent beyond your core team and sell it to a wider group of decision makers – the prospective daily consumers of analytics in your organization.”
5) Learn to lead: “organizations that successfully consume analytics are driven by leadership, which builds consensus in the organization and allows for moving ahead without the need to have everyone on board every step of the way,” the authors state. “Strong leadership has been found to be the most important trigger in the wider analytics adoption in organizations.”
6) Measure, measure, measure: “Use analytics to measure itself,” Minelli and his co-authors urge. They add that hard numbers actually aren’t necessary to gauge any progress – the availability of analytics may elevate discussions and awareness of what the business needs. “One often but profound change in organizations is the maturing of a culture of objective debates, arguments and viewpoints driven by data and not just ‘gut feel,’” the authors state.
7) Change your incentives: Big data analytics implementations will shake up the organization will shake up the flow of information across the organizations, and thus re-arrange the hierarchy. Such projects will “bring in new stakeholders in employees’ decisions as well as higher levels of oversight,” the authors point out. “Sometimes, a general tendency of status quo bias exists, and employees do want to venture out of their comfort zone. You need to create robust incentives to overcome these barriers.”
Hosting Big Data applications in the cloud has compelling advantages. Scale doesn’t become as overwhelming an issue as it is within on-premise systems. IT will no longer feel compelled to throw more disks at burgeoning storage requirements, and performance becomes the contractual obligation of someone else outside the organization.
Cloud may help clear up some of the costlier and thornier problems of attempting to manage Big Data environments, but it also creates some new issues. As Ron Exler of Saugatuck Technology recently pointed out in a new report, cloud-based solutions “can be quickly configured to address some big data business needs, enabling outsourcing and potentially faster implementations.” However, he adds, employing the cloud also brings some risks as well.
Data security is one major risk area, and I could write many posts on this. But management issues also present other challenges. Too many organizations see cloud as an cure-all for their application and data management ills, but broken processes are never fixed when new technology is applied to them. There are also plenty of risks with the misappropriation of big data, and the cloud won’t make these risks go away. Exler lists some of the risks that stem from over-reliance on cloud technology, from the late delivery of business reports to the delivery of incorrect business information, resulting in decisions based on incorrect source data. Sound familiar? The gremlins that have haunted data analytic and management for years simply won’t disappear behind a cloud.
Exler makes three recommendations for moving big data into cloud environments – note that the solutions he proposes have nothing to do with technology, and everything to do with management:
1) Analyze the growth trajectory of your data and your business. Typically, organizations will have a lot of different moving parts and interfaces. And, as the business grows and changes, it will be constantly adding new data sources. As Exler notes, “processing integration or hand off points in such piecemeal approaches represent high risk to data in the chain of possession – from collection points to raw data to data edits to data combination to data warehouse to analytics engine to viewing applications on multiple platforms.” Business growth and future requirements should be analyzed and modeled to make sure cloud engagements will be able “to provide adequate system performance, availability, and scalability to account for the projected business expansion,” he states.
2) Address data quality issues as close to the source as possible. Because both cloud and big data environments have so many moving parts, “finding the source of a data problem can be a significant challenge,” Exler warns. “Finding problems upstream in the data flow prevent time-consuming and expensive reprocessing that could be needed should errors be discovered downstream.” Such quality issues have a substantial business cost as well. When data errors are found, it becomes “an expensive company-wide fire drill to correct the data,” he says.
3) Build your project management, teamwork and communication skills. Because big data and cloud projects involve so many people and components from across the enterprise, requiring coordination and interaction between various specialists, subject matter experts, vendors, and outsourcing partners. “This coordination is not simple,” Exler warns. “Each group involved likely has different sets of terminology, work habits, communications methods, and documentation standards. Each group also has different priorities; oftentimes such new projects are delegated to lower priority for supporting groups.” Project managers must be leaders and understand the value of open and regular communications.
There are organizations truly reaping the rewards of Big Data, and then there are those who are just trying to catch up. What are the Big Data “leaders” doing that the “laggards” are missing? (more…)
Evolving from Chaos to Competitiveness: The Emerging Architecture of Next-Generation Data Integration
To compete on Big Data and analytics, today’s always-on enterprise needs a well-designed evolving high-level architecture that continuously provides trusted data originating from a vast and fast-changing range of sources, often with different formats, and within different contexts.
To meet this challenge, the art and science of data integration is evolving, from duplicative, project-based silos that have consumed organizations’ time and resources to an architectural approach, in which data integration is based on sustainable and repeatable data integration practices – delivering data integration automatically anytime the business requires it. (more…)
Last fall, The New York Times resident numbers geek Nate Silver provided a lesson in predictive analytics for the whole world to see – crunching big data to predict, with almost pinpoint accuracy – the winner of the U.S. presidential election.
The success of this high-profile project thrust big data analytics into the limelight, but there are many, somewhat more mundane applications, but with even more unforeseen revelations. (more…)
Data scientist may be the hot job of 2013, but many data professionals report they are already doing much of the work that would be defined as the data scientist role. They just aren’t calling themselves data scientists – at least not yet.
In a new survey of 199 data managers I conducted as part of my work with Unisphere Research and Information Today, Inc., we found that the traits of data scientists – individuals whose backgrounds include IT and programming; math and statistics; and a willingness to look at things differently—are already seen within today’s organizations, in the day to day work performed by database administrators, analysts, managers and consultants. The survey was conducted among members of the Independent Oracle Users Group. (more…)
Finally, there is now evidence of a clear link between financial performance and the broad use of data by employees. Specifically, organizations that take the lead in data analytics are more than three times more likely to be leaders within their industry groups than companies with standard analytics environments.
That’s the finding of a new survey of 530 senior executives, conducted by the Economist Intelligence Unit. There is little disagreement that the ability to make data available across the entire enterprise means greater productivity and performance. More than 80 percent of respondents believe that employees across their organizations “can and should be using data to do their jobs.” (more…)
Many organizations are rushing into big data efforts before it’s clear what business benefits will come from this new paradigm. And this is creating problems for big data analytics proponents. “At one very large financial services firm, we’ve heard that the next executive that uses the word ‘big data’ without a very precise explanation of how it will be used for the organization will be fired,” says Randy Bean, co founder of NewVantage Partners, quoted in MIT Sloan Management Review.
“The point is that there’s been so much overuse and misuse of the term that organizations need and want to understand precisely how big data capabilities and big data initiatives will help them,” he explains. (more…)
From the “it’s-About-Time” Department: More enterprises are embracing – or will soon be embracing – access to data analytics via mobile apps.
Having analytics available in a simple app fashion could be a major boost for efforts to “democratize” analytics in organizations. I once heard Competing on Analytics guru and best-selling author Tom Davenport wonder out loud at a conference why there weren’t more analytics being made available as a “cute little app.” By offering analytics through simple, single-purpose mobile apps, decision-making can be brought into a whole new realm. “I’ve heard of 50 analytical apps for the iPhone so far,” he points out. Examples include a nursing-productivity app, a truck-loading analysis app, and a social sentiment analysis app. (more…)