Tag Archives: big data analytics
In June, I was invited to present and participate in a panel discussion at a special program on Big Data at Stevens Institute Technology in Hoboken, New Jersey.
But my role wasn’t to join the other speakers and help pay homage to the power and potential of Big Data. Rather, I was asked by the organizer, professor Lem Tarshis, to play “Devil’s Advocate,” and talk about the issues and challenges Big Data brings up.
Indeed, there has been some pushback taking place against Big Data, alleging that its potential for knowledge advancement is being over-promised, its legal implications not well understood, and the possibility it may be outright dangerous for business leaders to be basing decisions on erroneous assumptions.
I began my talk with a little bit of history – close to 30 years ago, to be exact:
On September 26, 1983, the United States was rebuilding its nuclear arsenal, the Soviet Union was still the Evil Empire, and there was no trust between the two superpowers. In fact, the leaders of the Soviet Union were almost paranoid that the U.S. was planning a surprise attack against them. NATO was conducting war exercises at the time. Everyone was on hair-trigger alert. On the night of September 26th, the officer in charge of the Soviet Air Defense Forces was ill, so another officer, Stanislav Yevgrafovich Petrov, filled in.
Not long after the shift started, the center received a warning from one of its satellites that an ICBM missile launch has just taken place from the United States. All the systems were flashing red. Petrov looked at it and decided: It’s just one missile. If they were attacking, they wouldn’t just launch a single missile. So he overrode the attack warning. But then the center was alerted that a second missile had been launched from the midwestern U.S. Still, Petrov was undaunted. Then, there were alarms for a third launch. Then a fourth launch. Then a fifth launch.
I imagine many Soviet apparatchiks would have reflexively hit that red launch button at that point. But Petrov kept his cool. He had no information confirming whether the US launch reports were real or erroneous. He only had his gut at that moment. But something in his gut told him that this wasn’t the real thing. And he chose not to put through an order for a massive Soviet missile retaliation.
It turns out Petrov’s gut instinct was correct, of course. The stationary Soviet satellite above the continental U.S. was actually picking up glints of sunlight that were coming over the horizon, and mistaking it for missile launches. The data that was streaming into the Soviet command center was erroneous data.
But that was 1983, a long time ago, right with old Soviet technology? Our systems and data feeds are all perfect and flawless now, right?
Well, technology is more advanced, and yes, misreading Big Data doesn’t have to mean the end of the world. But perhaps every organization could use a Stanislav Petrov on staff. Someone who thinks critically, who can question the results the data is providing and put it into context.
Consider how, just a couple of months ago, someone highjacked the AP Twitter account with a false report of an attack on the White House. Sensing the immediate swoon in stocks, the high-frequency trading algorithms kicked into high gear and sent major US stock indexes into a nosedive, all in three minutes time.
A recent survey of 300 financial executives released by Experian finds that most executives feel they lack enough accurate information to successfully perform daily operations or make decisions. The main challenges identified by respondents are outdated information, linking different sources of information and inaccurate data. On average, companies thought that 25 percent of their data was inaccurate. Only 13 percent of companies thought the problems with their data were small enough that it did not require further investment.
In big data scenarios, you have managers not trained in statistics making bet-the-business decisions based on data of unknown quality originating from unvetted sources. Data analysts and scientists can write the algorithms that extract the data, but they aren’t necessarily in a position to understand the business implications.
That’s why, even though Big Data analytics is providing a lot of new types of information organizations can act on, business leaders and managers need to still understand the sources of this data, and how systems are delivering the information they will bet the business on. What is the source of the information? Are there other potential sources that will help build a conclusion? And, very importantly: What is the context of this data?
To be successful at Big Data, it’s incumbent upon organizations to encourage critical thinking among business users of the data.
Big Data isn’t a technology or solution set that gets dropped into organization, ready to deliver compelling insights that will put the business on an upward trajectory of intelligence and prosperity. Rather, it is a gradually building wave that organization’s leaders will need to learn to ride, or else get swamped on the sidelines. Understanding and working effectively with big data will take a lot of practice.
That’s the theme of a new book co-authored by Michael Minelli, vice president of information services for MasterCard Advisors, along with Michele Chambers, formerly general manager and VP of Big Data analytics at IBM, and Ambiga Dhiraj, head of client delivery for Mu Sigma.
In the book, Big Data, Big Analytics: Emerging Business Intelligence and Analytic Trends for Today’s Businesses, Minelli, Chambers and Dhiraj lay out the ways organizations can prepare to consume big data analytics.
1) Consider who is handling the “last mile” in data analysis: You need people who can look at the big picture with big data, and be able to explain its implications to the business. The authors quote Dr. Usama Fayyad, who talks about the crucial last mile in data analytics – the people “who are basically there to deliver the results of the analysis and put them in terms the business can understand. This last-mile group is made up of data analysts who know enough about the business to present to the CMO or the CEO.” At issue is the ability to find and hire these people, which is not an easy task. Also, a mistake many organizations make is putting these people to work on tactical assignments. “That’s a mistake, because these are people who can help develop and guide strategy, move the needle, and grapple with big issues,” Fayyad is quoted as saying.
2) Introduce the power of “geospatial intelligence”: Geospatial intelligence involves the gathering and analysis of data to form more of a 3D view of what’s happening around the organization. It’s about “using data about space and time to improve the quality of predictive analysis.” Minelli and his co-authors quote IBM’s Jeff Jonas: “It’s going to come from weaving together data that has traditionally not been woven together.” This means location data generated from sensors and smartphones, as well as social media data.
3) Separate the signal from the noise: With so much data and extremely large datasets, there’s going to be a lot of noise, with a lot of conflicting signals. “As data gets larger, it becomes increasingly difficult to fully grasp the meaning and magnitude of the data through exploratory analysis.” the authors state. The best way to help analysts decipher the nuggets of information needed is through visualization tools. For example, a “word cloud” of relevant terms plucked from a site or journal – and the most mentions, the larger the font – will provide, at a glance, the topics mentioned most often.
4) Collaborate: “successful analytics is a collaborative endeavor,” Minelli and his co-authors state. The first step in the process is to take your analytics intent beyond your core team and sell it to a wider group of decision makers – the prospective daily consumers of analytics in your organization.”
5) Learn to lead: “organizations that successfully consume analytics are driven by leadership, which builds consensus in the organization and allows for moving ahead without the need to have everyone on board every step of the way,” the authors state. “Strong leadership has been found to be the most important trigger in the wider analytics adoption in organizations.”
6) Measure, measure, measure: “Use analytics to measure itself,” Minelli and his co-authors urge. They add that hard numbers actually aren’t necessary to gauge any progress – the availability of analytics may elevate discussions and awareness of what the business needs. “One often but profound change in organizations is the maturing of a culture of objective debates, arguments and viewpoints driven by data and not just ‘gut feel,’” the authors state.
7) Change your incentives: Big data analytics implementations will shake up the organization will shake up the flow of information across the organizations, and thus re-arrange the hierarchy. Such projects will “bring in new stakeholders in employees’ decisions as well as higher levels of oversight,” the authors point out. “Sometimes, a general tendency of status quo bias exists, and employees do want to venture out of their comfort zone. You need to create robust incentives to overcome these barriers.”
The digitization of everything is creating a data explosion near you. Whether data is accumulating in the data center, in the cloud, on your laptop or mobile device, sometimes too much of something isn’t always a good thing. In a recent webinar cohosted by Informatica and Symantec, we polled our listeners to find out how the data explosion was impacting them. We also asked what type of unstructured and structured data is growing the fastest. Check out what they said. (more…)
There was an interesting story that surfaced recently. Indiana University researchers found that a pair of predictive modeling techniques can make significantly better decisions about patients’ treatments than can doctors acting alone. Indeed, they claim a better than 50 percent reduction in costs and more than 40 percent better patient outcomes. (See a story by Derrick Harris over at GigaOM for additional analysis, and I will also cover this subject in greater detail in a forthcoming column in TDWI’s “BI This Week.”) (more…)
Many organizations are rushing into big data efforts before it’s clear what business benefits will come from this new paradigm. And this is creating problems for big data analytics proponents. “At one very large financial services firm, we’ve heard that the next executive that uses the word ‘big data’ without a very precise explanation of how it will be used for the organization will be fired,” says Randy Bean, co founder of NewVantage Partners, quoted in MIT Sloan Management Review.
“The point is that there’s been so much overuse and misuse of the term that organizations need and want to understand precisely how big data capabilities and big data initiatives will help them,” he explains. (more…)
Make no mistake about it, executives are hungry for Big Data and the insights new forms of machine-generated and user-generated data can offer. However, Big Data analytics skills are hard to find, and even when they are available, hard to finance. As a result, the handling of Big Data analysis is defaulting to business users.
That’s one of the conclusions of a recent survey of 241 executives from across the globe, conducted by the Economist Intelligence Unit. The survey confirms that data democracy is a positive force – the vast majority of respondents, 77%, favor enabling more of their employees with better access to Big Data and the ability to analyze it in the context of other relevant data. There may be inertia at the top, but a grassroots movement within organizations is forcing the revolution into a reality. (more…)
A lot of media reports have been surfacing lately about “secretive” data mining activities taking place within the presidential campaign. Many articles paint the efforts with a sinister caste, implying that underhanded invasions of privacy are taking place.
But to any seasoned data professional, data mining is a discovery tool that pulls nuggets of insight out of mountains of data. For any business that wants to get ahead in today’s hyper-competitive global economy, advanced data mining and analysis is not a luxury, it is a necessity. As USA Today’s Jack Gillum describes the Romney campaign’s data analytics: (more…)
Quite a bit has happened on the topic of big data since my last post on Informatica Perspectives almost one and a half years ago. I have spent a career working with organizations on how to get control over their uncontrolled data growth and industry visionaries are promoting this brave new world of big data. (more…)