In June, I was invited to present and participate in a panel discussion at a special program on Big Data at Stevens Institute Technology in Hoboken, New Jersey.
But my role wasn’t to join the other speakers and help pay homage to the power and potential of Big Data. Rather, I was asked by the organizer, professor Lem Tarshis, to play “Devil’s Advocate,” and talk about the issues and challenges Big Data brings up.
Indeed, there has been some pushback taking place against Big Data, alleging that its potential for knowledge advancement is being over-promised, its legal implications not well understood, and the possibility it may be outright dangerous for business leaders to be basing decisions on erroneous assumptions.
I began my talk with a little bit of history – close to 30 years ago, to be exact:
On September 26, 1983, the United States was rebuilding its nuclear arsenal, the Soviet Union was still the Evil Empire, and there was no trust between the two superpowers. In fact, the leaders of the Soviet Union were almost paranoid that the U.S. was planning a surprise attack against them. NATO was conducting war exercises at the time. Everyone was on hair-trigger alert. On the night of September 26th, the officer in charge of the Soviet Air Defense Forces was ill, so another officer, Stanislav Yevgrafovich Petrov, filled in.
Not long after the shift started, the center received a warning from one of its satellites that an ICBM missile launch has just taken place from the United States. All the systems were flashing red. Petrov looked at it and decided: It’s just one missile. If they were attacking, they wouldn’t just launch a single missile. So he overrode the attack warning. But then the center was alerted that a second missile had been launched from the midwestern U.S. Still, Petrov was undaunted. Then, there were alarms for a third launch. Then a fourth launch. Then a fifth launch.
I imagine many Soviet apparatchiks would have reflexively hit that red launch button at that point. But Petrov kept his cool. He had no information confirming whether the US launch reports were real or erroneous. He only had his gut at that moment. But something in his gut told him that this wasn’t the real thing. And he chose not to put through an order for a massive Soviet missile retaliation.
It turns out Petrov’s gut instinct was correct, of course. The stationary Soviet satellite above the continental U.S. was actually picking up glints of sunlight that were coming over the horizon, and mistaking it for missile launches. The data that was streaming into the Soviet command center was erroneous data.
But that was 1983, a long time ago, right with old Soviet technology? Our systems and data feeds are all perfect and flawless now, right?
Well, technology is more advanced, and yes, misreading Big Data doesn’t have to mean the end of the world. But perhaps every organization could use a Stanislav Petrov on staff. Someone who thinks critically, who can question the results the data is providing and put it into context.
Consider how, just a couple of months ago, someone highjacked the AP Twitter account with a false report of an attack on the White House. Sensing the immediate swoon in stocks, the high-frequency trading algorithms kicked into high gear and sent major US stock indexes into a nosedive, all in three minutes time.
A recent survey of 300 financial executives released by Experian finds that most executives feel they lack enough accurate information to successfully perform daily operations or make decisions. The main challenges identified by respondents are outdated information, linking different sources of information and inaccurate data. On average, companies thought that 25 percent of their data was inaccurate. Only 13 percent of companies thought the problems with their data were small enough that it did not require further investment.
In big data scenarios, you have managers not trained in statistics making bet-the-business decisions based on data of unknown quality originating from unvetted sources. Data analysts and scientists can write the algorithms that extract the data, but they aren’t necessarily in a position to understand the business implications.
That’s why, even though Big Data analytics is providing a lot of new types of information organizations can act on, business leaders and managers need to still understand the sources of this data, and how systems are delivering the information they will bet the business on. What is the source of the information? Are there other potential sources that will help build a conclusion? And, very importantly: What is the context of this data?
To be successful at Big Data, it’s incumbent upon organizations to encourage critical thinking among business users of the data.