I’m glad to hear you feel comfortable explaining data to your friends, and I completely understand why you’ll avoid discussing metadata with them. You’re in great company – most business leaders also avoid discussing metadata at all costs! You mentioned during our last call that you keep reading articles in the New York Times about this thing called “Big Data” so as promised I’ll try to explain it as best I can.
Based on my original definition for data, “a thing you might want to remember that’s saved somewhere for future use,” I understand why you thought Big Data would simply be “a BIG thing you might want to remember.” But like most marketing and branding efforts within the data industry, with Big Data we’ve managed to confuse everyone – including ourselves.
So let me offer a definition for big data that you can share with your friends. Big data encompasses emerging best practices and technologies used to deal with three main things: (1) ridiculously large amount of data, bigger than the world has had to deal with in the past, (2) new, complex types of data most organizations have little experience with, and (3) changes to all this data occurring faster and faster.
Let me break down each of these things that make regular data “Big”:
- Large amount of data. “Large” is a relative term. You have 200 names and addresses in your address book, and when you and dad send out your annual holiday card it takes you quite a while to print the labels and stamp the envelopes – it feels large to you. But that’s child’s play for a retailer who wants to send holiday catalogs to 20 million households. But guess what? 20 million is way more manageable than dealing with over 1 billion users that Facebook manages. It doesn’t end there – consider earlier this year that Twitter estimated its users were sending 400 million tweets per day – that’s 146 billion tweets per year and growing.
- New types of data. Traditional “little” data is usually captured while businesses perform normal tasks like taking an order, answering a customer service call, or purchasing from a supplier. Think about when you place an order on Amazon.com and type in your first name, last name, bill to address, and credit card into all those separate fields. By capturing the data piece by piece in these fields, Amazon.com ensures it knows what each piece of data is meant to represent. This is called “Structured” data. But most of the information in the world is not so structured. Examples of unstructured data you create every day include emails, Facebook posts and Tweets, phone conversations (go away NSA!), and even data sent from your gas and electric meter to the utility company to report your usage so they can bill you.
- That might be updated/refreshed fairly often. Data is moving faster. A successful retailer may sell millions of items per day. As I mentioned above Twitter is tracking over 400 million tweets per day. In 2012 Google reported over 5 billion searches per day – that’s over 1.8 trillion searches per year. That’s big data!
So putting all this together, why all the excitement about Big Data?
Consider your recent Facebook post. You enjoyed a nice, long caffeine-deprived rant telling your friends how angry you were that the espresso machine you ordered online arrived broken, and that the shipping company and the retailer both provided horrible customer service when you contacted them. Well that Facebook post has a treasure trove of valuable information illustrating your customer experience that the shipping company, the retailer – and all of their competitors – would love to harvest. Big data capabilities can take your Facebook post – and hundreds of millions of posts just like it across Facebook, Twitter, LinkedIn, Blogs, emails to customer service, etc, – and deliver critical insights to these companies that can help them ensure they don’t lose more customers.
But the real excitement isn’t simply about customer satisfaction: many believe big data is the key to helping mankind solve some of our biggest medical, environmental, and social challenges. There’s a great book that you can display on your coffee table next to your “Sock Puppets from Around the World” book called “The Human Face of Big Data” by Rick Smolan. It shares a fantastic vision and is very easy to understand.
PS If anyone tells you that big data is all about strange things called Hadoop, MapReduce, or “the three V’s”, just nod your head and say “I understand” and walk away slowly…..