To Big Data or to Small Data is the question

If all you have is hammer, everything looks like a nail.

The concept known as the law of the instrument, Maslow’s hammer is an over-reliance on a familiar tool. It is convenient to look for the answers where it is easiest, i.e., in data, which begs the question –

Is Big Data overvalued?

The very term Big Data implies that quantity is paramount, and organizations collect it with the desire for accelerated returns. If your database grows by a 100 times, you expect to mine insights that are 100 times or even 300 times more accurate. From what I have seen, more of the same type of data provides only modest & incremental insights, not a quantum leap. Nothing in the data gathered, or in the way it is analyzed answers the questions you are asking. All that segmenting and clustering and scoring only adds to the already increasing mass of data.

Tasked with buying a cell phone, my siblings, with common genetic and environmental influences, will likely arrive at different consumption choices to mine.

If those closest to me exhibit different preferences, then why are these “previous customer,” i.e., strangers with no common nature or nurture to me being used to suggest products for me?

Big Data does not understand that consumers are bounded rational humans optimized over generations for “fight or flight.” In this complex and rapidly changing world, analytical models know very little about a customer’s present preferences and circumstances.

The circumstances of markets, like those of individuals, can change in an instant. Products sell out, forcing consumers to choose from what is available or to wait. Products stagnate. Promotions and discounts alter the relative attractiveness of one product compared with another, stimulating sales of one and depressing sales of another.

Think Small Data …

Focus on small data instead, i.e., product attributes and prices which change over time. This is the data consumers – your customers and your competitors’ customers – are using when choosing. To the extent of their ability, each consumer is assessing, comparing and evaluating the products and services on offer taking into account one’s own dynamically altering preferences over the attributes and one’s own changeable circumstances.

What you should be doing is maximizing the “willingness to pay,” to tap into the customer’s surplus. They will then tend to choose your product in preference to that of your competitors, depending on the bundle of attributes provided by your product. Analyzing data to reduce the error of estimation is not helping your customers to solve their problems – it is increasing them. The manifold combinations and permutations are adding to the burden, not reducing the load.

Customers will pay you for simply reducing the time they need to make a decision. Faced as they are with overwhelming choice, customers want up-to-date, reliable, valid, and trustworthy recommendations.