The Paradox of Big Data… or Why Data is Misery

Industry   |   
Published May 14, 2013   |   

We’ve heard it shouted from every rooftop recently– big data is the next big thing, we’re a data-driven company, data underlies everything we do….

So here’s a paradoxical, even heretical thought: Data is misery.

Why? Consider this.

We are all becoming adherents of the data-driven life and the quantified self movement. Arguments not backed by data just do not cut it, whether it is a business meeting, or a casual social conversation. Why, even economists tout the virtues of creating small data-led experiments that can reveal the best options to everything from improving government action to reducing poverty.

At the same time, we are swamped by ever vaster amounts of data, that comes at you in all shapes and sizes, at frightening speeds. We are all familiar with the 100,000 row, 25 tab, macro-littered, xl sheet, the ‘latest numbers show that’ comment, the real-time deluge of statistics that crawls across the bottom of every one of the 100s of screens we spend our lives staring at.

So what happens?

At work, we go into meetings armed with gigabytes of data and analysis, and the common refrains at the end of the meeting are: Let’s look into the numbers in greater detail, let’s resolve this difference, let’s get accurate and up-to-date numbers next time.

In our personal lives too, we see the same challenge. It’s not tough to find qualitative and quantitative reviews of practically any product or service, in fact, there are way too many, all different. We can find out everything about everything, each piece of data contradicting the earlier, and we routinely spend hours in this vast data swamp.

Our response? Most often, we freeze. We postpone decisions. We are unhappy with the decisions we do take. We are dis-satisfied because with all this data, we should be able to make a perfect decision and we are not.

All of these have been brilliantly described in a different context, by Prof Barry Schwarz in his Ted Talk on the Paradox of Choice a few years ago.

We are at an inflection point in the big data revolution.

With less data, it was easier to spend more time thinking and find the right answer, by bringing judgment and experience to play.

With more and more data, we need more effortto go through all of it and understand it, we find more contradictory signals to reconcile, and we spend more time aligning with others, different views on the same data.

And we have less and less time in which to navigate and interpret this data.

Instead of liberating us, instead of freeing up time to make better decisions, instead of making us happier that we are able to quantify stuff better, we seem to be headed the other way.

Indeed, Data can be misery….

So how do we cope? More on that in the next post.