When I began researching for this, the pilot episode, it was immediately apparent to me that Big Data was fundamentally different from data as we have known it for hundreds of years. We could just do so much more with it. It was changing industries, governments, people, and relationships between these entities rapidly and permanently. But what I couldn’t get a finger on was why.  Why was Big Data different from data as we have known it for hundreds of years? If it’s something ancient, just bigger, why does it change the world?

I didn’t have to wait too long to find the answer. In the very first chapter of Victor Meyer Schonberger’s Big Data, a book written about seven years ago, he gets to it pretty straight. When things change a lot in scale, they begin to change in essence also. If the rate of change is dramatic (double differential is out of whack), the thing itself begins to change. This is what makes Big Data essentially different from data ever before in human civilization.

This is how he explains it: an image is an image is an image. But speed these images up so the brain sees them as a continuum, and what you have made is a movie. A change in speed led to a change in essence. Here’s another example: gravity acts on all of us, but if you’re really small, like an insect, it barely has any effect on you. Again, change scale dramatically, and the essence begins to change.

A glass of water is fundamentally different from a tsunami. A drizzle is fundamentally different from a hurricane.

Think about the coronavirus pandemic. We’ve had economic downturns before. We’ve had unemployment before. We’ve had urban-rural migrations before. We’ve all experienced sickness before. Viruses come and go every year. We’ve had natural disasters before. What makes this pandemic so destructive is the scale and speed of it. The coming together of mass unemployment, global economic collapse, mental health falling off a precipice, distorted patterns of living and migration and human behaviour. Together, at this scale and at this speed, the essence of life today is different from the essence of life at the beginning of 2020.

The idea that after a point quantity and quality are inherently related is not a new one. It’s one of the three forgotten laws of Dialectical Materialism which inspired a lot of Karl Marx’s work. When quantity changes a lot, after a point, quality begins to change as well. This is the idea that explains why big changes in wealth distribution bring big changes in the essence of social structures.

This law, in turn, comes from a  much older philosophical idea: Aristotle’s Heap Paradox.

This is the Heap Paradox: if you have a heap of sand and you remove one grain from it, it’s still a heap. Remove one more, it’s still a heap. One more, it’s still a heap. But if you do this long enough, if you scale up the removal of grains, it’ll no longer be a heap. The essence of the sand heap will change.

This is at the core of understanding what Big Data is. When the quantity of data changed, its essence changed as well.

Listen.

Hosted on Acast. See acast.com/privacy for more information.