Big data and the study of connections

I read this very interesting article at Om Malik on Broadband, where Dr. Alex Szalay of Johns Hopkins argues that Big Data – the enormously increased availability and analyzability of data as the world increasingly becomes digitized – will mean as much to science as the microscope once did.

It made me think of Douglas Adams’ wonderful lecture on “Parrots, the Universe and Everything.” One of his central points there is that science is changing – from a focus on taking things apart to understand to one where we put them together so that we can watch them interact.

On a smaller scale, I think this is extremely important for businesses, especially those that can be characterized as value networks, i.e. companies whose main value provisioning consists of connecting people and helping them exchange information, goods or money (well, OK then, money is information, I agree, but still.) At present, these companies segment their customers mainly by demographics (age, gender, location, education, etc.) or, for business customers, by size, industry and location. Massive data analysis will allow them to stop segmentation (which is only a representation reducing your market transaction cost, but also providing a less tailored product for the customer) and instead offer services and connections based on which other users each member is connected to and what they exchange.

Imagine instead if your telephone company, bank or insurance company could analyze you in terms of your interactions with others. That would allow the telephone company to group and tailor their services for the customers that create the most traffic, have the biggest impact, prevent the most accidents or in other ways cause desirable changes in behaviors of those around them. This is now done in a very primitive way and after the fact – imagine if you could do it in real time.