Trending topic we listen people talking about Big Data and the enhancements and improvements that it will bring to our lives, but what is Big Data? How we will benefit from it? Is the use of Big Data good or bad? I have been making myself the same questions, I have decided to do some research and go a little bit deeper into the topic and I try to summarise my findings and ideas in this post.
What is Big Data?
I think there is no rigorous definition of Big Data. It has been coined by the huge amount of data been processed nowadays in companies like Google, which also brought up some new processing technologies like MapReduce to let one manage far larger quantities of data than before.
When people talk about Big Data, they mostly refer to the ability of getting meaningful insights from big amounts of processed data coming from many different sources and the correlation of datasets that at first sight might seem despair but that together can provide us with precious information.
An example of this can be found in the 2008 project called Google flu trends, that Google created to predict the spread of the winter flu. They used information from the CDC (Centers for Disease Control and Prevention) in which doctors inform about new flu cases and combined it with the 50 million most common search terms Americans type. All they did was look for correlations between frequency of certain search queries (googlers might have typed something like "pills for cough and fever") and the spread of the flu over time and space.
Big Data repercussion
The possibility of analysing large datasets marks an important step in humankind's quest to quantify and understand the world. We have started measuring things that we were unable to measure before, mood states, communication exchange, consumer and user behaviour, the whole world has become a big data generator helped by the fast development of Internet.
The processing of large amount of data rather than small and accurate, privileging more data of less exactitude opens the door to new ways of understanding.
Most of our institutions were established under the presumption that human decisions are based on information that is small, exact and casual in nature. Big Data uses data generated not only by nature but also for machines and it can be processed much quicker, tolerating inexactitude that will change our vision of the world completely.
We might find that the ideal of identifying casual mechanisms is a self-congratulatory illusion, as big data analysis will bring better evidence about what is happening by correlating big datasets.
As the world shifts from causation to correlation, how can we pragmatically move forward without questioning the very foundations of society, humanity and progress based on reason?
Laws and ethics
The idea of a world similar to the one depicted in the movie Minority Report in which you could be imprisoned not because you have committed a crime but because you are likely to commit it in the future might not be that far away.
The danger of Big Data to us as individuals is that everything will be based on probability: algorithms will predict the likelihood that one person will get a heart attack and therefore pay more for health insurance, or not get one at all, people might be get excluded from society based on data reports about their behaviour, potential diseases (based on DNA map), etc.
We are entering a world control and handle by data-driven prediction in which an engineer or a doctor will not be able to make a decision without consulting first a computer that will tell him if the decision is that decision is the right one.
Just as the printing press prepared the ground for laws guaranteeing free of speech -which didn't exist earlier because there was so little expression to protect - the age of big data will require new rules to safeguard the sanctity of the individual.
Big data will also help us to prevent diseases and catastrophes before they actually happen or to minimise the damages. Again every great power comes with a great responsibility, so it is up to us to decide how we want to use it.
As an IT professional I cannot help but being excited about the Big Data revolution however, as a human being, I can already predict it will be very challenging for us to keep its use strictly to good purposes, as stopping the world hunger.
In my opinion Big Data must be used to answer questions that helps to build a more fair and safe society questions like, Is there a correlation between the total persons sleeping in shelter each day and the unemployment rate? Is there a relationship between the number of crimes in a city and socioeconomic indicators (Median income, Poverty, Education, etc)?
The technology is out there, can we imagine new ideas using Big Data to help us changing the world?