Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it..
–Dan Ariely, Jan 2013

If you are even remotely connected to information technology in your daily life, chances are, you would have heard about ‘Big Data’ (or Hadoop, which unfortunately has become sort of a synonym) more than once, in the last few years. So, is it something new…. is it an invention, or a discovery … or a marketing gimmick, trying to sell old wine in a new bottle … or something else???

Let us start by looking at a system, built in the nineteen fifties…

The AN/FSQ-7 or second generation Whirlwind – SAGE

Given above are pictures of the ‘FSQ-7’ or the ‘whirlwind’ system used to build the SAGE air defense system for US military (24 such systems were used). Its aim was to collect information from all the radars across numerous centers in US and detect any unauthorized flying objects, in order to protect from Russia’s nuclear attacks. Imagine the amount of data that was flowing through this system on a daily basis (it was even diagnosing flying birds).. If this was not Big Data, nothing is…

Trivia: The SAGE system is estimated to have cost around $10 billion dollars in 1954, 
or about $67 billion in today’s money.. (while India's Mars mission Mangalyaan's cost
was $75million).  This was ‘probably’ the first active-passive cluster ever built, 
and was capable of executing 75,000 instructions per second (iPhone 6 has close to 
25,000 Million IPS processing power)..

In another example, the US National Snow and Ice Data Center’s (NSIDC) scientists needed to process billions of complex data objects to analyze how changes in Greenland’s ice sheet over time have affected global climate. Without the system that the NSIDC developed, processing this amount of information would have taken years, rendering any results a matter of historical record rather than actionable intelligence.

Similarly, the systems used by meteorological departments have been predicting weather using tons of data points… airline management systems have been handling hundreds of thousands of passenger bookings each day … Swiss trains have been running thousands of trains every day, on time … all this since long before the advent of so called today’s ‘Big Data’ technology/tools.

All these case studies make one thing clear… The problem of storing and analyzing huge amounts of data is not new, and nor is the capability to do it. So, why all the fuss over it in recent years… Why is it, that if you have Hadoop in your resume, chances of your landing a high paying job, improves considerably…

Well, the difference between then and now, is ‘cheap availability of technology‘ to the common woman. For e.g. the first computer was built in 1939 (must watch movie, ‘The Imitation game’), but it became talk of the town, only after the advent of IBM-PC/MS-DOS combination in 1981 made it affordable. Similarly, in India, first cars were introduced by the British as early as in 1900, but a common woman started to dream about learning to drive, and owning a car, only after 1983 (Maruti-Suzuki era).

And that is what has changed in the BigData landscape, in last few years. All the latest so called BigData or NoSQL solutions available today, and there are hundreds of them, are capable of running on ‘commodity’ hardware, can be learnt by anyone knowing basic programming (in any language), can be used by any organization producing data (the facebooks and linkedins), and above all, most if not all are ‘free’ to use.

The ‘era’ of Big Data is now. In the 2016 top technology trends (Gartner), there is no mention of Big Data. Not because it’s going out of fashion, but because, its no longer a trend to watch out for. It is already in use everywhere, and in fact, most of the items mentioned in that report, require and just assume, big data handling capabilities to be present.

It has transformed from ‘teenage’ to ‘adult’ sex… A necessity for continued evolution of human race .. 🙂

Posted on: 22nd October 2016, by : AdityaBhushan