2.5 Billion Gigabytes a Day – The World of Big Data

Darpa- Big Data

Last time on the subject of global data, we talked about the insane rate data was being created (2.8 billion terabytes yearly) and why that was the case. So, now that we produce 400 gigabytes yearly for every man, woman, and child on the planet, how is that data being put to use?

A term that has been frequently used in many IT and business discussions lately is “Big Data.” Big Data is pretty much exactly what it sounds like – data sets that are so massive in scale that they cannot possibly be made sense of without powerful computing processors and extremely complex algorithms.

Every single time a computer is interacted with, be it by a person or something inanimate, data is created. When a Facebook post is liked, a purchase is made in a store, or a weather balloon flies above the Midwest, data is being created that can be of use to somebody, somewhere. Getting a firm grasp on Big Data allows for organization to recognize important information among the tsunami of data they deal with.

Some examples of how various disciplines are putting Big Data to use are:

Business and Big Data

Large e-commerce sites depend on Big Data systems to operate as seamlessly as they do, none more so than Amazon. Every single profile on Amazon (about 137 million) is given a list of recommended products, based in near real-time on their shopping habits. There are over 230 million unique SKUs on Amazon, and every single one of them has its own rating, review portfolio, and real-time sales ranking in its category. When Amazon is selling over 300 items a second on Cyber Monday, 300 different credit card requests and logistic needs are processed every second. Now imagine trying to do all that without some of the most complex data management systems in the world. Not possible.

Big data isn’t just for e-commerce sites; brick and mortar stores also rely heavily on Big Data. Imagine how much data Walmart generates in just a single day – millions of people shopping in thousands of locations across the globe, purchasing hundreds of thousands of different products. One reason why Walmart is so successful is because of their grasp on Big Data, and the systems they’ve created around it. Wal-Mart tracks every single purchase made in every store the second it is entered into the cash register. If a $4.99 lawn chair is sold as fast as it can be stocked in Grand Rapids, but is gathering dust on the shelves in Omaha, they know that in real time and can adjust their logistics to more optimally meet demand.

Science and Big Data

Perhaps the biggest marvel of science in the last few years is the Large Hadron Collider. The Large Hadron Collider has over 150 million sensors that track 600 million particle collisions per second when running at full operating capacity. Obviously, no conventional data management system could possibly make sense of that much data. The systems of Big Data management they employ reduces those 600 million collision down to about 100 collisions of interest, which they record and use as the basis for any discoveries or adjustment.

Governments and Big Data

The NWS tracks the weather throughout the country every moment of every day. When a storm system begins to develop that may produce dangerous weather, the NWS’ Big Data systems analyze every facet of the conditions to identify how dangerous the storm may become – temperature, barometric pressure, humidity, etc. Every time you see a tornado or severe thunderstorm watch, it is because their data analysis systems detected patterns that commonly lead to deadly storms. By analyzing the large quantity of data in a short amount of time, the NWS is able to save lives by giving people warning well before the storm hits.

The Future of Big Data

Global data growth is showing no signs of slowing anytime soon, and how well organizations manage data is becoming more and more of a competitive advantage. All this points to even more complex and intelligent systems of data management being developed in the future. The rapid growth of the data management sector has led to a shortage of capable employees; a shortfall in the tens of thousands of jobs that pay extremely well. Something to consider if you’re looking for a lucrative career change…