The Use and Abuse of Big Data and Hadoop

Published February 25, 2014   |   
Rick Delgado

Big data analytics is having a huge impact on us all. The ability to collect, store and analyze massive amounts of disparate data—using analytics platforms such as Hadoop in the cloud to uncover hidden connections, correlations and insights—is playing a bigger role than ever in influencing almost every aspect of our lives. And as the amount of data created every day grows exponentially, it seems that there is no limit to what big data holds for us in the future.

Still, in light of the Snowden and NSA controversies, the idea of more information becoming “too much information” is raising concerns, especially in the area of personal privacy. And although we might chalk up big data abuse to Big Brother, the truth is that a number of organizations that we have come to know and trust—such as the big three listed below—could already be compromising our personal privacy by taking big data too far. – Starting out primarily as a book retailer, Amazon gained popularity quickly for its massive selection and the ability to deliver what we wanted very quickly. Using algorithms to show us what other customers like us were also buying, Amazon grew rapidly into an ecommerce juggernaut. Today predictive analytics is playing a pivotal role in enabling Amazon to leverage customer data like never before, through a process called “Anticipatory Shipping.” The end game of anticipatory shipping is to accurately select and ship an item to a customer before that customer has even made the decision to buy—all made possible by analyzing the customer’s personal behaviors.

Read More