Clients are asking us to help them analyze large datasets of what traditionally would be considered peripheral data - the kind of data collected but never intended to be used. Is this something that other consultants are being asked to do and in which we should develop a capability?
The past few decades have seen an exponential increase in both intensive and extensive data collection. The sources of these data range from discrete business processes to consumer behavior to geographical information to global finance. The resulting aggregate datasets provide an unprecedented ability to analyze trends and patterns of complex behaviors in business, politics and consumer behavior. We have also developed prodigious new technologies to collect, store, search, visualize, analyze and share these data. With deference to privacy concerns, the ability to link these datasets to each other provides the analytical foundation to model and understand and predict future behavior of complex systems.
The term "big data" refers to datasets that exceed the capability of traditional commercially available analytical software. What could Walmart do with the data from 1 million transactions per hour? How about a marketer and millions of LinkedIn person to person connections? Consider the implications fro healthcare, finance, manufacturing, services, government and R&D, with estimates of savings from use of big data ranging into the hundreds of billions annually in the US alone. As companies are more able to collect and use larger data sets, consultants must be aware of the potential applications and the techniques available to them. A growing number of companies are specializing in big data service, whose activities ar a good idea to follow and whose services you could use to best serve your clients.Tip:
The McKinsey Global Institute issued a report Big Data; The Next Frontier for Innovation, Competition and Productivity
That discusses these concepts and provides a good insight into where consultants are most needed.© 2011 Institute of Management Consultants USA