It is not about the data, it’s the analysis that counts

The “big data” thesis is that the next “big” opportunity to emerge from the Internet revolution will be the ability to mine and extract profitable insights from the mountain of data that is now being collected digitally.

This is the optimistic projection in a “big” report just released by McKinsey, “Big Data: The Next Frontier for Innovation, Competition and Productivity.”    An example of the potential value according to McKinsey is the $300 billion to the US Health Care System.

All well and good but as the report rightly points out, there will be a huge need for people who can mine and extract this value.

From my experience it is not the access to the data or the ability to run complex statistical computations that will be the bottle neck to extracting the value, but the ability to interpret and analyse the data.

1. We need to start with asking the right questions.   Are they going to improve the quality of our decisions?   Why are we engaging in this exercise?  What do we hope to get out of it?

2. When we get back answers from the mass of data, we need to put them in to a broader context.    From my experience working with online analytics the data rarely throws up a clear answer.  It needs to be interpreted, added to other data, and put in to the context of the business situation.

3. Most importantly and something that will not be found in any statistical package is that we must be willing to follow where the analysis leads us.    Too often we get married to an idea or have already predetermined the outcome in our minds.     This is on individual level and more difficult to tackle on an institutional level.

Who wants to bring the bad news that our “big idea” is not working.    There are a multitude of examples where the data told corporations and institutions that something was wrong but they kept going anyway.

Having access to more data, statistical packages and PHD’s to run the database queries is all useless if we don’t work on honest analysis and are willing to kill “bad” ideas and run with the good ones.  Mining the data is going to be easier than changing the human condition.