The quantity of digital data we generate every minute of every day is now literally unimaginable. IDC tells us that the digital universe was already 4.4 zettabytes in 2013 and growing by 40% per annum. Another estimate by IBM suggests we are creating 2.5 exabytes per day. Of course, a large percentage of that is inane video of kittens on YouTube and mindless TV. But, regarding the rest of it, a very simplistic story has developed: the more data you collect and analyse, the better will be your decision making and, consequently, the more successful will your business be. To be successful today and into the future, you must be data-driven. Or phrased more negatively: if you’re not data-driven, you’re doomed.

Before addressing the insidious myths embedded in the above paragraph, I want to state that I am a big believer that businesses can and must make better use of information, need all the insight they can get, and can benefit from better use of all types of information. This was the premise of data warehousing and business intelligence long before big data inundated the world. And, even then, it was an extraordinarily difficult journey for most companies. Because the real issues lie beyond the quantity of data you have, or even its quality, and certainly far beyond the hardware and software you use. The real bottleneck lies in our ability to make decisions and take actions of real value based on information—and more. In truth, this depends as least as much if not more on people and their psychology than the data you have.

First, let’s deconstruct the myth.

1. You need to collect and analyze “all” data. 

The word all here poses a challenge. That’s clearly impossible. Often it’s qualified by relevant. Except that relevance can be fully known only at analysis time (and even then maybe not). So, let’s be clear: it’s going to be a subset.

Given the size and exponential growth rate of data, even collecting this potentially relevant subset is going to be a challenge, and managing whatever you have collected an even bigger one. Assuming that you can justify that expense, the really difficult task will be building sufficient meaningful context around this data mountain from completely diverse and disintegrated sources to do sensible analysis. There has only been limited progress in tools that can do this.

2. More data leads automatically to better decisions. 

This claim has been repeated since the earliest days of data warehousing thirty years ago (yes, data warehousing is 30 this year!). If it were that easy, data-driven would surely have become commonplace long ago. The reality is that most decision making is driven by a combination of factors ranging from organizational and social factors to the emotional and psychological foundations of simply being human. A 2014 study by the Economist Intelligence Unit showed that 90% of executives surveyed said if the available data contradicted their intuition, that they would reanalyze it, ignore it or collect more information, while only 10% said they would take the course of action suggested by the data. These percentages have remained constant for many years, despite enormous increases in the quantity and quality of data available and huge improvements in BI tools. I suspect that little has changed since then.

So, we need to understand a little of the psychology and neurobiology of how people really make decisions.

Nobel Economics prizewinner and psychologist Daniel Kahneman, and author of “Thinking, Fast and Slow” (2011) is often quoted as evidence of the multiple ways that our intuition can fail us in the modern business world and rational choice theorists and practitioners have been using this evidence as a justification for collecting ever more data. Yet personal experience suggests that information overload is becoming an ever greater problem facing decision makers and often leads to analysis paralysis and eventual plumping for one option.

Pyschologist Gerd Gigerenzer has long promoted the view that intuition can and should play an important role in decision making in books such as “Gut Feelings: Short Cuts to Better Decision Making” (2007) and “Risk Savvy: How To Make Good Decisions” (2014). Further scientific evidence is emerging to support the proposition that gut feelings should often be given credence—particularly if it works in the context of real information. Mousavi and Gigerenzer (2014) describe heuristics for successful decision making tied to Frank Knight’s 1920s distinction between uncertainty and risk, where he:

“associated generating economic profit with making entrepreneurial decisions in the face of fundamental uncertainties. This uncertainty is complex because it cannot be reliably hedged unless it is reducible to risk. In making sense of uncertainty, the mathematics of probability that is used for risk calculations may lose relevance. Fast-and-frugal heuristics, on the other hand, provide robust strategies that can perform well under uncertainty.”

Their conclusion is that in complex decision making deliberately ignoring information may be a more effective strategy in an uncertain world. Given that we live in an increasingly uncertain world, if would certainly be valuable to question the value of data-driven. Perhaps we should consider instead “intuition-and-information-informed” decision making as a viable alternative?