How can big data lead to more rational decision-making?
In this age of disruption, organizations are increasingly being forced to make existential choices. How can big data support better and more rational decisions?
We all fall victim to some cognitive bias or another, which can prejudice our decision-making processes. All too often, decisions are based on subjective “gut feelings” rather than hard objective facts. Confirmation bias also has a dangerous influence on decision-making. People often seek out information to confirm a hypothesis as opposed to try and prove it wrong — the spirit of the scientific method that underpins modern science.
With a general psychological tendency for over-confidence, over-optimism and pattern recognition, executives can dismiss evidence that contradicts their opinion, while overemphasizing supporting evidence. Such cognitive biases can translate into underestimating competitor responses, R&D and acquisition overspend, and divesting a business far too late in the mistaken belief of being able to salvage the situation.
More data, more confusion?
The digital universe is overflowing with information. Around 2.5 quintillion bytes of data are created every day, while 90% of the world’s data has been created in the last two years alone. This trend shows no sign of slowing down as the total amount of data is predicted to double every two years.
This is far too much information for even the most gifted individuals to possibly process and understand without some help. This information clouds, rather than informs, decision-making.
Herbert Simon, a Nobel Prize laureate, defined this human constraint as “bounded rationality” — a stark deviation from the classical economic view that humans are fully rational.
Alongside dealing with imperfect information and time restraints, Simon argued humans have a limited capability to process the vast amount of available information. Collectively, these cognitive constraints or “boundaries” limit our ability to make fully informed and rational decisions.
Investing in the unknown
Andrew Haldane, the Chief Economist and Executive Director of the Bank of England, highlighted the issue of bounded rationality and investments in CDO-squared derivatives. He argued that investors would need to read one billion pages to understand fully these financial products.
With such complexity, it is unsurprising that many investors lost heavily in the 2008 financial crash with these securities. They simply did not — and could not — understand what they were buying.
So how do we overcome these biases and bounded rationality to make better-informed and optimized decisions? The only way we can realistically, and objectively, make sense of this vast digital universe is through skilled big data analytics.
Good analytics enable executives to identify objectively their organization’s value drivers and examine how potential decisions and events can affect them. Armed with big data insights, executives can stress-test the validity of their decisions through “what if” scenario planning and optimize their tactical and strategic decision making.
As more data is generated through the internet of things and smart-connected technologies, executives will have even more comprehensive data to inform decisions — but only if the right insights can be drawn out.
The human element
Big data insights can overcome our biases, but its real value is paradoxically realized only through the human element of analytics. Human judgment is required to intelligently analyze, review and act upon insights to create value from big data. We just need to make sure that this human lens is not clouded by our cognitive biases.