I truly believe that a Data and Analytics function has the mandate to enable better decision making within an organisation. Very few practitioners would disagree with this argument, however for us to truly drive our vision, it is important to understand how people make decisions.
The human decision-making process is ambiguous, to the extent of being mysterious. One thing researchers all agree on is that all decisions have an element of emotion attached to it, which really hits at the core mission of a Data and Analytics function. How can data be used to influence emotion, and subsequently the decision-making process?
I tend to think that all humans make decisions based on one or a combination of two factors. This is either intuition or information.
Decisions made through intuition are usually fast, people don’t even think about the problem. It is quite philosophical in nature, meaning that someone who made a decision based on intuition will have difficulty explaining the reasoning behind it. The decision-maker would often utilise her senses in drawing conclusions, which again is based on some experience in the field of study.
On the other side of the spectrum we have decisions made based on information. These decisions are rational – it is based on facts and figures, which unfortunately also means that it can be quite slow. The decision-maker would frequently use reports, analyses and indicators to form her conclusion. This methodology results in accurate, quantifiable decisions, meaning that a person can clearly explain the rationale behind it.
My initial field of study was metallurgical engineering, and as a fresh graduate I worked on a ferro-alloy smelter. I recall how the older guard would remove respiratory PPE whenever we tapped molten alloy from the furnaces, and smell the fumes emanating from it. They would also look at flags like alloy flowing slowly and bearding, or sparks illuminating the air above the alloy. Based on these indicators, they would recommend changes to the input materials. This is decision-making based on intuition. They used their senses and couldn’t really explain why they are making certain recommendations, however it was based on many years of experience.
My approach to the same problem was slightly different. I would request for a sample to be taken of the molten alloy. This then needed to cool down, be taken to a lab and if I was lucky I would have results back in a few hours. The chemical assay results would then be plugged into a metallurgical mass balance, and based on the output of this model, I would make changes to the input materials. My approach was clearly based on information, with little experience to back it up, except for academic studies based on the fundamentals of chemistry.
So which is the right approach? Well, I found that in the majority of cases, the old guards had it right in terms of what needed to be changed. What they didn’t get right though was how to quantify the change. Their approach resulted in much faster changes in the process, and therefore a step in the right direction for process control, however it could also easily overshoot the control bands in the opposite direction. My approach was more accurate, but took longer to implement, meaning that the process would run for a longer period out of control, however the potential for overshoot was smaller. The answer to the question of which approach is right is something we in Data and Analytics need to understand very clearly.
There shouldn’t necessarily be conflict in these two approaches. We should accept that decisions will always have emotion attached to it, and as such intuition will always be trusted more than information. That is absolutely fine. In some cases we simply don’t have the luxury of wasting time analysing a problem. We have to make quick decisions based on our intuition. These decisions are not just tactical in nature - just think of a CEO that needs to make a quick decision to ensure market penetration before a competitor jumps first.
In my view though, our role in Data and Analytics are never negated. We always have to enable better decision making, and knowing what we do about the decision-making process, this means influencing emotions. Instead of pushing our organisations into information driven decisions, we should rather be driving the agenda of informed intuition. It’s a subtle difference, but the value is immense. By affirming intuition, you will also be able to sculpt it, and once that happens, intuition will be informed.
The same principle should apply when we consider advanced analytics solutions to problems. Mike Bugembe, the founder of lens.ai, recently said that a key mistake made by organisations is the expectation that data scientists should address business challenges in isolation. He makes an important observation that successful AI deployments marry human experience with the capabilities of technology. This confirms the absolute need for and value of informed intuition in our approach to data and analytics, as we will simply never be able to code all human experience or isolate emotion in the decision making process.
In conclusion, although organisations should have the ambition to automate decisions and operate mainly in the prescriptive realm, it should never discount the value of experience in the workforce and subsequently intuition based decisions. Where automated decisions are not feasible or possible, data should be used to inform intuition, and hopefully craft it to result in improved decision making.