Lacking certainty? Then go faster

Epistemic uncertainty should drive decision making behaviour in the opposite direction to normal practices

Photo by Saffu on Unsplash

Human knowledge is not, in all of its varied daily uses, something abstract that is disconnected from what we decide and how we act. Instead, it is better thought of as "a stock of reliable information about the world that we can act on without needing to think about it". The purpose of knowledge is to make it easier for us to act. And this close connection to action is one factor that seems to distinguish human knowledge from AI - we can go out into the world and test our ideas through action. However, this connection is not always immediate and not always direct.

In a simple world, humans would always decide how to act based purely on knowledge and facts. However, we live in a world where there are unknowns and uncertainties and our knowledge is never fully reliable. We often are confident we know things but they turn out to be wrong. To make good decisions, these uncertainties need to be incorporated into our decisions, but it is unclear how to do so. The key to this often is not in the decision that we make, but in how and how often we make our decisions.

Decision Loops

If you look through the literature on decision making frameworks, it is common to find lots of different decision cycles. The insight behind these is that any process for reliable action needs to loop through similar stages to enable us to learn and adapt as we go. In all of these, a decision process is expected to keep repeating through a cyclic structure to respond and adapt to insights or new information. One example that readers may be familiar with are the OODA Loop:

This was devised by John Boyd, a military strategist. Source

And some of my readers will be familiar with this schematic:

From the Australian Public Service Commission.

This looped approach makes very good epistemic sense as it reflects the structure I have argued humans use to build new knowledge:

However, these theories or schematics rarely grapple explicitly with what to do when we are faced with uncertainty. How do we decide or act when we are unsure about information or only have limited confidence in what we know? This is not uncommon as it is typical of human knowledge. We almost always attach implicit confidence ratings or credences to information, knowledge and recommendations. I am far more confident in my knowledge about my neighbourhood than, say, I am about the medical fact my cousin told me over drinks last winter. This is normal human behaviour, and is something that generative AI currently lacks.

Earlier this year I suggested it would make sense to add confidence ratings or credences to information and recommendations, but the idea met with a lukewarm reception. One of the problems was that I didn’t provide any clear mechanism or approach as to how this should make a difference. In this previous article, I only offered an intuitive example based on a medical diagnosis under different levels of uncertainty. What was notable about the example is one of the biggest impacts of different levels of confidence in the diagnosis was not the prescribed treatment, but the recommendations for behaviour, future check-ups and expectations of future work. The more confident the doctor was, the less he thought there was a need to check back in with the patient.

If we think about this in terms of a decision cycle structure, this means that the doctor's confidence changed the length of time for one decision making loop to finish and the next iteration to start. When the doctor was entirely certain about the diagnosis, then the expectation was that he would never have to revisit the situation. There was no need for looped decision making. The less confident he was in the diagnosis, the faster he expected to loop back through the decision - and so the sooner he would ask the patient to report back. The greater the uncertainty, the faster the cycle of decision making should be.

This makes intuitive sense as a broader principle. The less certain we are about our knowledge, evidence and our decision, then the more attentive we should be to changes and the faster we should be to revisit or tweak our plans or actions. For example, if I am building a house with a tried and tested construction technique, the builders will likely just put the house up without running many checks or structural tests. On the other hand, if we are trying an experimental construction, then the builders and architects will be checking in, running tests, revisiting plans as they go to make sure it is going to succeed.

Speed up decision making!

To set up our human and organisational decision making processes for success, this principle requires that the speed they operate at should be correlated to our levels of confidence or uncertainty. The more certain we are, the less often we need to revisit or adapt and a slow pace for decisions makes sense. The less certain we are, the more often we need to revisit our thinking and so the faster we need to be able to make decisions.

We see this play out in practice when organisations hit a crisis. The expectation under the uncertainty of a crisis is that an organisation will set up some kind of crisis response team that can rapidly absorb what is happening and make decisions. However, this rarely happens when we are facing ongoing or structural uncertainties.

My observation is that, where an organisation faces high levels of uncertainty about the facts, the evidence or the right course of action, the norm is to escalate decisions up to a higher level of authority. Common and routine tasks with high degrees of certainty get delegated downwards while decisions where there is higher uncertainty get pushed up to higher decision makers. But pushing a decision to a higher level of authority almost always leads to a slower decision making pace. There are greater expectations of due diligence, more people involved and the people making the decisions have less time available.

In other words, it is often exactly those decisions where we need to rapidly iterate our decision making cycle to learn and adapt that we shift into processes that proceed more slowly. This creates a huge structural tension in organisations between the epistemic demands for speed to deal with uncertain decisions and the organisational imperative to escalate uncertain decisions to higher and more careful decision processes. Something has to give. And, for many reasons, it is rarely to speed up decision processes.

Instead, organisations tend to either slow down the decision even further to try to achieve greater certainty, or assume a greater certainty than is really justified. Many modern building and procurement processes fall prey to the first. They get stuck in endless requirements and reviews while we try to find the certainty we need to make a decision. On the other side, research or evidence are often twisted or even fabricated to support decisions as we feel the need to manufacture apparent certainty where it doesn't really exist. Readers can surely think of examples and are welcome to discuss in the comments.

As a broader point, this mismatch between the certainty required by the pace of decision making processes and the real world uncertainties we are facing is one likely factor in the widely discussed dysfunction in many organisations and governments today. As is endlessly repeated, we live in a time period where we are facing greater upheaval and uncertainties than in the past. And yet all our decision making processes and structures are codified and regularised in ways that don't allow for speed or adaptability.

We need to reform the ways that we make decisions to better reflect the underlying epistemic structures of human action. Concepts like 'agile' and 'adaptive' are increasingly common buzzwords as they are pointing to something important. But the needed changes cannot materialise until we ensure that the decision making cycles for decisions are matched to the epistemic certainty involved. Decisions where there is greater uncertainty need to be made through faster decision making cycles, whereas organisational approaches today tend to slow them down.