The Primacy of Chaos?

Insights into knowledge and doubt from the science of uncertainty

I have just finished the book The Primacy of Doubt by Tim Palmer. So much of it is relevant here at Humble Knowledge that I decided it deserves an article of its own. The book is incredibly wide-ranging, in a coherent and thought-provoking way, covering mathematics, climate modelling, economics, quantum physics and relativity, psychology and philosophy. I don’t agree with all his ideas but it is the most thought provoking book I’ve read this year. This article will only sketch some core ideas and I recommend the full book for the complete experience.

By way of background, the author, Tim Palmer, has had an eclectic but serious scientific career across a range of fields. He did his PhD in physics but turned down a spot working as a post-doc with Stephen Hawking to join the British Met Office - where he was a pioneer in developing a range of weather forecasting and modelling methods that are now routine. He has produced a concise and readable book (only 250 odd pages) that reads like a summary of his life’s work and thought.

As the title suggests, limits to our knowledge of the world lay at the heart of Palmer’s work. As he puts it, the book is about “how the science of uncertainty can help us make sense of our very uncertain and unpredictable world.”¹ Based on a range of findings, primarily from the mathematical study of chaotic systems, he takes it that there are limits to what we can know and predict. However, by accepting and working with these limits, he demonstrates how we can develop a better understanding of the world than if we ignore them. This is a theme I have touched on before but the book explores it in more detail.

Core insight: the non-computability of the world

The starting point of the book, and Palmer’s thinking more broadly, are mathematical results that show that there are a range of physical systems that are not predictable. For these systems, we can know with exact precision the starting points and rules governing the behaviour of everything in the system, but we cannot exactly predict the evolution of the system over time.

The simplest example starts with classical physics and Newton’s laws of motion and is called the ‘n-body gravitational problem’. If we have three (or more) bodies in space that are close enough to influence each other via gravity (think a solar system), there is a well established mathematical result (first proven by Henri Poincaré) that shows we cannot calculate or provide a general solution for the position of the bodies into the future.

Moreover, it is possible, under ordinary rules of gravity, for systems like this to suddenly start exhibiting chaotic or unexpected behaviour without any external shocks. This line between expected and chaotic behaviour is often dependent on very small differences in the starting conditions. While we cannot solve these systems and provide equations to predict their future, we can now model simple enough systems through computer simulations (running a system through a simulation is different from calculating a prediction). If anyone is worried, Palmer reassures us that modelling of our solar system shows the planets will not start heading in different directions any time in the next couple of billion years.²

These results are the basis of what is known as chaos theory in mathematics: systems where the outcomes cannot be easily predicted from starting positions as we can only work through the system over time. Often these systems are highly sensitive to small changes - so a small change (‘a butterfly flapping its wings’) can occasionally lead to large system changes (‘a storm’).

Palmer’s career change from physics to meteorology was a result of his interest in these systems and the mathematics underlying them. What we experience as weather is produced by ‘chaotic’ systems in this sense - which is why weather forecasting is so difficult to get right. We have invested heavily in super-computing facilities for weather forecasting but as the weather remains highly sensitive (at times) to small input changes and events, perfect predictions are not possible. At current computing capacity, we are only beginning to reduce the resolution of weather models to 9km by 9km squares - which remains far too large to capture many important effects.³ It is estimated that there is a hard theoretical limit of 14 days on meaningful weather prediction.

Weather is only one of many systems that betray similar features and which are not computable. Palmer uses some examples that we have covered here previously, such as Turing’s Halting Problem, and the uncertainties of quantum physics. The lesson from all of this, for Palmer, is that we need to embrace the uncertainties, or doubts, and seek to harness them - rather than trying to eliminate uncertainty or doubt and aim for, in my words, epistemic certainty. For Palmer (using the mathematical sense of the words):

chaos is a property of the fundamental dynamics of the universe.⁴

A second insight: the geometry of chaos

If we take Palmer’s view seriously, the universe will exhibit properties that the mathematics of chaos theory has uncovered. Palmer uses mathematical language and describes these as properties of the geometry of chaos. That is, there are certain types of properties that these types of chaotic systems always have. To simplify the exposition, and avoid the mathematics of fractals, Cantor sets and Lorenz attractors, we will focus on two key features or properties.

The first is that these systems can, in certain circumstances, switch suddenly to very different states or situations - and the exact causes are hard or impossible to identify. The waterwheel (with deliberately leaky buckets) in this video illustrates this dynamic nicely:

The waterwheel changes the direction it is spinning in seemingly at random and definitely unpredictably over any significant period of time. The transition is never smooth between directions and not spinning is not a stable state for it. The behaviour seems spooky or random, but it is all governed by simple and well understood processes.

We are all familiar with this type of behaviour when we consider the weather. Take summer afternoons in many parts of the world: the weather will stay hot and sunny, or a storm will blow in and there will be significant rain. Similar days will end up very differently for no obvious reason.

A second feature of these chaotic systems is that, in many circumstances, the possible outcomes are discrete. That is, there are possible states the system can be in, but it cannot be in any of the situations in between those states. Thus the chaotic waterwheel turns fairly quickly in one direction or another, but doesn’t slowly turn. Or the summer afternoon either is sunny or a rain-storm. It won’t turn gently overcast and cool down.

The mathematical results behind this are fascinating as this discrete nature - some outcomes are possible but the ones in between aren’t - applies at all scales and resolutions.⁵ It isn’t just about whether there is a storm or not, but also the amount of rain or length of the storm. Thus the storm might produce 2mm, or 5mm, or 8mm of rain, but not any value in between. And the discrete nature of possible outcomes continues at every smaller level of detail.

These two properties of chaotic systems, or features of the geometry of chaos, are treated by Palmer as fundamental to understanding the world. He draws on them to explain phenomena from weather to quantum mechanics. We will restrict ourselves here to implications for knowledge and practices around building knowledge.

A chaotic universe requires humility

Both of these insights, that the world isn’t computable and that it often matches the geometry of chaos, place hard limits on what we can know. For a start, the first principle means we cannot devise formulae (or algorithms) to predict outcomes we want to understand. The best we can do is partially simulate the world to see how it might end up.

The second insight means that significant events in the world can occur (think the waterwheel changing direction) without there being any significant knowable cause - especially as the world can flip between discrete states without staying in anything in between. Moreover, the geometry of chaos means that the world can be incredibly sensitive to small changes of inputs in a way that is not easily knowable.

Taking these two together means, on my understanding, that the only way we could know and predict many features of the world with complete certainty is if we could model an exact replica of the universe down to sub-atomic levels. This is clearly a practical (and likely theoretical) impossibility.

To put it in our words: knowledge is hard and we will get it wrong. We should be humble about what we know. However, this is only the starting point of Palmer’s thinking. He provides interesting examples of how we can harness uncertainty to give us better knowledge of the world.

A practical solution: ensemble modelling

As most of his career was focused on meteorology, Palmer’s thinking is inspired by work he lead there. Essentially, instead of using weather models to make a single ‘best guess’ prediction, he helped pioneer running model ensembles. This idea is quite simple: run the model (or models) multiple times with slightly different starting points and look at the outputs of the collection of runs. It is now standard practice.

Conducting multiple runs of models is a response to the non-computability of the world. We cannot get it right, so cover a range of sensible options that might be right. It also helps us identify which of the possible discrete outcomes are likely, and whether flipping into a different state is possible.

This approach gives two types of useful information. One is it provides a range of plausible outcomes (i.e. they start from current observations) and can weight the likelihood of these. So when you read today that there is a 20% chance of rain on Monday, that technically means that it rains in 20% of the times they ran the weather model.

The second type of information is that it tells us how stable or unstable the weather system currently is. Sometimes all the runs in the ensemble give similar outcomes. Other times they diverge rapidly - as hurricane or cyclone prediction maps can sometimes show. Understanding this uncertainty at play is itself useful for making decisions and understanding what we can predict.

This is an application of a fairly intuitive human approach, although it appears to have been less intuitive to many scientists. Their training stressed the importance of getting the single, right answer and the precise mathematics involved lead to expectations of precise, accurate answers.

The intuitive idea is that running an analysis multiple times, possibly from different perspectives, should give us more robust outcomes. We instinctively apply a similar approach when we seek out multiple perspectives on any particular problem. And we will often note that the range of responses gives us useful information, not just the content. Notably, these approaches assume a level of epistemic humility: no one person or analysis can get it all right, so we need to look at a range.

Through the middle part of the book,⁶ Palmer takes this idea of ensemble modelling and considers what it might look like when applied to some other fields, particularly to pandemics, economics and conflict or war situations. He freely admits that he is dabbling outside his area of expertise and while it provides some very interesting ideas worth exploring, in my view, some weaknesses also show.

Essentially, he argues that these fields should explore formal ensemble modelling approaches that build somewhat bottom-up models of the areas of interest. A key finding in a number of fields is that, over time, ensemble approaches almost always outperform any single model - hence his suggestion of expanding the approach. Versions of this in epidemiology evolved over the course of the covid pandemic but applications to economics and conflict studies are more speculative.

Whether or not his specific ideas have value, and I see a more limited role for mathematical modelling in systems where social dynamics come into play, the success of these approach depends on different norms of communication around knowledge. We are used to probabilistic weather predictions and broadly know how to take them into account. Palmer’s approach, or any one that takes the limits to our knowledge seriously, depends on us accepting and communicating the uncertainties in our knowledge. Experiences during the pandemic, for example, showed how much we demand certainty even when it doesn’t exist.

A less intuitive solution: add noise into analysis

While ensemble modelling was the focus of much of the book, Palmer explored a second way of using the science of uncertainty in calculations and decision making. In cases where we can accurately predict outcomes, we need to make sure our inputs are as precise as possible. However, in the many real world situations where chaotic systems reign, Palmer points to the effectiveness of a range of techniques that deliberately introduce noise into the inputs.

There are mathematical algorithms that deliberately introduce noise either by adding in randomness or by reducing the precision/resolution of the calculations. These can often more accurately model chaotic systems or produce efficient (but not perfectly accurate) algorithms using vastly fewer processes or steps.⁷ Palmer speculates that similar processes underpin how the human brain works.⁸

The details are beyond this article but the idea is intriguing, especially given the evidence behind it. Given that we cannot know or predict with certainty, seeking ever greater precision and detail turns out (at least in some circumstances) to consume greater resources for similar (or sometimes worse) outcomes.

Does that mean that too much information, or too much detail, can lead to worse analysis and worse decisions? Can we sometimes be better off taking a higher level, lower resolution view of a problem? Palmer’s account suggests that sometimes the answers to these would be ‘yes’.

Uncertainty and daily life

In summary, The Primacy of Doubt starts with an argument that there are limits to what we can know and predict - due to the chaotic nature of the systems we live in. In order to know more about the world, Palmer argues that we need to embrace this uncertainty (i.e. be more humble). In his approach, this involves running multiple varied modelling approaches and deliberately introducing noise, rather than aiming for singular outputs and exact precision.

From a philosophical perspective, however, I’d argue there is an ongoing weakness to the book. Palmer uncritically holds the perspective of a physicist who is almost purely interested in physical processes and assumes everything can be explained by them. This means that he failed to reflect on what his conclusions about doubt and knowability mean for these starting assumptions.

Nevertheless, it does not undermine the scope and quality of the book. It is a deeply interesting read that provides practical ideas about how to incorporate humility about our knowledge into decisions.

To finish, I should note that this article has only summarised his core ideas and deliberately left some of the more fascinating topics in The Primacy of Doubt out. As these are all topics I have already written about, in time, I will likely come back to his new philosophical interpretation of quantum physics, the intriguing account of human mental processes and the claimed implications for our thinking on consciousness and free will. Before then, I would recommend reading the book yourself!

1. Palmer, T., The Primacy of Doubt, Oxford University Press, 2022. p. xi

2. The Primacy of Doubt, p. 15

3. The Primacy of Doubt, p. 105

4. The Primacy of Doubt, p. 24

5. In mathematical terms, this is because the possible solutions form fractal sets.

6. The Primacy of Doubt, pp. 133-180

7. The Primacy of Doubt, pp. 47-64

8. The Primacy of Doubt, pp. 221-226