Trains, catwalks and epidemiological models

How far does scientific modelling produce knowledge?

While scientific modelling has been growing in importance over many years across a range of public policy issues - climate and economic models are classic examples - the covid pandemic pushed it to the centre of public debates and consciousness. Day-to-day decisions and options depended crucially on what different epidemiological models said. What was less clear in these debates was the epistemic status of the modelling: how true and accurate was it?

While the published epidemiological models from trusted scientific institutions were often taken by governments and media organisations as definitive, this was at odds with best practice and theory behind modelling. As a number of commentators across a range of perspectives have ably explained, scientific modelling tries to illuminate real world dynamics but relies on many assumptions and simplifications.

In any context, a wide range of models can be constructed, only some of which will end up being accurate. More importantly, we often can't know which those will be in advance. To emphasise this, there is a common aphorism in statistics and modelling, attributed to the statistician George Box: All models are false but some are useful.

The very model of an ambiguous term

While there has been significant discussion about the reliability of epidemiological models (which applies similarly to economic, climate and any other similar models), it is less clear why the confusion arose around what models can tell us. There are clearly some psychological factors involved, particularly a desire for certainty in a rapidly changing and fearful situation, linguistic ambiguities were also likely important.

If you look in a dictionary, the English word model typically has over 15 distinct meanings listed - some of which are at odds in key aspects and which have very different implications. It is helpful to look at some of these to illuminate the confusions over the epistemic status of scientific modelling.

As a core concept, to quote from the Oxford English Dictionary, a model is “something that accurately resembles or represents something else.” However, different meanings of the world model have ended up carrying very different nuances and implications.

Model meanings

On one meaning, this time quoting the more concise Merriam Webster dictionary, a model is "a usually miniature representation of something." We would commonly think of a model train or plane. On this meaning, the model is smaller and simpler than the original; it resembles it in some respects but falls short in many others. For example, a model train may look like the real world original but it cannot actually run on steam or diesel and any mechanism is very different. In this case, we will typically judge the quality of the model by how well is represents the real world original on the aspects where we think it should match. And in so far as it does that, it tells us useful information about the original.

Another meaning of model, again from the Merriam Webster, is "an example for imitation or emulation." This might be a model citizen, a model argument or more commonly (but less directly) in our image saturated world, a fashion model. Notably, the relationship between the real world and the model is, compared to the previous example, flipped. The model here is the ideal and the real world is judged by how well it lines up with the ideal. If someone is less than a model citizen, the flaw is in the person not the model. Likewise, if a coat doesn't look the same on me as it did on the model on the catwalk, it is because I don't live up to the ideal.

A further relevant meaning, commonly when used as a verb, is (according to Merriam Webster) "to construct or fashion in imitation of a particular model." You might model a house, a gameplan or a constitution on someone else's. In this sense, a model is a kind of copy, with the expectation that the model and original are comparable on most measures. They are both typically of similar scale and complexity - and either one can be considered better than the other.

Moving from these common uses of the word model, we can now consider how a scientific model works, or as Merriam Webster puts it, "a system of postulates, data, and inferences presented as a mathematical description of an entity or state of affairs." In terms of its relationship to the relevant real world entity or state of affairs, is it more like a model train, a catwalk model or simply a copy of the real world?

Clearly, a scientific model is meant to describe the real world and is therefore judged on how well it matches reality. This means it is not like a catwalk model or a model citizen. So we should not treat a scientific model like the ideal against which the real world is judged. Nevertheless, there are occasions when a scientific model is taken to be the standard of truth, including by many organisations at various points in the pandemic. It is plausible that the common association of the word model with an ideal or exemplar makes this confusion more likely.

We would like our epidemiological models to be accurate copies of the real world that we can explore and use to make accurate predictions. And we often treat them like this - the model says X and therefore X will happen. However, as the articles linked above explain, in practice models can only ever be (to quote the Oxford English Dictionary) a "simplified or idealized description or conception of a particular system, situation, or process." As they are simplified, they are therefore more like a model train - also a simplified representation of the real thing.

Model trains, not supermodels

Our scientific models are, in an epistemic sense, quite a lot like model trains and planes. They are an attempt to capture certain characteristics of the real thing but in a smaller or simplified way. We judge their value as a model on how closely they capture the relevant features of the original. And just as it is possible to build a bad model plane that really isn't like the original in any real sense, it is equally possible to build a bad epidemiological model that doesn't tell us anything about the dynamics of a disease in the real world. In practice, it is much easier to build bad models than good ones - and just because we have a model, it doesn't mean we have a good one.

To build on our analogies, while we would all prefer to be building supermodels and model citizens, in practice scientific modellers are only making model trains and trying to ensure they function in some way like the originals. So next time you hear someone talk about an economic, climate, epidemiological or similar model, remember it is a model train, not a supermodel, and look for evidence as to whether it is an accurate representation of the reality it is meant to describe.

To put it differently, we need to be humble about the knowledge that scientific models give us and recognise that they commonly fall short of our hopes and expectations.