Fire up our imaginations to improve scientific research

Modern research methods and incentives are making profound new discoveries less likely. We need to think and work differently.

Photo by NOAA on Unsplash

Universities and research funding are one significant front in the ongoing cultural and political wars in many Western countries. At the moment, most focus is on various moves of the Trump administration, from cutting funding to various high profile universities to promoting new standards to ensure that government funded science is gold standard. In the heat of the moment it is easy to forget that there are longstanding concerns about the nature and standard of academic teaching and research that cut across ideological lines.

One core thread of these concerns is the ongoing slowdown in the discovery of groundbreaking new research as new discoveries typically require more time and effort than in the past. I have covered this topic in the past and I would recommend you read that post if you want some background on the topic. Here the focus will be on a particular epistemic factor underpinning this dynamic.

Lies, damn lies and research distractions

Adam Mastroianni, in a provocative post, argued that modern research methods, and specifically statistics, have worked against the discovery of new knowledge. In short, his argument is that statistical methods incentivise researchers to focus on the wrong research questions. The precision and sophistication of statistical results make it attractive to focus on fine-grained analysis of potentially small real world effects, while genuinely ground-breaking research typically involves radical big new ideas. As he put it:

stats can seduce you into studying minuscule effects that may or may not actually exist, which then allows you to spin hand-wavy theories about why those effects sometimes appear and sometimes don't.

The problem, in his view, is that "statistics are counterproductive .... because they create the illusion that there's a discoverable truth" when the real problem is that the researchers are looking in completely the wrong place. Mastroianni provides an interesting example by considering what would have happened if the researchers who identified Down Syndrome in the late 19th century had focused on a statistical analysis of the causes. As DNA and chromosomes had not yet been discovered, it was impossible for them to find the real causes but they could have spent a huge amount of effort on statistical analyses exploring factors like parental tuberculosis. This work would have appeared to provide valuable research insights yet not provide any real insight.

In short, the concern is that statistics encourage researchers to play with numbers and data to look for different effects, rather than focusing on attempts to genuinely understand what the real dynamics, systems or factors are.

This is not a critique of any individual researcher. Instead, the issue is that the norms, incentives and expectations of academic research strongly encourage certain styles of research. In my previous post, I noted that the social dynamics in research community, particularly given the importance of processes like peer review, encourage the entrenchment of existing research programs and paradigms over genuinely groundbreaking ideas.¹ We can now extend this critique to the methods we rely on in modern research. Statistics, in particular, encourages the exploration of small effects within existing paradigms, rather than the discovery and exploration of new theories, limiting the incentives that reward groundbreaking new ideas.

Imagine something new

As he makes clear in various places, Mastroianni strongly believes that we do not currently have the concepts, theories and ways of thinking to understand the world in important areas - and he focuses on his research specialisation of psychology. This is why a reliance on statistics can be counter-productive. Statistics does not provide us with any new concepts or theories. It only allows us to test if our current theories stack up against the evidence.

This line of thought means that our slowing rate of research discoveries exists because we are looking in the wrong places. We are focusing on tweaks and statistical analyses of our existing theories, whereas we really need to be building new and better theories. To put it differently, our research limitations are due to a lack of creativity and imagination. We are not looking for and rewarding new theories or ideas that are genuinely different and promising. Instead we prioritise and reward repeated statistical analyses based on what we already think is true.

In other words, an important problem in our research community is caused by the same issues I wrote about recently that are driving social polarisation and fragmentation: people are unable to imagine how the world could be different to what they think it is. In research, it means they cannot consider and imagine new theories - or be taken seriously when they try. Online, it means they cannot imagine how other people see the world.

AI will not be our saviour

There are a range of people who think that generative AI will supercharge academic research as capable AI agents will massively expand the research workforce. My arguments here, if correct, suggest this confidence is poorly grounded. As a starting point, if the over-reliance on statistics is a significant cause of poor research productivity, then relying on generative AI - which is fundamentally statistical - seems like the wrong solution.

More significantly, generative AI has another limitation that makes it equally unsuited to the task. The problem facing research is that we likely need to build some fundamentally new theories, understandings or models of how the world works. However, as I've argued previously, generative AI doesn't have any theories or models of how the world works. It can only have theories or models of the relationships between words and pictures. This means it doesn’t have the right epistemic framework to be able to build theories or models of the world, let alone imagine new and better ones.

This isn't to say that AI will never be useful for scientific research, only that it isn’t suited to solving the fundamental issues facing many areas of science. It performs exceptionally well where we know the fundamental theories or models are correct (enough) and it has a well-defined set of problems or options to work with. The well-known work on protein folding is a perfect example. The fundamental research task was to identify new, viable options out of hundreds of millions of well-defined possibilities. 'Looking inside the box' like this is well suited to AI. Building entire new boxes, as we may need to do in various research areas, is not.²

Humans, on the other hand, fundamentally see the world through theories or models of what is going on - and we have many different theories jostling for attention and validation in any different community. We are very capable of the imagination needed, whether to understand how someone else sees the world or come up with new scientific theories.

The starting point, however, is that we have to be less certain that what we think we know is correct and encourage people to play with new and radical ideas. Most will turn out to be wrong, but - like a venture capitalist investing in start-ups - we only need a few new scientific theories to pay off to make better research progress. We need greater epistemic humility, at individual, community and societal levels, so people can fire up their imaginations. If we can embed these attitudes as normal, I am confident we will see really interesting new research and a calmer societal temperature.³

1. An interesting example of how science can work differently can be found in this summary of what happened when a research funding body, the Astera Institute, decided to ban funded research from publishing in peer reviewed journals.

2. Picking the fields where we need ‘new boxes’ is fraught, as we don’t know what we don’t know until someone comes up with a better theory. However, psychology, fundamental physics and various fields of medicine or human biology are good candidates for new theories or paradigms.

3. This is a very big IF that seems unlikely at the moment. I would like to think it will be easier to achieve amongst those pursuing research than for the population writ large as these are smaller communities united around common interests, but that may be highly optimistic.