Generative AI might dream, but it cannot hallucinate

The structure of popular programs like ChatGPT don't allow them any genuine claim on reality or truth

2023 has undoubtably been, in the popular imagination, the year of AI. ChatGPT set the record for the fastest growth in users of any technology ever and worries about existential threats posed by AI have gone mainstream. However, all of these types of generative AI (which also includes art generators like MidJourney and DALL-E 2) have turned out to have an indifferent relationship with the truth. As it is commonly known, the deep neural networks involved in these programs are prone to ‘hallucination’ - they 'see' things that don't exist and behave as though clearly false things are true. A couple of lawyers in New York recently found this out the hard way, when they relied on ChatGPT for a court case and it turned out that many of the cases they used were fictitious.

However, hallucination is a vivid turn of phrase that hides some substantive assumptions. According to one dictionary definition, to hallucinate is "to seem to see, hear, feel or smell something that does not exist." In other words, you experience and take something to be real when it isn’t. To say that generative AI, like ChatGPT, hallucinate therefore assumes they have a clear sense of what is real and what isn’t. This assumption is, however, hard to justify.

To explain, it is helpful to draw on the schema for human knowledge and truth that I developed in my most recent article. I argued there that humans seek to encapsulate our understandings of the world through various abstracted representations - theories, stories, models and even language. These are functionally distinct from reality and our direct experiences of it. Truth, including our judgements about what exists, occurs when we compare our representations with our experiences of reality. Generative AI, and neural networks, differ from this schema in two important ways.

If the inputs aren’t real, the outputs will be more like dreams than reality

To start with, all of the information inputs for ChatGPT, MidJourney and similar programs are made up of human produced language, pictures or data. In other words, all the information that these programs have access to are abstracted representations created by humans. They do not have any direct access to reality (including what we would call experiences) and so have nothing to compare these representations against.

No matter how much information these programs ingest, and the volume of data used to train ChatGPT is enormous, all of that information is only on one side of the representation / reality divide. This is a fundamental structural divide, so no amount of analysis that is purely based on representations can create any access to reality (and therefore reliable judgements about truth).

This isn’t necessarily a problem. Humans regularly operate in, value and enjoy situations where they are only worried about representations and the relationship to reality isn't important. This occurs, for example, when we are telling fictitious stories or when we are dreaming. Given all their inputs are already representations, generative AI programs operate the same way (in an ontological sense) as someone telling stories about stories, or someone dreaming. AI art generation programs often make this visually clear as their output often mimics the oddities and wackiness that pop up when we dream.

Truth depends on having a model or theory of reality

This brings us to a second structural difference. Humans build and work with a range of theories and representations that are deliberately meant to be a description or model of reality, but are structurally separate from the world.¹ Humans regularly test their models against reality in various ways and update the models as they test. Generative AI programs, on the other hand, have not been designed to build and maintain models of reality like this. They are programmed to identify (highly sophisticated) patterns within their input data and generate outputs from these.

As the schema I explained makes clear, truth depends on our ability to compare our theories and models against reality. This means that as generative AI programs have no model of reality (or theory of the world) there is no basis on which they can make any substantive decisions about what is true (or not). Put simply, they cannot genuinely tell the difference between fact and fiction because they have no cohesive theory or model of what is true. This provides a second reason why such AI cannot hallucinate: they cannot mistake something fake as real because they have no sense of what is real.

This structural feature of generative AI is partly a design decision (albeit one made for good reasons). There are computing systems that do set up a model or representation of reality and test it via a type of experiences of reality. The software used in driverless cars is a good example. My understanding is that these systems build an internal 3D model of all the objects around the car and continually update it based on the data coming in from all of their sensors.² Notably, these models are dynamic and continually predicting what different objects are likely to do - which gets corrected if the prediction doesn't match sensor data.

Structurally, this is the same dynamic that humans use - we also maintain an internal representation of the world and update it by testing. It could therefore make sense to say a driverless car hallucinates if it generates something in its internal 3D model that both doesn’t exist in reality and isn't quickly corrected by the information from its sensors.

Generative AI programs, like ChatGPT, however differ from this situation in two important ways. They don't build and maintain a model or theory of reality to use as the test of truth. And all of their information inputs are indirect - they are other representations - rather than any direct experience of reality. This doesn’t mean they aren’t useful - if we are confident their inputs are reliable, then they will produce reliable outputs. But it does mean these programs operate permanently in the realm of representations, theories and models, and so are (ontologically) only ever dreaming - never hallucinating.

1. Every scientific theory is an abstract representation of this sort. So are all of our theories about what happened in any contentious political situations. The list is endless.

2. See for example the flowchart here: https://www.researchgate.net/figure/fig3_304657131 My understanding is based on presentations I have heard that aren’t available online.