Again, do LLMs really know things?

Continuing the discussion about whether LLMs have genuine knowledge

Photo by Shahram Anhari on Unsplash

My recent post on Large Language Epistemology has provoked a really interesting discussion in a few places, including the workshop where I presented it. Excitingly, there were some new ideas and questions that came up, alongside a number I didn’t have space to go into. Covering all the relevant ideas will be a project for at least a few months, but I thought I’d start with a summary of the most interesting ideas and arguments that came up on whether LLMs know anything. These will be grouped by topic.

There is one point I need to make clear from the start. My arguments were not that human knowledge is superior to AI or LLMs, but rather that it is different and we need to understand where the differences are. There are things that AI is, and will always be, better at than humans - like chess. There are other things where I think humans will always have an advantage.

Do LLMs have beliefs?

This question provoked a a lively discussion with a wide range of responses and intuitions. There are some who thought that LLMs obviously have beliefs, and others who argued that they clearly cannot. It is worth looking at the intuitions behind these different positions.

A common starting point for those who held that LLMs have beliefs is that there are very consistent patterns in the assertions that LLMs make. This means that their assertions exhibit similar properties to someone who has beliefs. If you have no beliefs then you would make assertions at random, and LLMs clearly don’t do that. So presumably they have beliefs.

A sharper version of this argument is that what LLMs are doing is functionally indistinguishable from human behaviour. When we make belief assertions, humans are simply responding to input stimuli in a way determined by the structures and chemistry of our brains. LLMs may have a different physical make-up, but their assertions are just responses to stimuli determined by particular structures. Therefore, if we are happy to hold that humans have beliefs, then we should also accept that LLMs do.

Unsurprisingly, the starting intuitions for those who thought that LLMs do not have beliefs were very different. One was that LLMs do not have a theory or concept of the world, and therefore they cannot have beliefs. On this view, the logical structure of a belief is that it is necessarily a belief about the world or that something is the case in reality. The thought was that as LLMs do not have any concept of an independent reality or world then they cannot satisfy this logical structure and so do not have beliefs.

A second intuition was the one I referenced in my article. That is, while LLMs might exhibit consistency in their outputs, they do not exhibit strong commitments to their statements once they are challenged on them or told they are wrong. If a human believes something, they will not easily change their minds. LLMs, even only if it is because they are programmed to please, do not do this.

These competing intuitions show there is a lot more to this question, so I will come back to it in a later post. While I work on this, I'll be very interested in the thoughts of my readers, so here is a poll where you can tell me what you think! Don't feel like you need to spend a lot of time deciding, your gut instinct is more than enough.

Higher fidelity to reality

One argument that came up repeatedly was that the difference between LLMs and human reasoning is one of degree rather than of kind. That is, LLMs are doing the same sort of reasoning as humans, it is just that - as expressed in the workshop - they cannot yet achieve the same degree of fidelity to the world that humans can.

This was nicely expressed as the argument that when, in the future, we can build a robot that sees, feels, hears and has the same sorts of sensory experiences as humans, then there will be no real differences between a robot like that and a human. The intuition is that current differences are due to different types of input with the result that LLMs cannot yet achieve a high enough fidelity understanding of the world.

As someone who has argued previously that AI is an alien intelligence, longer-term readers will not be surprised that I disagree. You can refer to that article for some of my reasons, but the particular question of fidelity and details is an interesting one. My view is that this argument misunderstands how human reasoning and intelligence works. We do not rely on access to highly detailed information and our knowledge isn’t constructed from high fidelity understandings of the world. Often we get powerful insights with incredibly low fidelity and low resolution types of reasoning - a loosely sketched and poorly defined concept often changes everything.

To illustrate these differences in a concrete way, it is important to remember that (we are told) the human brain operates on a constant power draw of around 20 Watts. That is the equivalent of about three new school, energy efficient light bulbs. By contrast, a single Nvidia GPU used for LLM processing, draws at least 700 Watts, and all LLMs operate using vast data centres with thousands of GPUs like this. Human reasoning is incredibly energy efficient for what it does.

This strongly suggests it is highly optimised and not based on crunching huge amounts of data. I think this matches our personal experiences of the way humans tend to think and make decisions. We tend to draw quick generalisations, use intuition or gut feeling, and get overwhelmed with too much information. Despite this, or perhaps because of it, humans are often highly capable and at times brilliant thinkers.

I have argued elsewhere that humans fundamentally store and encode knowledge in a very different way to computers. We work with fuzzy, multi-faceted concepts that we cannot precisely define. They are often low-resolution and context sensitive - trying to define the difference between a stool and a chair is a good example. This is a fundamentally different approach to anything digital or computational, including AI. All these require highly precise, tight definitions that draw on vast amounts of data. A good illustration of the difference is that a human often only needs to see an animal a couple of times to identify it. AI requires a training library of thousands and thousands of images.

Human knowledge depends on action and agency

Another theme that emerged in the discussion is the importance of the connection between human knowledge and human agency. That is, we know things because we can do things and act in the world. As a general point, we only know we can trust some information or knowledge when we have (or we are confident someone has) gone out and tested the information in the world.

By contrast, if there is no clear link between the person telling us and some way that it could have been genuinely tested in the world, then we discount it. To pick one of many examples, if my cousin tells me about some crazy thing the government is doing, I will normally discount what he is saying. However, it would be different if he is married to someone who works in the government and has a security clearance.

This question of agency and action is one where this is a clear difference, at least at the moment, between humans and LLMs. As a human, we can always choose to go out and collect more data, do an experiment, bang something, or generally test our knowledge against the world in some way. LLMs are restricted to the data that already exists in some digital form and, while they can interrogate that, are limited in what they can do.

People were divided on whether this is a fundamental limitation with LLMs. For some, LLMs have genuine agency but they are just limited in what they can do at the moment. That would mean, for example, that once someone builds a robot like the one mentioned above, there will be no difference in the abilities of LLMs and of humans to do things. By contrast, others thought that LLMs have no genuine agency so can never really choose to do things, which means they cannot build genuine knowledge.

In many ways, this is a version of the age old question of whether humans have genuine free will - so I can’t resolve it in a paragraph or two. For now, I’ll just note that there is good evidence that humans depend on their emotions to make decisions. If correct, this means we might need to train AI agents to have emotions if they are to have genuine agency.

In the article comments, Simon eloquently captured why this question of agency, and emotions, matters for our worries about the future of AI and AI safety.

I wonder if it’s our experience of (or at least our innate desire to engage in) tasting and banging things, butting heads and stubbing toes, picking things up, pulling them apart and playing with them that actually cause humans to want to do stuff that we’re afraid of AI doing (e.g. achieve control of things and people). In other words perhaps AI is as likely to want to achieve world domination as a library of textbooks, because they don’t experience the joys and frustrations of living in the world like humans do.

This is a fascinating thought and your thoughts would be appreciated. I’ll come back to it in a future post.

LLMs lack confidence and doubt

There was one idea that came out of the workshop that I think would be fascinating to try as a new way of programming LLMs (and AI). Even thought it came up independently of anything I said, it falls very naturally out of some concepts I have been writing about.

One key feature of human knowledge is that we assign what are known as credences to our knowledge. That is, we have different levels of confidence in different statements or beliefs. There are things we are certain about, and others where we only take a tentative position. Importantly, these credences are an important input into knowledge acquisition and decision making. We will make a different decision if we are certain about the relevant information than if we are only, say, ~60% sure.¹

It was observed in the workshop that LLMs and AI lack this information or architecture. They do not record any level of confidence for the assertions, statements, or sources of information they are drawing on and this limits their usefulness. It was suggested that rectifying this might be a next evolution in LLM programming that could radically increase the power of LLM reasoning, plus potentially reduce the 'hallucination' problem.

In principle, there are at least two types of confidence that LLMs could track. One that was raised in the workshop is the quantity of source information an LLM is drawing on for its response to a topic. Some fields of knowledge have thousands of credible sources, while others have a few (if any). If an LLM can track and express a confidence rating based on this difference, it should improve quality and reduce chances of confident hallucinations. At the very least, we would have some way of knowing where it isn’t drawing on much source evidence and so is more clearly making stuff up.

Another type of confidence an LLM could track is to measure the degree of agreement within the relevant source material. Even if there are thousands of credible sources, there are some fields where there is violent disagreement amongst the sources while there is consensus in others. It would be helpful for LLM processing, and for our understanding, if these differences could be tracked and flagged.

I expect, without knowing the programming well, that this could be fairly easy to produce based on the probabilistic reasoning that LLMs use. As is well known, they pick the 'most likely' next word or sentence. However, 'most likely' can range from 90+% down to 10% likely or lower (when every other option is less likely). Intuitively, when there is great consensus in a field, then the ‘most likely’ response will have a high numerical probability. If there isn’t consensus, then the numerical probability should be low.

All of this needs to be tested, but if anyone is interested in trying out this approach, I'd be very interested to hear about it. I have worked on related ideas in formal logic and reasoning as part of my doctoral research so might have something useful to contribute.

Testing LLMs, and humans

In both the article and the workshop I put a challenge out to see if anyone could design tests that we could run to help clarify the differences between LLMs and humans. One reader, Michael Wijnen, reached out with an idea by email. His idea deserves quoting in full:

Can an LLM come up with new knowledge that isn’t just a synthesis of existing knowledge? I have a test in mind for this which would be to train an LLM but only using knowledge that was available up until a certain discovery was made (probably in maths where it’s purely an exercise in reasoning), and then we try and get the LLM to make that discovery.

This would be a fascinating thing to test, but unfortunately I doubt any of my readers have the resources to conduct the experiment.² For the sake of posterity, my hypothesis is that the outcome would depend on the type of discovery. Some major discoveries involve unearthing ideas, or teasing out the logic, from within existing theories or knowledge. Others involve a fundamentally new insight that no-one had thought of before. I expect LLMs could be quite good at the first and not at all capable of the second. Maybe one day I’ll find out how wrong I am!

Of course, all of these questions are just as useful to, as noted by a workshop attendee, reflect on what humans are like as to think about how LLMs operate. In that spirit, another commentor posed a test or challenge to all of us. That is, perhaps humans often act more like the ‘spaceship dweller’ and LLMs than we should:

Another thing that occurred to me is how willing we are becoming to be like the machines. When we read the news, we 'experience' what is going on far from us, where the real lived experience might be a very different thing……. Like the LLMs, we just process second hand information and treat it like it's a real part of our lives when it isn't.

This suggests another potential difference between humans and LLMs. We accept there is a gap between how a human should act and how they do act. It isn’t clear this distinction makes sense for an LLM. I’ll add that to my list of topics for later posts.

1. After my post on this earlier in the year, I’ve heard that some people have started actively using credences or levels of confidence in their work. Thanks to those people for sharing and I’d love to hear from anyone else who has tried it out.

2. I would love to be proven wrong on this point!