Which parts of AI systems are the sources for information?

We need to look more closely at the 'who' in an LLM we are treating as a reliable source of information

Photo by Mohamed Nohassi on Unsplash

I was recently talking to an academic who subscribes to this Substack. Last semester he gave his students the group assignment to analyse whether an AI system (or Large Language Model) has a worldview. Unexpectedly, different groups used the same LLM but came back with very different results - and had good evidence for their conclusions. Apparently, different users’ histories of using the relevant LLM shaped how it responded to questions about its worldview.

This behaviour is not surprising if we consider the way AI systems are designed to respond to users and learn from ongoing interactions. However, it has interesting consequences for whether LLMs can have knowledge that I missed in my previous articles.

Exactly ‘who’ are we relying on

Traditionally, knowledge has been philosophically analysed in terms of what it means for a person A to know that P. There are various ways we could critique this method of analysis but it has a key advantage as the key terms are clearly defined. The example we have just seen, however, shows that it doesn’t hold when we are considering AI systems and LLMs. For these systems, who exactly is the relevant A? Is it the underlying system/ technology - e.g ChatGPT, DeepSeek or Grok? Or is it a specific release, model or version - e.g. ChaptGPT 3o, o4, or 5? Or is it the specific instance of an LLM tied to my account, or the specific AI agent being used?

Different answers to this question will have consequences for Large Language Epistemology, but also for how far we trust LLMs as reliable providers of information or knowledge. In human contexts, whether we believe some claim that P is reliable knowledge depends on the person A who made or demonstrated the claim. There is no reason to think this will be different for LLMs, and so we need to pay close attention to exactly what scale or type of AI system we consider to be discrete units that might ‘know’ different things.

As a starting point, it is worth noting the way that people most commonly refer to LLMs. My observation is that people will most commonly attribute information to ChatGPT or Grok or Claude - that is, to the whole company or system. While this is a natural consequence of how these systems are branded and sold, it should only be a shorthand reference. There are often notable differences between different releases or models developed by the same company or within the same broader system. For example, OpenAI released ChatGPT-5 recently but faced enough criticism from users that they have re-opened access to the previous model ChatGPT-4o.

Given the differences between releases or models, these are a relevant unit of analysis for knowledge or considering the reliability of information. Thus we ought to be more precise when we attribute information to LLMs and differentiate whether it was, say, Claude Opus 4 or Claude Sonnet 3.7.

However, the example we started with suggests this might not be accurate either. If individual AI agents, or instances of an LLM tied to specific users, provide different information in response to the same inputs, then we can’t really treat the whole model or release as a single epistemic unit. To pick up the terminology above, each different instance or agent is acting like they are different people or different As who ‘know’ different things. So if my interaction with Grok 4 on my account is going to give different information to you, then we need to treat each different instance of Grok 4 tied to a different account as a different ‘A’ that we cannot rely on in the same way as a source of information.

There are various technical reasons why this happens. One is, as mentioned above, that LLMs are designed to respond to users and adapt to the user over time. Another is that LLMs are not deterministic - the same input can lead to different outputs in different circumstances. But in all cases it means we need to think carefully about how we analyse LLM epistemology and how we treat them as potentially reliable sources of information.

AI reliability is murky

It is fairly common to assume that we can (broadly) trust the outputs from LLMs because they have access to all human knowledge (or at least everything published online). The hunch is that the amount of information absorbed by a system like ChatGPT-5 makes it really smart and should guarantee the information it provides is reliable.

However, if different log-ins or AI agents give different information (even only occasionally) then this assumption is significantly weakened. How much is the response I get from an AI based on the full set of training data (i.e. all of human knowledge)? And how much is it shaped or trained by my previous (or current) inputs? While this question is almost always unanswerable in any precise way, the more it is the latter, the less of an independent, trustworthy source of information LLMs are. Instead of being independent, the output I get from an LLM will be increasingly a team effort between me and the AI system - for better or worse.

If this dynamic is correct, it is not widely understood, and it means that LLMs will amplify many unhelpful information dynamics that are already present online. Many people treat LLMs, whether ChatGPT or Grok or any other as an authoritative source of information. However, if people are receiving information that has been shaped by their previous inputs and statements, then LLMs will reinforce existing views with the appearance of objectivity. In other words, they will supercharge the well-known algorithmic 'filter bubble' problem and more people will end up with more extreme views over time.¹

For me, one example of this dynamic can be clearly seen in the debates about whether AI systems or LLMs are conscious. There is a steady stream of people who become convinced LLMs are conscious based on their interactions with them. But, crucially, other people are unable to replicate their experiences. The AI systems have been trained by the individual user to provide certain sorts of information and so the outputs that seem to prove consciousness may not be a general product of the system but idiosyncratic to individual users.

This all means that questions about the reliability of LLMs cannot be reduced simply to questions about the whole system or model. Instead, we need to think about the way that individual users shape or affect the reliability of the information provided. The more an output is shaped by a user’s inputs, the more its reliability is tied to what the user already knows.

All of this also complicates my questions about whether LLMs (or their instances) can be correctly said to know things, as we need to pay closer attention to what part of a system we want to analyse. It also raises the question of whether we might need to extend our concept of the person A who might know something so that the relevant ‘person’ is a combination of a human and an LLM working in tandem. These are topics to consider in future posts.

1. This tendency towards digital systems reinforcing existing views is, in my view, driven by the financial incentives in existing business models. Digital companies are always trying to increase engagement or the time spent on a service. The easiest way to do this is to give people what they think they want right now. For the vast majority of people, they instinctively prefer to have their own views confirmed and supported.