Poorly Structured Notes on AI Part 7 - Gopnik et al

One widely repeated claim is that LLMs are information “retrieval” systems. Often the person who says this has tried to guess how they work by drawing an analogy with other familiar technology. But sometimes this claim comes from extremely well informed people with expertise in machine learning. Consider this paper from Science, March 2025:

• Large AI models are cultural and social technologies

The authors are no slouches. Gopnik is a renowned psychologist (particularly in child development and learning) and cognitive scientist. Shalizi’s career has spanned physics, cellular automata, statistical analysis and machine learning. The other two authors add insights from social science.

Some statements jumped out at me:

Some AI systems—in robotics, for example—do attempt to instantiate similar truth-finding abilities. There is no reason, in principle, why an artificial system could not do so at some point in the future. Human brains do, after all. But at the moment, all such systems are far from these human capacities.

Why robotics? A robot interacts with its environment and so gets immediately feedback on whether its attempts to model its environment are successful. This, I assume, is what is meant by instantiating truth-finding abilities, and so by “robot” we here mean an AI that has some way to test its own assertions against an external reality, performing actions that will yield new information that the AI can react to. So a line is being drawn between such attempts (even by robots) and the abilities of humans. But I’d note that by this definition, an LLM meet this definition of a robot once it is hooked up with the ability to execute tools, which has become far more commonplace since this paper was published.

But large models are fundamentally different from intelligent agents, and scaling will not change this. For example, “hallucinations” are an endemic problem in these systems because they have no conception of truth and falsity (although there are practical steps toward mitigation). They simply sample and generate text and images.

I can’t help groaning just a little at this kind of drastic dismissal. While no one would deny that hallucinations are easy to come by, it is easier still to find examples of an LLM actively identifying contradictions, both in information provided to it by the user and in its own prior conclusions (though sometimes it needs a nudge to check).

What do we mean by truth and falsity? We say a theorem is true if it is consistent with whatever system of axioms we’re working within. Likewise we say a statement about reality is true if it is consistent with reality, but all we have of reality is information derived from our senses. That information (which may itself be unreliable) is the raw material we interrogate for inconsistencies. Ultimately it’s a kind of democratic process; where two possible conclusions are contradictory, we reject the one with the weakest support. We can be optimistic about our chances if we have a constant stream of fresh raw data from reality.

If we can analyse such streams of information and identify the inconsistencies, then we demonstrate the necessary equipment to identify truth and falsity. If we don’t do this with 100% reliability, well, we’re only human, after all. And yet all of this is also true of LLMs. They get the answer wrong in different and surprising ways compared to typical people, but they also get the answer right impressively often. It strikes me as unreasonable black-white thinking to try to draw a hard line, with sensible humanity’s firm grasp on reality on one side and silly old Claude doing the backstroke through Wonderland on the other. The bleed between them is (apart from anything else) really interesting. Besides, have you met many humans? How was their grasp on reality?

And that last sentence. “They simply sample and generate text and images.” I hope there was a draft where they put “(!)” after the word “simply”, and they should have left it in.

Similarly, I wince slightly at the two uses of that favourite word of the motivated simplifier, “really”, in this part:

The relatively simple though powerful algorithms that allow large models to extract statistical patterns from text are not really the key to the models’ success. Instead, modern AI rests atop libraries, the internet, tens of thousands of human coders, and a growing international world of active users. Someone asking a bot for help writing a cover letter for a job application is really engaging in a technically mediated relationship with thousands of earlier job applicants and millions of other letter writers and RLHF workers.

Let’s stop at libraries: they alone are sufficient to make the point, given that you could train an LLM on the contents of a library. Now, what’s the difference between on the one hand a library and on the other a person who has attempted to read all the books (once) in the library? The library is a passive, static but reliable store. The person is a hugely unreliable store, hardly a store at all. They will have retained fragments, flavours, will have confused parts of one book with another, but it is possible that they will, in attempting to summarise what they’ve learned, produce a synthesis that is possibly garbage or possibly points in an interesting new direction. This is a thumbnail sketch of what it is to be a student, after all.

But again this description would also work for an LLM. And again, I’m not claiming an LLM is fully a person or anything so absurd; an LLM doesn’t even have a hairstyle. I merely point out that if you wish to find some hard dividing line, this is probably not the place to look for it. If the impressive capabilities of LLMs are merely a wafer-thin dusting on the academic achievements of human society, then so are the academic achievements of every generation of students only a thin layer of sediment laid down on the existing strata. We’re all standing on the shoulders of giants, right?

Is it possible that LLMs could mash together certain pieces of knowledge in such a way as to yield important new insights, and perhaps do this at a similar rate to humans, if deployed well? The authors do actually consider this:

Large models are surprisingly effective at abstracting subtle and nonobvious patterns in texts and images. This suggests that such technologies could be used to find patterns in text and images that crisscross the space of human knowledge and culture, including patterns invisible to any particular human… By wiring together so many perspectives across text, audio, and images, large models may allow us to discover unprecedented connections between them for the benefit of science and society.

We might even consider charitably amending that last part to say “large models may themselves discover unprecedented connections…” Unless of course we’re about to abandon this line of thought and beat a rapid retreat back to a more stringently dismissive point of view:

Of course, as we note above, there may be hypothetical future AI systems that are more like intelligent agents, and we might debate how we should deal with these hypothetical systems, but LLMs are not such systems, any more than were library card catalogs or the internet.

The aim of the paper is to reset our language and thinking around LLMs, to regard them as like earlier revolutionary information technologies like the printing press, which enabled the transmission of our intellectual products with far greater fidelity, enabling ever more rapid progress driven by our powers of thinking. To reach this simplistic conclusion, we have to suppress easily identifiable facts about the actual capabilities of LLMs and how they aren’t in fact anything like fibre optic cabling.

In the analogy of a high-fidelity medium enabling communication between human thinkers, LLMs are not the media, nor are they the humans, but they are somewhere between the two. Leaping to either of those two extreme conclusions is misleading and unhelpful.

One other thing of note: they bring up Hayek’s concept of price signals as the interface between market operators. I find this a little funny in a paper partly aimed at calming the hyped-up fears of a super intelligence that takes over the world and we can’t switch it off. I’d contend that the market system is just such a system! It’s a form of globally distributed intelligence that plays a significant role in directing our lives, of which no one can fully grasp the workings, in which (when we do what the price signals tell us to do) we are little more than neurons. How would we go about unplugging that? What would happen afterwards? Cue the music!

Thanks to my AI assistant (yes, that’s real life now!) I now know of this paper from mid-2024, which is a provocative response to earlier output of Gopnik and others. In generating problematic examples for “bibliotechnism” (their name for the Gopnik position) they weren’t quite as playful as I was in my early experiments, such as:

• “Write a biography of the British newsreader Kenneth Kendall in the voice of a pirate.”
• “Write some advertising copy for a Lego set of a castle that releases a unique Taylor Swift song generated by AI when the last piece is put into place.”
• “I have tucked my nose behind my ears to see if I can smell sound.”

Some of the philosophising in the paper is of the kind that tries my patience, but briefly:

• If a machine starts to become terribly good at something, it is pedantic to continue insisting that it is only pretending to do that thing.
• While you may hold that an LLM can only produce often-erroneous mashups from the data it was trained on, that is how most of us produce content, so while you may have a point somewhere, this ain’t it.

As I said at the beginning (before I knew the word), usually when someone takes sanctuary in this bibliotechnism, I suspect they have taken a wild, ill-informed guess at the internals of LLMs. They might imagine the LLM uses a kind of fuzzy search mechanism to find an item from their training data that most closely matches the user’s request, and that is the basis of the superficially impressive parlour trick. The implication is that the appearance of intelligence is entirely due to the human-contributed intelligence in the training data, being recalled parrot-fashion. Sometimes there will be a grudging acceptance that it might “find” more than one suitable item in the training data, and perform some kind of fuzzy mash-up operation to combine them into a single response. Of course, even that would be a task requiring a remarkable degree of intelligence. If you took alternate words from two streams of text so as to merge them like a zipper, the result would be gibberish. To do it at all well, you’d have to use something like discernment, taste, understanding.

But at least one of the of the first paper’s authors is extremely well acquainted with how LLMs actually work, having taught a course on it this year. He started making rough notes on them in 2023 and as a mere dilettante I am very encouraged to find my own feeble attempts to interpret the attention jargon are like pathetic, garbled echos of an actual subject-matter expert getting properly acquainted with the same thing years earlier.

Could it be that such accomplishment in the details, bearing down on the eyepiece of the microscope, looking primarily at the low-level machinery, especially when you already tried some variation of that machinery and in your experience it definitely kind of sucked, could this tend to produce a heightened scepticism about how successful someone else has been via almost the same route? He is creditably (and amusingly) open to this possibility.