Poorly Structured Notes on AI Part 4

I complained last time about how little resemblance there is between the matrices used inside an attention layer and a traditional neural network. But in fact the attention part is immediately by a series of densely-connected feed-forward layers. This ensemble constitutes a single transformer block, and then there may be dozens of such blocks linked in sequence.

The presence of these neural layers (plus things like $\text{softmax}$ in producing the attention weights) mean that the way the information is encoded in the vectors defies attempts to think of them in a “physics” way. The way $\text{softmax}$ works is to pick out the strongest peak component and dampen all the weaker ones, i.e. to approximate a one-hot normalized vector in the standard basis. But this means that the standard basis has some special significance: rotational symmetry is broken. The representation of the vector in one basis can be said to be privileged.

Even so it’s striking how the positional encoding, using adjacent pairs of vector components to represent the cosine and sign functions, suggests that the vectors might be better thought of as complex, i.e. the vector space is over the field of $\mathbb{C}$.

In training such a model its interesting how special tokens are used. For example, by putting an extra token at the start of every sequence, always the same token ID, that location is privileged, never corresponds to a real token, and in the loss function it can be tested for its ability to predict something about the text. Thus the model learns to let all the other tokens determine the output vector for the first token, and for it to summarise the entire text somehow (e.g. it might be a measure of the sentiment of the writing.)

Likewise, some percentage of the tokens in every input stream, randomly chosen, can be replaced with a “mask” token. In the loss function these token positions are compared with the real token concealed by the mask, and so the model learns that mask tokens don’t carry any signal but are instead determined by the surrounding text, training it to be able to predict the missing tokens.

In neither of these cases is there anything special about the tokens in terms of how the model itself handles them; it’s all about how the loss function forces the model to adjust, to make the tokens serve their purpose.

• The Annotated Transformer

Random change of focus: it’s not at all uncommon, even among people promoting the benefits of using AI, to see it reductively described as glorified autocomplete. Geoffrey Hinton in 2023:

Now, let’s analyze that. Suppose you want to be really good at predicting the next word. If you want to be really good, you have to understand what’s being said. That’s the only way. So by training something to be really good at predicting the next word, you’re actually forcing it to understand. Yes, it’s “autocomplete” - but you didn’t think through what it means to have a really good autocomplete.

That is, to be very good at predicting the next word, you need some measure of intelligence. But as has been observed, we once thought you needed some measure of intelligence to be good at chess, so when machines starting beating us at chess we just move the goalposts.

Douglas Hofstadter had a regular column in Scientific American in the 80s in which he would ask questions like Can inspiration be mechanized? (September 1982):

The gist of my notion is that having creativity is an automatic consequence of having the proper representation of concepts in a mind. It is not something you add on afterward. It is built into the way concepts are. To spell this out more concretely, if you have succeeded in making an accurate model of concepts, you have thereby also succeeded in making a model of the creative process, even of consciousness… the property of being a concept is a property of connectivity, a quality that comes from being embedded in a certain kind of network and from nowhere else. Put this way, concepts sound like structural or even topological properties of the vast tangly networks of sticky mental spaghetti.

In that article he also takes on Lucas’s infamous attempt to use Godel to prove the unique specialness of humanity (later taken up by Penrose). It’s interesting that with his “vast tangly networks” he is basically talking about what was then called the connectionist paradigm, i.e. if we make neural networks big and messy enough eventually they will exhibit brain-like behaviour.

But the general flavour of what he’s saying is that there is no reason to suppose that the actual machinery of the brain cannot be transplanted to some new context, i.e. the patterns and the way the information flows are preserved while the specific implementation in electrochemistry is left behind. What’s he saying more recently?

It’s a very traumatic experience when some of your most core beliefs about the world start collapsing. And especially when you think that human beings are soon going to be eclipsed. It felt as if not only are my belief systems collapsing, but it feels as if the entire human race is going to be eclipsed and left in the dust soon… there is a certain kind of terror of an oncoming tsunami that is going to catch all humanity off guard… it also just renders humanity a very small phenomenon compared to something else that is far more intelligent and will become incomprehensible to us, as incomprehensible to us as we are to cockroaches.

• Generative AI Should Not Replace Thinking at My University

Isn’t it already incomprehensible to us? Researchers who want to know how an AI understands things have to adopt the attitude of cognitive scientists, albeit ones who are blessed with a license to (figuratively speaking) conduct open-skull brain probes on living patients.

• Tracing the thoughts of a large language model

(Worth noting that Hofstadter’s confusion over how such an impressive demonstration of thinking can emerge from feed-forward networks is well addressed in the above paper.)

Being able to construct something that can think is not the same as understanding how it thinks. The real Hofstadter research programme was to attempt the latter, whereas we’re currently all in thrall to the former.

Among the very uninformed it’s often claimed that the current batch of LLMs cannot be “creative”, while at the same time they are prone to inaccuracy because they hallucinate things that never were. So it seems that the emergence of artificial intelligence has triggered a sudden interest in the dual concepts of creativity and error, in people who have barely thought about them before now, and have never realised how much they overlap, or are mainly distinguished by a difference of perspective.