“The cat is sleeping on the…” Before the sentence even ends, you have guessed: the rug, the bed, the doormat. Large language models, or LLMs, do nothing more than this, but on a scale that lets them write almost like us. How does such a simple principle produce such convincing language, and what do linguists make of it?
After “creole” and “heritage language”, the Linguodex, the EFL project’s podcast (in French), continues its tour of the vocabulary of linguistics with a third word: LLM. For this episode 3, Benoît Crabbé, professor of computational linguistics at Université Paris Cité and member of the Laboratoire de linguistique formelle, starts from the definition and works his way up to the questions these machines raise for research and for society.
First of all, a mathematical object
Before it is a chatbot, a language model is, in Benoît Crabbé’s words, “simply a mathematical object that gives a probability to any sentence one can imagine saying in a language” (translated from French).
That probability can be broken down into a series of predictions: “given a context of words, predict the next word” (translated from French), he explains. This is what makes it possible to generate text, word after word.
The next-word game
The model does not always pick the most probable word. It draws lots according to the probabilities: “The more probable you are, the better your chances of being drawn. But you can also be drawn while being a little less probable than the best candidate” (translated from French). This mechanism, he notes, is what drives public assistants such as ChatGPT, Claude or Gemini.
There is nothing mysterious about the task itself: “humans can even play the game of predicting what comes next” (translated from French). Faced with “the cat is sleeping on the…”, everyone suggests the doormat, the rug, the bed. A language model simply gives a high probability to words of that kind.
An idea more than a century old
The notion of a language model “is very old, it is more than 100 years old”, Benoît Crabbé points out (translated from French). It goes back to the Russian mathematician Andrey Markov, a specialist in probability, who “was interested in studying Russian poetry, in fact Pushkin’s poems” (translated from French). In 1913 he counted by hand how vowels and consonants follow one another in Eugene Onegin: “it was all pen and paper” (translated from French).
In the 1950s, Claude Shannon, the father of information theory, showed that humans are very good at predicting what comes next in a text. The idea has stayed central in cognitive science and psycholinguistics: “when we produce language, or even when we listen to it, we are able to predict what comes next” (translated from French).
Why it works now
If the principle is so old, why did it take until the 2020s? “It is the arrival of modern neural networks that largely made it possible” (translated from French), Benoît Crabbé answers, together with the computing power of graphics processors, which allows these models to be trained on a very large scale.
On a small scale, the results were poor. Text generated by the machine translation systems of the 1980s and 1990s looked like the target language, and little more: it sounded “a bit clunky, robot text that’s not quite run in” (translated from French). Scaling up, notably from GPT-3 onwards, produced texts that began to look like something “almost human” (translated from French).
What linguists do with them
For several decades, a link has been observed between the probability of the next word and the way we read: “if a model is very sure of the next word, people tend to read it faster” (translated from French). Signals measured by electroencephalogram can also be related to this degree of certainty, which Benoît Crabbé calls “surprise”.
Above all, we cannot open up a human brain to see how it produces sentences. But “a language model talks”, and “it’s a neural network, it can be dissected” (translated from French). Looking inside these models for mechanisms useful to linguistic theory is, he says, “an ambitious line of research” (translated from French), and a very active one today. They also allow experiments that are impossible with humans: “You can’t tell a human, here, we’re going to give you a little memory ablation” (translated from French). With a model, you can remove part of the context and measure the effect on prediction.
An artificial object, not a human being
Should we then credit these machines with a mind? Benoît Crabbé remains cautious (“I don’t know if it’s a misconception”, translated from French), but he observes “a strong tendency towards anthropomorphisation” (translated from French). An LLM produces language and simulates reasoning, but “it remains an artificial object”, and “normally a language model has no goal of its own” (translated from French).
He closes on the stakes: these models raise “societal questions, which are probably major ones for the years to come” (translated from French), with uses that may be very good or dangerous. In his view, “public research has a role to play, precisely, in informing” (translated from French).
🎧 Listen to the episode (in French): https://www.podcastics.com/podcast/episode/llm-438963/