How do LLMs work?
How do LLMs work?
A machine that guesses the next piece of text. Very, very well.
Mary had a little
lamb
lamb
dog
bit
Made-up odds, to show the idea.
You guessed it. An LLM does the same, with odds it learned from text.
69%
of letters guessed right
In 1951, Claude Shannon asked a person to guess a passage one letter at a time. 🔤 They got 89 of 129 letters right. Guessing what comes next is an old idea.
The weather today is
sunny, so let's go outside.
It adds one piece. Then it reads everything again, and guesses the next one.
Five and Fifteen, cut into four pieces: Five, and, Fif, teen
Five and Fif teen
each piece is a number · 50986 · 326 · 37631 · 17109
It reads in tokens. Common words are one piece. Rarer ones get cut up. These pieces are real.
token
[toh·kuhn]
noun
a small piece of text.
often a whole word, sometimes part of one.
in English, about ¾ of a word on average.
From What are tokens and how to count them? and Glossary (Claude Docs)
Same sentence. Different price. 💸
Payment is due today: 4 tokens in English, 7 in Japanese, 6 in Korean.
Japanese and Korean usually need more tokens than English. That costs more and fills the memory faster.
Counted with GPT-4o's tokenizer. A 2023 study found 2.3 times as many for Japanese and 2.4 for Korean with GPT-4's older one.
From tiktoken, the open-source tokenizer (o200k_base) and Language Model Tokenizers Introduce Unfairness Between Languages
strawberry, cut into three pieces: st, raw, berry
st raw berry
302 · 1618 · 19772
How many r's in “strawberry”? 🍓 Hard to say when you never see the letters, only the pieces.
From tiktoken, the open-source tokenizer (o200k_base) and The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models
Each token gets a spot on a map of meaning. 🗺️
Even the directions mean something.
A map where cat, kitten, dog and puppy sit together, car, bus and truck sit together, and apple and pear sit together. Then: the step from man to woman is the same as the step from king to queen.
Close spots, close meanings. In a 2013 study, king − man + woman landed next to queen.
Drawn flat to show the idea. Real maps have thousands of directions.
From Linguistic Regularities in Continuous Space Word Representations
2017
A paper called Attention Is All You Need.
Eight researchers describe the Transformer. Most LLMs today are built on it.
The cat sat down because it was tired
The sentence: The cat sat down because it was tired. A thick line joins it to cat; thin lines join it to the other words.
Reading “it”, the model looks back at “cat”.
Attention lets each token check the ones before it. Thick lines matter more.
…based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
Vaswani et al., 2017
It drops older tricks. It only uses attention.
The sky is
blue
clear
grey
falling
Made-up odds, to show the idea.
It doesn't always take the top word. It rolls weighted dice. 🎲 Temperature sets how wild. So the same question can get different answers.
Once upon a
time
Read
Cover the next word
Guess
Check
Nudge the dials
Training is this guessing game, played trillions of times.
15 trillion
tokens
What one model, Meta's Llama 3, read while learning in 2024. 📚 About 7 times more than the model before it.
Fresh from reading, it is not a helper yet. It just keeps the text going.
On its own, it isn't good at answering or following instructions.
So people teach it to help. 🧑🏫
People write good example answers. It copies the style.
People rank its answers, best to worst.
It learns to give answers people rank high. This is RLHF.
From Training language models to follow instructions with human feedback and Glossary (Claude Docs)
People liked the small one's answers more.
1.3 billion
parameters, trained with feedback
175 billion
parameters, no feedback
Circles drawn to scale by area. A parameter is one tiny dial.
From Training language models to follow instructions with human feedback and Language Models are Few-Shot Learners
It writes one token at a time. But it can plan ahead. 🐇
He saw a carrot and had to grab it,
His hunger was like a starving
rabbit
Researchers saw Claude pick “rabbit” before writing the line. Then it wrote a line to land there. (Anthropic's own study, 2025.)
It can sound sure and be wrong.
It predicts what sounds right. That is not the same as checking facts.
From Why Language Models Hallucinate and Tracing the thoughts of a large language model
Why would it make things up?
Tests reward guessing.
On a test, a lucky guess can score. “I don't know” scores zero. 📝 Models are trained and graded the same way.
It knows A is B. But B is A? 🔄
“Who is Tom Cruise's mother?”
“Who is Mary Lee Pfeiffer's son?”
How often GPT-4 got it right, asked both ways. Same fact, flipped.
From The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”
It has two kinds of memory. 🧠
What it learned
From training. It stays fixed until a new version comes out.
What you tell it
Your chat so far, its context window. Usually gone when the chat ends.
🔮 It guesses the next token, again and again.
🎲 It picks from odds, so answers can change.
🧑🏫 People's feedback makes it helpful.
🔍 It can sound sure and be wrong. Check what matters.
Words from the story
- LLM
- Large language model. A program that learned from huge amounts of text to predict what comes next.
- Token
- A small piece of text: a word, part of a word, or a sign. The model reads and writes in tokens.
- Training
- Teaching a model with examples. After each guess, its dials get nudged a tiny bit.
- Transformer
- The design most LLMs use today. It was described in 2017.
- Attention
- How a model decides which earlier tokens matter for the next one.
- Temperature
- A setting for how adventurous the picks are. Low is careful. High is playful.
- RLHF
- Reinforcement learning from human feedback. People rank answers, and the model learns from that.
- Parameter
- One of the many tiny dials inside a model. Training turns them.
- Context window
- Everything the model can see right now: your chat so far. Its working memory.
Where this comes from
- Prediction and Entropy of Printed English. Claude E. Shannon, Bell System Technical Journal, January 1951
- Glossary (Claude Docs). Anthropic, the maker of Claude
- What are tokens and how to count them?. OpenAI Help Center
- tiktoken, the open-source tokenizer (o200k_base). OpenAI, GitHub; we counted the tokens ourselves on 23 Oct 2026
- Language Model Tokenizers Introduce Unfairness Between Languages. Petrov, La Malfa, Torr and Bibi, NeurIPS 2023
- Linguistic Regularities in Continuous Space Word Representations. Mikolov, Yih and Zweig, NAACL 2013
- Attention Is All You Need. Vaswani et al., 2017, arXiv:1706.03762
- Language Models are Few-Shot Learners. Brown et al., 2020, arXiv:2005.14165
- Introducing Meta Llama 3. Meta AI, 18 April 2024 (the company's own numbers)
- Training language models to follow instructions with human feedback. Ouyang et al., 2022, arXiv:2203.02155
- Tracing the thoughts of a large language model. Anthropic, 27 March 2025 (the company's own research)
- Why Language Models Hallucinate. Kalai, Nachum, Vempala and Zhang, 2025, arXiv:2509.04664
- The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”. Berglund et al., 2023, arXiv:2309.12288
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al., 2022, arXiv:2201.11903
- The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Models. Cosma, Ruseti, Radoi and Dascalu, EMNLP 2025
The odds on the guessing and dice pages, the dots on the map and the attention lines are drawn to show the idea. The tokens and their numbers are real: we cut them with OpenAI's open tokenizer, o200k_base. Other models cut text differently.
Checked on: · Next check: