Language Modeling from Scratch
Build a working language model end to end (tokenizer, transformer, training loop, the whole thing) with no libraries hiding the magic.
AI tutor is turned off for this class
Use the CS336 lectures, notes, and assignments below to keep learning.
Big Idea
Natural Interaction
Grade bands
K-2 · 3-5 · 6-8 · 9-12
AI literacy pillar
How AI works · Ethics
Lesson overview
Build a working language model end to end (tokenizer, transformer, training loop, the whole thing) with no libraries hiding the magic. This module climbs from an everyday intuition to the real mechanism, then names the Stanford course it descends from.
Teacher script · ~45 min
- 0–5
Hook
A language model finishes your sentence. Give it 'The capital of France is' and it answers 'Paris,' not because it looked it up, but because in everything it read, those words almost always come next. Scaled up enormously, that one ability, predict the next piece of text, produces essays, code, and conversation.
- 5–15
Explore
Students do the activity in pairs: Invent a code where common letter-pairs (th, er, ing) each get one symbol. You just sketched 'byte-pair encoding,' how real tokenizers work.
- 15–30
Explain
To predict well, the model must decide which earlier words matter most right now: in 'The trophy didn't fit in the suitcase because it was too big,' what does 'it' mean? The transformer's 'attention' mechanism lets every word weigh its relevance to every other word. That, stacked deep and trained on oceans of text, is the modern LM.
- 30–40
Connect to the summit
Show students this is the real thing professionals build: CS336, the real thing. Build a working language model end to end (tokenizer, transformer, training loop, the whole thing) with no libraries hiding the magic.
- 40–45
Check
Run the formative check below. Anyone who can explain a key term in their own words has it.
Student activity
Invent a code where common letter-pairs (th, er, ing) each get one symbol. You just sketched 'byte-pair encoding,' how real tokenizers work.
Slides
Formative check
- 1.In your own words, what is "Token"? (Looking for: A chunk of text (word or word-piece) that the model treats as one unit.)
- 2.In your own words, what is "Transformer"? (Looking for: The network architecture behind modern LMs, built around attention layers.)
- 3.In your own words, what is "Attention"? (Looking for: A mechanism letting each token weigh how much every other token matters to it.)
Carry-away concepts
- Token
- A chunk of text (word or word-piece) that the model treats as one unit.
- Transformer
- The network architecture behind modern LMs, built around attention layers.
- Attention
- A mechanism letting each token weigh how much every other token matters to it.
- Pretraining
- Teaching a model general language ability by predicting next tokens over huge text.
From the summit · the Stanford source
You implement a transformer LM from raw components, train it efficiently on real hardware, and understand every layer from byte-pair encoding to inference.
This module descends from CS336 at Stanford. Students who climb the full ladder arrive here.