Language Modeling from Scratch
Build a working language model end to end (tokenizer, transformer, training loop, the whole thing) with no libraries hiding the magic.
AI tutor is turned off for this class
Use the CS336 lectures, notes, and assignments below to keep learning.
Lesson overview
Build a working language model end to end (tokenizer, transformer, training loop, the whole thing) with no libraries hiding the magic.
Teacher script · 45 min
- 0–5
Hook
A language model finishes your sentence. Give it 'The capital of France is' and it answers 'Paris,' not because it looked it up, but because in everything it read, those words almost always come next. Scaled up enormously, that one ability, predict the next piece of text, produces essays, code, and conversation.
- 5–15
Explore
Students do the activity in pairs: Invent a code where common letter-pairs (th, er, ing) each get one symbol. Tokenizers use the same idea, called 'byte-pair encoding.'
- 15–30
Explain
To predict well, the model must decide which earlier words matter most right now: in 'The trophy didn't fit in the suitcase because it was too big,' what does 'it' mean? The transformer's 'attention' mechanism lets every word weigh its relevance to every other word. That, stacked deep and trained on oceans of text, is the modern LM.
- 30–40
Connect to the summit
Read aloud the "From the summit" section below. Ask students which ideas from today they recognize in it.
- 40–45
Check
Run the formative check below. Anyone who can explain a key term in their own words has it.
Student activity
Invent a code where common letter-pairs (th, er, ing) each get one symbol. Tokenizers use the same idea, called 'byte-pair encoding.'
Slides
Formative check
- 1.In your own words, what is "Token"? (Looking for: A chunk of text (word or word-piece) that the model treats as one unit.)
- 2.In your own words, what is "Transformer"? (Looking for: The network architecture behind modern LMs, built around attention layers.)
- 3.In your own words, what is "Attention"? (Looking for: A mechanism letting each token weigh how much every other token matters to it.)
Carry-away concepts
- Token
- A chunk of text (word or word-piece) that the model treats as one unit.
- Transformer
- The network architecture behind modern LMs, built around attention layers.
- Attention
- A mechanism letting each token weigh how much every other token matters to it.
- Pretraining
- Teaching a model general language ability by predicting next tokens over huge text.
From the summit · the Stanford source
You implement a transformer LM from raw components, train it efficiently on real hardware, and understand every layer from byte-pair encoding to inference.
This module is based on a Stanford course.