Week 4 · Season 1 · Spark
What's inside the mind
You can explain tokens, embeddings, attention, training and sampling in plain words with the small maths behind them, and give FRIDAY a personality, rules and answers you can check.
This week
- Tokens: how text is chopped up
- Embeddings: meaning as a direction
- Attention: who to listen to
- Training: loss and a slope that points downhill
- Sampling, temperature and why models make things up
How you learn
Tokenize a sentence by hand, compute one dot product on paper, and predict what changing the temperature will do before you try it.
What you build
Personality and rules: a system prompt, JSON answers your code checks, and a model chosen for the job with the reason written down.
Stretch: Compare three free models on ten questions you wrote and publish the table.
What you ship
- persona.md and validated JSON answers with tests
- MODELS.md comparing two free models
Lessons
- Words into pieces
After this deck you can cut text into pieces by hand with byte pair encoding, and explain why spelling, capital letters, numbers and some languages cost more pieces.
- Meaning has a direction
After this deck you can say what an embedding is, work out a dot product and a cosine similarity by hand, and explain why closeness is measured by angle, not length.
- Who should I listen to?
After this deck you can explain attention as each word deciding which earlier words to listen to, work out a small example by hand, and say why doubling a prompt makes attention do four times the work.
- Learning from mistakes
After this deck you can explain training as guess, score, nudge, repeat; say what a loss is and what it should be for a random guesser; and take steps downhill by hand with gradient descent.
- Dice with opinions
After this deck you can explain sampling and temperature, pick a temperature for a job, and say why models make things up and how you check them.
- Go deeper: one block, many floors
After this deck you can draw a transformer block, explain how RoPE gives attention a sense of order, say what the KV cache stores, and estimate a model's memory, including mixture of experts.
Labs
Python in the browser, next to the lessons.
Weekend build
- FRIDAY, part 2: a home and a personality
Give FRIDAY a proper home on GitHub, then a personality, rules and answers you can trust.
Skills you prove
- Backprop
- Derive backprop through softmax + cross-entropy on paper
- Attention
- Write causal self-attention, a training loop and BPE from a blank file
- Paper → code
- Read a paper in three passes and reimplement its core
Free links for this week
Hand-picked official docs, papers and talks.
- nanoGPT · repo
- Stanford CME295: Transformers & LLMs · course
- How to Scale Your Model · course
- Kaggle notebooks (free GPU) · tool
- Context windows · docs