Week 9 · Season 3 · Memory
How do you know it's good?
You can give every FRIDAY call a trace, sort failures by hand, choose code checks or AI judges, and build a test set that measures improvement.
This week
- Vibes don't ship: what an eval is
- Logs and traces: leave a trail
- Read a hundred: open coding and failure modes
- Code checks first, AI judges second
- A test set on purpose: real questions, dimensions, gaps
How you learn
Read a hundred traces by hand and name every failure before you write a single check. Count the piles, weigh them by harm, write the cheap code checks first, and use an AI judge only where code can't decide.
What you build
FRIDAY's first test: a trace for every call, a failure list with names and counts from a hundred traces you read, and 50 real questions covering every combination of your dimensions, with a script that prints a score.
Stretch: Generate questions for your empty cells from dimension recipes (one call per recipe card, then read every one), or send FRIDAY's spans to a trace viewer in OpenTelemetry's format.
What you ship
Project 2: An eval report and a CI gate. 50 real questions, a judge checked against your own labels, and a CI gate that blocks a worse FRIDAY.
- A trace for every FRIDAY call, and FAILURES.md: three to five failure modes with names and counts
- friday/evals/questions.jsonl with 50 questions covering every cell, and a score script with code checks
Lessons
- Vibes don't ship
After this deck you can say what an eval is (the same questions, known right answers, a score), explain why a few good-looking answers and a star rating can't see a costly mistake, and put the improvement loop in order: read real answers first, then name, count, fix and re-run.
- Leave a trail
After this deck you can say the difference between a log (one line: something happened) and a trace (the whole story of one request), read a trace's spans to find where the time went and what broke first, and list what to record on every step.
- Read a hundred
After this deck you can run error analysis on real traces: write one plain note on each failure (open coding), sort the notes into a few named failure modes, count them, rank them by how often times how much they hurt, and know when to stop reading.
- Code first, judge second
After this deck you can write a code check (one yes-or-no question about one failure mode), decide when a mistake needs an AI judge instead, plan your checks on a budget, and run cheap checks on every live answer as a guardrail.
- Test data on purpose
After this deck you can build a test set on purpose: start from real questions, list the dimensions that vary, find the gaps in the grid, fill them with made-up questions marked as such, write the right answer for each, and lock the set.
- Go deeper: recipes, codes and a shared trail
After this deck you can make synthetic test questions from dimension recipes (one call per mix, then a human read), turn open-coding notes into failure modes with a codebook and know when to stop (saturation), and say why traces written in OpenTelemetry's shared format can be read by any viewer.
Labs
Python in the browser, next to the lessons.
Weekend build
- FRIDAY's first test: 50 real questions
A trace for every call, a hundred answers read and sorted, and a test of 50 real questions that prints FRIDAY's score.
Skills you prove
- Error analysis
- Turn 100 traces into failure modes and binary evals
Free links for this week
Hand-picked official docs, papers and talks.