Week 10 · Season 3 · Memory
Grading the grader
You can grade FRIDAY with pass or fail judges, measure a judge's true positive and negative rates, report a score with its interval, and block a worse version in CI.
This week
- Binary judgments: pass or fail, not 1 to 5
- Judge the judge: TPR and TNR
- Sample size and confidence intervals
- Repeated runs: pass^k and pass@k
- CI with GitHub Actions: evals as a gate
How you learn
Count a judge's four boxes, compute TPR and TNR, bootstrap a confidence interval, estimate pass@k and pass^k, and write the gate script that exits 1, all in plain Python.
What you build
Give FRIDAY code checks and one AI judge, measure the judge on answers it never saw, publish an eval report with n and a 95% interval, and add a GitHub Actions gate that refuses a pull request that makes FRIDAY worse.
Stretch: Run every test question five times and report pass^5 next to the plain pass rate, or correct the judge's pass rate with its measured TPR and TNR.
What you ship
Project 2: An eval report and a CI gate. 50 real questions, a judge checked against your own labels, and a CI gate that blocks a worse FRIDAY.
- EVAL-REPORT.md: score, n, 95% interval, results per check
- A judge report with the four counts, TPR and TNR on a held-out split
- An evals workflow that blocked a worse FRIDAY in a pull request
Lessons
- Pass or fail, not 1 to 5
After this deck you can explain why a binary judgment (pass or fail) beats a 1-to-5 score, and write a judge instruction that checks one failure, gives a reason first, and replies PASS or FAIL.
- Judge the judge
After this deck you can sort a judge's verdicts against your own grades into four boxes, compute its true positive rate and true negative rate, and explain why plain agreement can hide a useless judge.
- How sure are you?
After this deck you can explain why small tests wobble, find a confidence interval with the jar trick (resampling with replacement), and say how many questions you need for a given precision.
- Lucky once, reliable always
After this deck you can explain why one try proves little, compute pass^k (every try right) and pass@k (at least one right), and choose the one that matches the promise you make.
- A robot that checks every change
After this deck you can explain continuous integration, read a GitHub Actions workflow that runs FRIDAY's evals on every pull request, and write a gate script whose exit code blocks a worse version.
- Go deeper: honest numbers
After this deck you can correct a judge's pass rate with its measured true positive and true negative rates (calibration), carry its wobble into a bootstrap interval, and run a paired comparison between two versions of FRIDAY.
Labs
Python in the browser, next to the lessons.
Weekend build
- Project 2: Project 2: FRIDAY's report card and gate
An honest eval report for FRIDAY, and a robot in CI that refuses to ship a worse FRIDAY.
Skills you prove
- Judge
- Build an LLM judge and report its agreement with human labels
- Eval gates
- Gate deploys on evals in CI, and explain pass@k vs pass^k
Free links for this week
Hand-picked official docs, papers and talks.
- Hamel Husain and Shreya Shankar: LLM evals FAQ · post
- Hamel Husain: Using LLM-as-a-judge · post
- Anthropic: Demystifying evals for AI agents · post
- Adding error bars to evals (Miller, 2024) · paper
- tau-bench: pass^k for reliability · paper
- GitHub Actions: workflow syntax · docs
- GitHub: protected branches and required checks · docs
- promptfoo · tool