Season 3 · Weeks 9–12 · 4 weeks
Season 3: Memory.
FRIDAY answers from your own notes. Learn how professionals measure "good", then build search that actually finds things.
FRIDAY answers questions about your own notes and files. But is it right? Learn how professionals measure "good": traces, failure lists, code checks, AI judges you have checked, confidence and a CI gate (Project 2). Then build search that actually finds things, and ask-my-notes with citations and a passing eval (Project 3).
How you learn
Read failures by hand, work out judge rates and confidence, and write BM25 and recall@k in plain Python.
What you build
An eval report with a CI gate, then ask-my-notes: answers with citations that pass an eval.
The weeks
- Week 9How do you know it's good?
You can give every FRIDAY call a trace, sort failures by hand, choose code checks or AI judges, and build a test set that measures improvement.
- Week 10Grading the grader
You can grade FRIDAY with pass or fail judges, measure a judge's true positive and negative rates, report a score with its interval, and block a worse version in CI.
- Week 11Finding things
You can build keyword, meaning and hybrid search over your own notes, add a reranker, and prove which wins with recall@k and MRR.
- Week 12Knowing your stuff
You can answer from your notes with RAG: compare retrieval with long context, label chunks, cite or abstain, budget search steps, and evaluate retrieval and answers separately.
Projects
- Project 2: An eval report and a CI gate
50 real questions, a judge checked against your own labels, and a CI gate that blocks a worse FRIDAY. Weeks 9–10.
- Project 3: Ask-my-notes
Answers from your own notes with citations, "not in my notes" when it should, and a passing eval. Weeks 11–12.
Skills you prove
Tiles on the periodic table of skills that this season lights up.
- Retrieval
- Build search that finds the right passages (keywords + meaning) and cites them (week 11)
- Error analysis
- Turn 100 traces into failure modes and binary evals (week 9)
- Judge
- Build an LLM judge and report its agreement with human labels (week 10)
- Eval gates
- Gate deploys on evals in CI, and explain pass@k vs pass^k (week 10)
- Ablation
- Run a controlled ablation and publish a negative result (week 11)
Checkpoint test: Memory
After week 12. A measured gain, and answers you can trace to their source.
- Your AI judge agrees with your own labels, with its true and false positive rates reported
- Your CI gate blocked a worse version of FRIDAY at least once, and you can show the run
- Ask-my-notes cites its sources, and says "not in my notes" on questions it cannot answer
- Explain, with your own numbers, whether hybrid search beat keyword or vector search alone on your notes