Week 12 · Season 3 · Memory
Knowing your stuff
You can answer from your notes with RAG: compare retrieval with long context, label chunks, cite or abstain, budget search steps, and evaluate retrieval and answers separately.
This week
- Long context vs RAG: cost, delay and lost-in-the-middle
- Chunking at the seams, chunk size and context labels
- Citations checked in code, and saying "not in my notes"
- Agentic search: find, read, search again, with a step budget
- RAG evals: retrieval vs answers, faithful vs correct
How you learn
Price long context against RAG by hand, write a chunker and a citation checker from scratch, build a grep-and-read search loop with a budget, and grade a run of answers in plain Python.
What you build
Project 3 and the Season 3 boss, Ask my notes: FRIDAY answers questions about your notes with a checked receipt for every claim, says "Not in my notes." to traps, and passes a 50-question Trick Exam behind your week 10 CI gate.
Stretch: Add contextual retrieval (a model writes a line of context for every chunk) or an agentic-search mode, and keep it only if your exam scores improve.
What you ship
Project 3: Ask-my-notes. Answers from your own notes with citations, "not in my notes" when it should, and a passing eval.
- POST /ask with citations checked in code, and honest abstains
- 50 exam questions with gold chunks, 10+ of them traps
- An eval report (search and answers graded separately) behind a green CI gate
Lessons
- Just send it all?
After this deck you can price two ways of giving FRIDAY your notes: long context (send everything) and retrieval (search first, send a few pages), and pick one for a given pile, budget and question count.
- Cut it right
After this deck you can cut notes into chunks at their natural seams, pick a chunk size with your test set, and add a context label to each chunk so search can still find it.
- Show your sources
After this deck you can make FRIDAY cite the chunk behind every claim, check those citations in code, and make it say "not in my notes" instead of guessing.
- Let it go looking
After this deck you can explain agentic search (a model that searches, reads and searches again in a loop), and keep that loop safe with a step budget and errors it can read.
- Grade the answer too
After this deck you can grade a RAG system in two parts (did search find the right chunk? did the answer use it right?), sort every wrong answer into a search miss or an answer miss, and build the grader for your boss fight.
- Go deeper: sharper retrieval, honest scores
After this deck you can explain contextual retrieval with model-written context, when grep-style agent search beats an index, and how to score faithfulness as claims supported ÷ all claims.
Labs
Python in the browser, next to the lessons.
Weekend build
- Project 3: Project 3 and the Season 3 boss: ask my notes
FRIDAY answers questions about your own notes, with a receipt for every claim, an honest "Not in my notes.", and an eval it has to pass.
Free links for this week
Hand-picked official docs, papers and talks.
- Anthropic: Introducing Contextual Retrieval · post
- Anthropic docs: Citations · docs
- Anthropic docs: Reduce hallucinations · docs
- Anthropic: Effective context engineering for AI agents · post
- Liu et al., Lost in the Middle (2023) · paper
- Ragas docs: Faithfulness · docs
- Hamel Husain: A field guide to rapidly improving AI products · post
- Eugene Yan: Patterns for LLM systems · post