Converge
Open the app

Week 21 · Season 6 · Home

Fast and cheap

You can explain batching and the KV and prompt caches, price a million tokens, route easy questions to a small model, and fall back when a model is down.

Open week 21 in the app

This week

  • One GPU, many users: batching
  • Don't recompute the past: KV and prompt caching
  • The price of a million tokens
  • Route by difficulty
  • When the big one is down: fallbacks

How you learn

Count a static and a hop-on batch by hand, price a million tokens on paper for home, rented and cloud, then write the sorting rule, the time limit and the breaker yourself before you wire in a model.

What you build

A router: the small local model answers easy questions, a big free model takes the hard ones, and FRIDAY times out, retries once, falls back honestly and trips a breaker when a model is down. Tested with fakes, priced per thousand questions.

Stretch: Serve the local model with llama.cpp or vLLM, try 1, 2 and 4 seats with friends asking at once, and measure how batching changes total and per-person speed.

What you ship

Project 6: Teach a small model. A small open model fine-tuned with LoRA to answer in your format, with an eval that proves it.

  • A router with a cheap sorting rule, a time limit, one retry, a fallback and a breaker, with tests
  • A cost table: price per thousand questions, before and after

Lessons

  1. One GPU, many users

    After this deck you can explain batching (many requests sharing each pass through the model), why it multiplies the words a chip makes per second, how continuous batching keeps seats full, and the trade-off between total speed and each person's speed.

    10 minFull access

  2. Don't recompute the past

    After this deck you can explain the KV cache (saved work for every word so far), why it turns a staircase of work into a flat road and what it costs in memory, and prompt caching: why a repeated start must match exactly, and how to order a prompt so it does.

    10 minFull access

  3. The price of a million tokens

    After this deck you can work out the price of a million tokens on a chip you rent by the hour, explain utilization (why idle hours make tokens expensive), price one question with separate input and output prices, and compare home, rented and cloud by the price per thousand questions.

    10 minFull access

  4. Route by difficulty

    After this deck you can design routing: a cheap rule that sends easy questions to the small local model and hard ones to the big cloud model, decided by measured scores, plus a cascade that lets the small model try first.

    10 minFull access

  5. When the big one is down

    After this deck you can follow the clues in a log to find why FRIDAY froze, and fix it with a time limit based on normal slow answers, one careful retry, a fallback to the local model that says so, and a circuit breaker that stops waiting on a model that is down.

    10 minFull access

  6. Go deeper: drafts, pages and engines

    After this deck you can explain the roofline (a chip's two speed limits), why speculative decoding speeds up writing without changing the answer, how paged attention packs the KV cache into small pages, and what serving engines do for you.

    10 minFull accessGo deeper · optional

Labs

Python in the browser, next to the lessons.

Weekend build

Skills you prove

FLOPs math
Compute parameter count, FLOPs and KV-cache memory from a model config
Serving math
Do serving math: batching, quantization, prefix caching trade-offs

Hand-picked official docs, papers and talks.