Converge
Open the app

Week 23 · Season 6 · Home

Give it a voice

You can build a voice loop that answers fast and lets you interrupt, keep a computer-use agent inside four walls, check whether a model fits a device, and distil a big model into a small one.

Open week 23 in the app

This week

  • Under 800 milliseconds: real-time voice and streaming
  • Let me finish! Turn detection, barge-in and latency budgets
  • Hands on the mouse: computer use, boxed in
  • Small enough to carry: on-device models and the memory sum
  • The student and the teacher: distillation

How you learn

Write a latency budget for every helper in a voice turn before you build, measure each one, and handle an interruption yourself.

What you build

Boss weekend: talk to FRIDAY out loud. Speech in, a spoken answer out in under a second for most turns, waiting while you think, and stopping when you interrupt.

Stretch: Run the whole voice loop on your laptop, or give FRIDAY one safe computer-use task inside a throwaway box.

What you ship

  • A latency budget and a voice log of 20 real turns, with the median and the 95th percentile
  • Turn detection that waits through "um…", with tests
  • Barge-in that works, shown in a short recording

Lessons

  1. Under 800 milliseconds

    After this deck you can draw FRIDAY's voice pipeline (speech to text, the model, text to speech), add up the wait before its first sound, and explain why streaming every stage makes it feel instant.

    12 minFull access

  2. Let me finish!

    After this deck you can split a latency budget between the helpers, explain how voice activity detection and smarter turn detection decide you've finished, and design barge-in so FRIDAY stops when you talk over it, but not when you cough.

    12 minFull access

  3. Hands on the mouse

    After this deck you can describe the computer-use loop (look, decide, act), build four walls around it (a throwaway box, a list of allowed sites, your approval for anything irreversible, limits and a log), and explain why everything on a screen is information, never orders.

    12 minFull access

  4. Small enough to carry

    After this deck you can say what an on-device model gains and gives up, do the memory sum (weights, notes, spare room) to check it fits, pick between a bigger model with fewer bits and a smaller one, and explain what WebGPU does for models in a web page.

    12 minFull access

  5. The student and the teacher

    After this deck you can explain distillation (a small student model trained on a big teacher's answers), build its practice set (real questions, teacher answers, checks that filter, no exam questions), and judge the result with the rule: at least 90% of the teacher's quality at no more than 10% of its cost.

    12 minFull access

  6. Go deeper: one model or three, and safer hands

    After this deck you can compare a voice pipeline with a speech-to-speech model, explain how a voice detector judges slices of audio (and why it waits a moment before calling silence), and harden a computer-use agent with least privilege, a closed network and a replayable log.

    12 minFull accessGo deeper · optional

Labs

Python in the browser, next to the lessons.

Weekend build

Hand-picked official docs, papers and talks.