Voice mode is the feature that feels most like magic and the one that most clearly exposes the piece-based architecture of this series. Because it is nothing like one thing: it is three models in sequence, and the user feels every millisecond you add.
📚 Series "Build Your Own ChatGPT" — 9 parts
1. What's inside · 2. The brain · 3. The eyes (OCR) · 4. The memory (documents) · 5. Images · 6. Video · 7. Voice ← you are here · 8. Tools and agents · 9. Putting it together
The full loop
You speak. Here is what happens:
- Listen — a transcription model turns audio into text.
- Think — the text model receives the transcript and writes the answer. It never heard anything.
- Speak — a synthesis model turns the answer into audio.
Three models, three possible machines, three latencies that add up. And here is the number that governs the whole project: above 1.5 seconds of silence, a conversation stops feeling like a conversation. That is the threshold at which the human brain reads the pause as "it froze".
| Stage | Typical latency | Where to save |
|---|---|---|
| Detecting that you stopped speaking | 200–300 ms | Local silence detection, on the device |
| Transcribing | 200–400 ms | Medium model instead of the largest; transcribe while speaking |
| First word from the text model | 300–800 ms | Streaming, short prompt, smaller model |
| First synthesised sound | 200–500 ms | Synthesise sentence one while the rest is written |
| Perceived total | 0.9 to 2.0 s |
💡 The trick that fixes latency: never wait for anything to finish
The naive implementation is sequential: transcribe everything, wait for the whole answer, synthesise everything, play. Adding worst cases, that is four seconds of silence.
The right implementation overlaps the stages: as soon as the text model has written one complete sentence (streaming, part 2), that sentence goes to synthesis and starts playing — while the model is still writing the second. The user hears the answer begin in under a second, even if it takes six to finish.
Listening: Whisper and its sizes
Transcription is the most solved piece in this whole series. Whisper is open, handles dozens of languages well, and the only real decision is size:
| Version | VRAM | Speed | When to use |
|---|---|---|---|
| small | ~2 GB | ~12× real time | Clean audio, one speaker, you only want the gist |
| medium | ~5 GB | ~6× real time | Meetings with decent audio |
| large | ~10 GB | ~2.5× real time | Accents, noise or heavy jargon |
| turbo | ~6 GB | ~8× real time | The best balance for large batches |
"8× real time" means one hour of audio is transcribed in seven and a half minutes. On a card at R$ 2.78 per hour, that is R$ 0.35 per hour of audio.
A thousand hours of meeting or call-centre recordings: about R$ 350 and five days of card time — or an afternoon, if you split the batch across several machines. It is the piece with the best return in the series, and the one fewest people realise is this cheap.
What breaks in practice
- Overlapping speakers. Two people talking at once become scrambled text. The fix is speaker diarisation before transcription, at the cost of a little more machine time.
- Jargon and proper nouns. Product names, internal acronyms and surnames come out wrong. There is a cheap fix, just below.
- Numbers and amounts. "One six five zero" instead of "1,650". Same fix.
- Telephone audio. 8 kHz cuts the high frequencies and drops accuracy. Use the large model for those.
✅ The R$ 0.02 correction that improves everything
After transcribing, pass the text through the economy model with a glossary of your terms: "correct proper nouns, acronyms and numbers in this transcript. Client vocabulary: [list]. Do not rewrite the style, only fix recognition errors."
One hour of speech is around 9,000 words, or ~13,000 tokens. On the cheapest model that costs R$ 0.02 per hour of audio corrected. It is the best quality-per-real trade in the series.
Speaking: open synthesis
On the speaking side there are more mature open options, all running on small cards:
| Model | Strong at | VRAM |
|---|---|---|
| XTTS v2 | Multilingual, cloning from a few seconds of reference | ~8 GB |
| F5-TTS | Naturalness and speed | ~8 GB |
| OpenVoice v2 | Style and emotion control | ~10 GB |
| Chatterbox | Conversational, looser prosody | ~8 GB |
One minute of narration takes 4 to 8 seconds to generate. On a card at R$ 1.27 per hour, that is R$ 0.002 per minute of audio. A 10-hour audiobook costs a little over one real of machine time.
August 2026 prices, single GPU, no scheduled interruption. Prices are in Brazilian reais (BRL), the billing currency. The catalogue moves — check the live one before committing to a number.
The stage almost everyone skips
A synthesis model reads exactly what is written. "R$ 1,250.00" becomes a literal string of symbols. "CNPJ" becomes an attempt to pronounce an acronym as a word.
Before synthesising, normalise the text — and this is another perfect job for the economy model: numbers into words, acronyms expanded, long sentences broken into speakable ones. It costs cents per hour of audio and is the difference between professional narration and robot reading.
⚠️ Voice cloning: written consent, always
Open models clone a voice from 10 to 30 seconds of reference. The technology does not ask whose voice it is — the law does.
Voice is biometric data and identifies a person. Cloning without express, recorded authorisation is direct legal exposure under Brazilian data protection law and elsewhere. Product rule: only clone a voice with documented authorisation, store that authorisation with the voice model, and allow revocation.
The use case that actually pays
Talking to an assistant by voice is what appears in launch videos. What customers buy, in practice, is this:
A company with 40 meetings a week, one hour each, records everything. Transcribing those 160 monthly hours costs R$ 56 of machine time. Running each transcript through the brain to produce a summary, decisions and an action list with owners — two passes on the mid model — costs about R$ 0.03 per meeting, or R$ 5 a month.
Sixty-one reais a month so that no meeting decision is ever lost. That is the product. The pretty voice mode is the demo.
📈 Where the value sits, again
Notice the pattern repeating across every part: capture is cheap and bursty; understanding is constant and is what matters.
Transcribing is a card running for a few hours at the weekend. Summarising, classifying, extracting actions and answering questions about the archive happens every day, all day — and that is what you pay for per token.
Capture is yours; understanding is per token
Whisper and voice synthesis deploy in one click on a card rented by the hour and shut down when the batch ends. What comes after — summarise, correct, extract, answer — is billed per token.
See the token API →Next: the hands — tools, web search and agents.
Keep reading: Whisper at scale · latency optimisation · part 8: tools and agents