Almost everyone who sets out to "build my own ChatGPT" starts in the wrong place: by choosing the model. Two weeks of benchmark comparisons later, they pick one, put it on a GPU, wire a chat box to it — and discover that what they built looks nothing like the thing they wanted to copy.
No file context. It can't read the photo a user sends. It has no idea what happened yesterday. It doesn't generate images. It doesn't speak. And it invents numbers with alarming confidence.
The model wasn't the problem. The problem is that ChatGPT, Claude, Gemini and Grok are not a model. They are ten distinct pieces, stitched together by an orchestrator, and the chat box is only the façade.
📚 Series "Build Your Own ChatGPT" — 9 parts
1. What's inside ← you are here · 2. The brain · 3. The eyes (OCR) · 4. The memory (documents) · 5. Images · 6. Video · 7. Voice · 8. Tools and agents · 9. Putting it together
The single-box illusion
The interface is deliberately misleading. One text field, one answer, all of it apparently coming from the same place. Underneath, a single message from you can fire four different models, running on different machines, with costs that differ by three orders of magnitude.
Send "summarise this PDF and chart it for me" and the path looks roughly like this: a small model classifies the intent, an extractor reads the PDF, a text model summarises, an interpreter runs the charting code, a moderation model checks the output. Five pieces. One sentence.
The ten pieces
1. The orchestrator (the conductor)
Not a model — code. It takes the message, decides what to do with it, calls the pieces in order, assembles the parts and returns the result. It's the piece nobody sees and the one that most separates a good product from a bad one.
2. The brain: the text model
The piece that converses, reasons, summarises, writes code. It is the only one invoked on 100% of interactions — hold on to that, it's the thread running through this whole series.
3. The eyes: vision and OCR
When you drop in a photo of an invoice, a screenshot of an error or a scanned PDF, the thing reading it is not the text model. It's a vision model turning pixels into text — and only then does the text reach the brain.
4. The memory: document search
"Based on the files you uploaded" is neither magic nor infinite memory. It's a search system: documents are chunked and indexed, and on every question the relevant passages are retrieved and pasted into the prompt. The model still remembers nothing.
5. Today's information: web search
Every model has a training cutoff. Asking a bare model for today's exchange rate produces an invented number delivered with total conviction. Platforms fix this by fetching the current data before calling the model.
6. The hands: tools and code execution
Running Python, calling an API, querying a database, clicking through a website. This is what separates "an assistant that talks" from "an assistant that does".
7. The brush: image generation
A different model, a different architecture, a different machine. The text model doesn't draw — it rewrites your request into a better prompt and hands it off.
8. The camera: video generation
By a wide margin, the most expensive piece of all. Seconds of video cost what thousands of text messages cost.
9. The ears and mouth: voice
Two models, not one: transcription (audio to text) and synthesis (text to audio). The text model sits in the middle and never hears a thing.
10. The guard: moderation and guardrails
A small, cheap model that reads everything coming in and much of what goes out. Costs almost nothing and prevents the headline.
The open-source equivalent of each piece
All ten have a mature open substitute in 2026. This is the table worth printing:
| Piece | What it does | Mature open option | Where it runs |
|---|---|---|---|
| Brain (text) | Converses, reasons, writes | Kimi K3, GLM 5.2, Qwen 3.8, DeepSeek V4 | Token API or dedicated GPU |
| Eyes (OCR) | Reads images and scanned PDFs | Surya, Marker, GROBID | GPU by the hour |
| Memory (documents) | Searches your files | Qdrant + embeddings, AnythingLLM | Small GPU or CPU |
| Web search | Brings in today's data | Search API + prompt injection | Alongside inference |
| Tools | Runs code, calls APIs | n8n, Langflow, browser-use | CPU or small GPU |
| Brush (image) | Generates and edits images | FLUX, Qwen-Image, ComfyUI | GPU by the hour |
| Camera (video) | Generates short video | Wan 2.2, LivePortrait | Large GPU by the hour |
| Ears (speech→text) | Transcribes audio | Whisper | Small GPU |
| Mouth (text→speech) | Synthesises voice | F5-TTS, XTTS v2, OpenVoice | Small GPU |
| Guard | Filters input and output | Small model + rules | Alongside the brain |
The question that decides your bill
Here is the point almost nobody works out before building the infrastructure, and which gets expensive later: how often is each piece actually called?
The numbers below come from the typical usage pattern of a corporate assistant. Your distribution will differ, but the shape won't:
| Piece | % of interactions | Load pattern | Best home |
|---|---|---|---|
| Brain (text) | 100% | Constant, unpredictable, spiky | Per token |
| Memory (documents) | ~40% | Alongside the brain | Small always-on machine |
| Eyes (OCR) | ~8% | Bursty — a batch of documents arrives | GPU by the hour, on demand |
| Web search | ~15% | Constant, but it's an API call | Alongside inference |
| Brush (image) | ~3% | Bursty | GPU by the hour, on demand |
| Voice | ~2% | Short bursts | Small GPU, on demand |
| Camera (video) | ~0.2% | Rare and heavy | Large GPU, only when needed |
💡 The rule that falls out of that table
Constant and unpredictable → pay per use. The brain is called all day long, at hours you don't control. Keeping a GPU running for it means paying 24 hours a day for a card that is idle most of them.
Rare and heavy → rent by the hour, only when needed. OCR, image and video arrive in bursts: a thousand documents at once, a campaign batch, one video a week. You start the big machine, process, shut it down, and pay only for the time it ran.
Why the brain is unlike every other piece
It's counter-intuitive, so here is the explicit arithmetic. An internal assistant used by 40 people generates something like 2,000 messages a day. In a typical conversation each message consumes around 3,000 input tokens (history plus retrieved context) and 500 output tokens.
That's 6 million input and 1 million output tokens a day. Sounds like a lot. In GPU terms it is almost nothing — a 24 GB card handles it comfortably. The problem is that those 2,000 messages don't arrive evenly: they arrive between 9am and 6pm, peaking mid-morning and after lunch.
So you pay for 24 hours of card to genuinely use about three. And if the peak doubles on a month-end day, the card can't keep up and the queue grows.
⚠️ The classic first-build mistake
Standing up a dedicated GPU for the text model before having predictable volume. The maths looks lovely on paper ("R$ 2.78 an hour, that's cheap") and arrives as R$ 2,000 a month of idle card — for consumption that would cost a fraction of that paid per token.
The inversion is real and it does happen: above a few million messages a month, with steady load, the dedicated GPU wins. Part 9 of this series runs that number.
The right order to build in
Nobody builds ten pieces at once. The order that works:
- Day 1 — the brain. An API key, one call, a chat screen. You have a product that answers. If you stop here, you have solved 70% of real use cases.
- Week 1 — the memory. Your documents, indexed. This is what turns "another chatbot" into "our company's assistant".
- Week 2 — the eyes. If your business receives PDFs, invoices, contracts or photos, this is the piece with the highest value per hour of work saved.
- After that, as demand asks. Tools, image, voice, video — almost always in that order.
And one piece of advice that saves months: don't build a piece before someone asks for it. Video generation is the most fun piece to build and the one fewest people use.
What the next eight parts cover
Each part takes one piece, explains plainly how it works underneath, shows which open option to use and what it costs to run — with numbers you can check:
- Part 2 — the brain: choosing a model, what context really is, why the answer costs more than the question.
- Part 3 — the eyes: real OCR, what breaks on messy documents, cost per page.
- Part 4 — the memory: why "just send the whole PDF" doesn't scale, and how document search actually works.
- Part 5 — image · Part 6 — video · Part 7 — voice · Part 8 — tools and agents.
- Part 9 — putting it together: the full architecture, the model router, and the monthly bill of three real products.
Start with the piece that solves 70%
The brain of your product can be answering today: five open models behind an OpenAI-compatible API, billed per token, with no subscription and no machine to administer.
Create your account →Next: the brain — choosing and paying for the text model.
Keep reading: part 2: the brain · a hands-on ChatGPT clone · AI infrastructure guide