Build your own ChatGPT (3/9): the eyes — reading images and PDFs

You drag a photo of an invoice into ChatGPT and it answers questions about the contents. It looks like the model "looked" at the image. It did not — at least not the way most people imagine.

This part builds the eyes: how pixels become text, which open options to use, and above all the two mistakes that make this stage cost ten times what it should.

The decision that saves 90% and nobody makes

Before any model, one question: does this PDF already contain text?

PDF is a treacherous format because the same extension holds two completely different things. A PDF generated by a system — an e-invoice, a bank statement, a contract exported from a word processor — already contains text as text; you can select and copy it. A scanned PDF is a sequence of photographs of paper: as far as the computer is concerned there is not a single letter in there.

In the average corporate archive, most files are the first kind. And pulling text out of a PDF that already has text costs approximately nothing: it is a library, it runs on CPU, it handles thousands of pages a minute and it never misreads a character.

The four ways to read a document

The last three run as one-click templates on the platform, on a card starting at R$ 1.07 per hour. Start the machine, process the batch, shut it down.

What breaks on real documents

• Accents and non-English characters. Classic OCR drops diacritics constantly. Modern neural OCR handles them — which is the main reason not to use the tooling of the 2000s.
• Dense tabular forms. Invoices and tax documents pack tables with no grid lines and small type. You need table detection, not just text detection.
• Stamps and signatures over text. They degrade the whole line. There is no magic here: flag low confidence and route to human review.
• Phone photos. Skewed, shadowed, with a thumb in the corner. Deskewing before OCR measurably raises accuracy.
• Handwritten fields. This is the frontier: handwriting OCR is substantially worse than print. If your process depends on it, plan human review into the flow.

The expensive mistake: asking a big model for JSON directly

It is tempting: throw the image at the most capable model and ask for "invoice number, tax ID, total and line items as JSON". It works in the demo. In production it has three serious problems.

1. It is expensive. An image consumes a lot of input tokens. Multiply by 10,000 documents and the bill lands an order of magnitude above the alternative.
2. It hallucinates numbers. This is the severe one. When a digit is smudged, a generative model does not say "I could not read this" — it completes with what is plausible. An invoice total that becomes R$ 1,850.00 instead of R$ 1,650.00 goes unnoticed for months.
3. It cannot be audited. You have no page coordinates for where each field came from, so you cannot show the evidence when someone disputes it.

The pipeline in four stages

import pdfplumber, json
from openai import OpenAI

client = OpenAI(base_url="https://gpubrasil.com.br/v1",
                api_key="gpub_live_yourkeyhere")

def page_text(page):
    """1) Try the text that is already there. 2) Only then call OCR."""
    direct = (page.extract_text() or "").strip()
    if len(direct) > 100:
        return direct, "direct"
    image = page.to_image(resolution=300).original
    return ocr(image), "ocr"           # Surya running on your instance

SCHEMA = """Extract from the text below, as JSON and nothing else:
{"number": str, "supplier_id": str, "date": "YYYY-MM-DD",
 "total": number, "items": [{"description": str, "amount": number}]}
If a field is not clearly legible in the text, use null. NEVER invent a value."""

def structure(text):
    """3) The model turns text into fields — cheap, because the input is text."""
    r = client.chat.completions.create(
        model="gpub-fast",
        messages=[{"role": "system", "content": SCHEMA},
                  {"role": "user", "content": text[:8000]}],
        max_tokens=800,
        temperature=0,
        response_format={"type": "json_object"},
    )
    return json.loads(r.choices[0].message.content)

def validate(record):
    """4) The safety net the model does not give you."""
    problems = []
    if not record.get("total"):
        problems.append("missing total")
    if record.get("items"):
        s = sum(i["amount"] for i in record["items"] if i.get("amount"))
        if record.get("total") and abs(s - record["total"]) > 0.02:
            problems.append(f"items sum to {s}, total says {record['total']}")
    return problems

with pdfplumber.open("invoice.pdf") as pdf:
    for page in pdf.pages:
        text, source = page_text(page)
        record = structure(text)
        issues = validate(record)
        print("review" if issues else "approved", source, issues or "", record.get("number"))

The arithmetic of 10,000 pages

A 10,000-page scanned archive, from zero to structured database:

August 2026 prices, single GPU, no scheduled interruption. Prices are in Brazilian reais (BRL), the billing currency. The catalogue moves — check the live one before committing to a number.

Under nine reais for 10,000 pages, or R$ 0.00087 per page. Managed document APIs from the large clouds bill per thousand pages at a level that sits orders of magnitude above this — and that gap is why it is worth assembling the piece rather than subscribing to it.

Notice the division of labour too, which is the pattern of this whole series: the GPU works for three hours and is shut down; the brain is paid per token, with no machine to administer.

Three details that decide quality

• 300 DPI. Below that accuracy falls off quickly; far above it you only burn memory.
• Deskew first. A tilted page is the most common defect and the easiest to fix automatically.
• Keep the OCR text, not just the JSON. When someone disputes a field six months from now, you want to be able to show where it came from.

Next: the memory — why sending the whole PDF does not scale.

Keep reading: part 4: the memory · vision-language models · building a RAG system