132 GB will not fit on one card: how a large model is split across GPUs

Every infrastructure decision for AI starts with the same arithmetic, and it fits on one line:

VRAM ≈ parameters × bytes per parameter + headroom

A 64-billion-parameter model in BF16 (2 bytes each) needs 128 GB just to exist in memory — before processing anything. No single card on the market has that.

From there, you either shrink the model or split it. This article is about the second option — and about the price it charges.

The arithmetic, with numbers you recognise

Add execution headroom on top: activations, attention cache, runtime workspace. For a language model serving a few users, 15–20% usually suffices. For a video model, far more — because it holds dozens of frames alive at once, with attention between them.

That is where part 2's claim comes from: 128 GB of weights will not run on a 141 GB card, because that leaves 13 GB for the part that consumes the most.

The five possible cuts

"Splitting a model across GPUs" is not one thing — it is five different cuts, and they compose. Each cuts in a different direction.

1. Tensor parallelism — cuts each layer in half

Every weight matrix is sliced across the cards. They all process the same input, each with its slice, and exchange results at the end of every layer.

Gain: solves memory and speed at once. Cost: the hungriest for communication — an exchange at every layer. Outside a machine with a fast interconnect between cards, it degrades badly. Typical flag: --tensor-parallel-size.

2. Sequence parallelism (Ulysses) — cuts what is being processed

Instead of slicing the model, it slices the input: each card takes a chunk of the sequence — of tokens for text, of frames for video.

Gain: the cut that makes long context and long video fit. Cost: requires redistribution at every attention block. Flag: --ulysses-degree.

3. CFG parallelism — the trick unique to diffusion

This one deserves attention because it is the most elegant and the least known. Diffusion models with classifier-free guidance compute, at every step, two predictions: one conditioned on your prompt and one unconditional. Then they combine them.

Since the two are independent, each can run on a different card — in parallel, with no communication until the end of the step.

4. Sharding (FSDP / HSDP) — each card holds a piece, assembled on demand

Parameters live scattered; when a layer is about to execute, the cards gather only the piece needed, use it and discard it.

The H in HSDP is for hybrid: shard inside the group of cards that talk fast, replicate across groups. Flags: --use-hsdp --hsdp-shard-size N.

5. Pipeline parallelism — each card takes a stretch of layers

Layers 1–10 on card A, 11–20 on B, and so on. Data walks the line.

Gain: minimal communication. Cost: the "bubble" — while card B works, A waits. It only pays off with a full queue of requests, which makes it better suited to training than to serving one person.

And if you only have one card? Offload

There is a plan B: keep only the executing layer in VRAM and park the rest in system memory, fetching each layer when its turn comes. That is --enable-layerwise-offload.

It works, and it is honest to say it is slow — the card spends much of its time waiting on the bus. It is for proving the thing runs, or for batch work with no deadline. It is not for serving requests.

The measurement almost nobody makes

Here is the practical point of this entire article. Everyone assumes more cards is proportionally better. NVIDIA's own published figures for Cosmos 3 say otherwise:

Catalogue prices as of 1 September 2026, per single GPU, no scheduled interruption. Prices are in Brazilian reais (BRL), the billing currency. The catalogue moves within the same day — check the live one before committing to a number.

Four times the cards, 3.3 times the speed. The difference — about 22% — is what inter-card communication consumes, and you pay it in money.

This is not an implementation defect: it is Amdahl's law showing up. There is always a fraction of the work that does not parallelise, and it sets the ceiling. The more cards you add, the more expensive each unit of purchased speed becomes.

A practical sizing recipe

1. Compute the weights: parameters × bytes per parameter.
2. Add headroom: +20% for a language model; +50% or more for video, and more again if the output is long.
3. Check whether quantisation is available — do not assume it is.
4. Divide by the card's VRAM and round up. That is your minimum.
5. Measure before scaling. Run at the minimum, time it, and only then decide whether the extra speed is worth the extra cost. Real speed-up is almost never what you expected.

Catalogue prices as of 1 September 2026, per single GPU, no scheduled interruption. Prices are in Brazilian reais (BRL), the billing currency. The catalogue moves within the same day — check the live one before committing to a number.

Row four against row three is the summary of everything: the same 70B model costs four times less per hour quantised on one card than in BF16 on two. When quantisation is available it is usually the most profitable infrastructure decision there is — and it is precisely the door Cosmos 3 closes.

Keep reading: part 2: inside Cosmos 3 · part 1: what a world model is