Inside NVIDIA Cosmos 3: 64 billion parameters that generate video, audio and robot commands

On 31 May 2026 NVIDIA published Cosmos 3 with open weights and a commercial licence. It is the most ambitious model in the category: a single thing that takes text, image, video, audio and robot action trajectories, and returns any combination of those back.

This article is the model card read without the press release — what the architecture actually does, what NVIDIA itself admits does not work, and what an hour costs.

The architecture: two towers, not one

The detail that explains everything else: Cosmos 3 is a Mixture-of-Transformers (MoT). It is not one transformer doing everything — it is two, complementary, inside one model.

• An autoregressive tower, generating text the classical way: one token at a time, predicting the next.
• A diffusion tower, generating image, video, audio and action by iterative refinement — starting from noise and cleaning it up.

This solves a real problem. Text is discrete and sequential: token-by-token generation is natural. Video is continuous and simultaneous: 120 frames that must be coherent with each other are badly served by sequential generation. Models that force everything through one mechanism pay for it. Cosmos 3 uses the right mechanism per modality and pays the price of carrying both towers.

The five checkpoints

The practical recommendation is to start with the Nano. 16B against 64B is a fourfold difference in the bill, and to find out whether the category solves your problem the Nano answers the same question.

What goes in and what comes out

The card has two distinct modes, and confusing them is the most common mistake:

The "reasoner" takes text, image or video and returns text only — with a context window of up to 256,000 tokens. Not just description: it returns structured chain-of-thought, 2D and 3D point localisation and bounding boxes. It is the mode for understanding a scene.

The "generator" takes text, image, video and action trajectories, and returns image, video, audio or action. It is the mode for creating a scene.

Note the asymmetry: 5 input frames, up to 400 output frames. It is not built to edit an existing video — it is built to continue from a moment.

The part nobody covers: action

This is what makes Cosmos 3 a world model rather than a video generator. The action input is a per-frame sequence of the agent's state or control values — joint positions, gripper state, camera pose. Formally a (T, D) array: T frames by D dimensions.

And D is not free. Values exist only for bodies the model knows:

If your robot is not on that list, the action input does not serve you directly — and that is the most concrete limitation in the whole package. Part 5 deals with it.

Licence: you can use it commercially

The model shipped under OpenMDW 1.1, and the card states explicitly that it is ready for commercial and non-commercial use. It is not a research-only licence and has no revenue ceiling clause. For anyone evaluating adoption in a product, that is half the decision — and it is rare at this size.

What it costs to run — and the counterintuitive arithmetic

Here is the number nobody publishes: the Cosmos3-Super repository weighs 132.7 GB across 27 files. That is 64 billion parameters in BF16 — and the card warns that only BF16 has been tested: FP8, FP4 and FP16 are not officially supported. The easy escape hatch of "quantise it until it fits" is not available.

132.7 GB of weights fit on no single card. NVIDIA's reference configuration is 8 cards of 141 GB, 80 GB or 80 GB (H200, H100 or A100). Against today's catalogue:

Catalogue prices as of 1 September 2026, per single GPU, no scheduled interruption. Prices are in Brazilian reais (BRL), the billing currency. The catalogue moves within the same day — check the live one before committing to a number.

How you actually serve it

Four runtimes work: PyTorch directly, vLLM-Omni, Hugging Face Diffusers and SGLang Diffusion. The shortest path is vLLM-Omni, which brings up an OpenAI-compatible endpoint:

docker pull vllm/vllm-omni:cosmos3

vllm serve nvidia/Cosmos3-Super \
  --omni --host 0.0.0.0 --port 8000 \
  --cfg-parallel-size 2 --ulysses-degree 4 \
  --use-hsdp --hsdp-shard-size 8 \
  --init-timeout 1800

For 2 cards, swap in --cfg-parallel-size 2 --use-hsdp --hsdp-shard-size 2. On a card with less memory, --enable-layerwise-offload helps. What each of those flags actually does is the subject of part 4.

Three details that save you an afternoon:

1. The --init-timeout 1800 is not decorative. That is 132.7 GB to download and load; the vLLM default expires first. Provision disk generously — 200 GB is the comfortable minimum.
2. Linux only. NVIDIA states it has not tested anything else. Supported architectures: Ampere, Hopper and Blackwell.
3. The prompt has to be "upsampled" first. For decent quality, the short text has to become a structured JSON — and NVIDIA's own official example does that by calling a frontier LLM over an API. It is a separate step, with its own cost, that no demo shows.

What NVIDIA admits does not work

The limitations section of the card is honest enough to deserve direct translation. Declared defects: temporal inconsistency, unstable camera or object motion, imprecise physical interaction, inaccurate audio-video synchronisation and action-state drift — especially in long-horizon or high-resolution output.

In reasoning mode: object state, causal relationships, spatial geometry, temporal ordering and agent intent can be misinferred; long context can produce hallucinated entities.

And the declared root cause: there is no physics simulator in the model. 3D geometry, space-time evolution, object permanence, contact dynamics and physical laws are only approximated.

Is it worth it?

• "I want to generate marketing video." No. Far too expensive, and there are much better open models at that price-performance point.
• "I want to understand video scenes with spatial reasoning." Maybe — and start with the 16B Nano, which fits on one card.
• "I want to manufacture training data for a robot or vehicle, with action control." Then yes. It is the only open package that does this with a commercial licence today, and it is the subject of part 3.

Keep reading: part 1: what a world model is · part 4: splitting across GPUs