If you train or run AI models on GPU rented by the hour, a question comes up early: where, exactly, do your models, datasets and checkpoints live? And, more importantly, what survives when you shut the machine down? This guide organizes the topic and shows how to avoid re-downloading everything each session.
โก Summary
The disk that ships with the machine is temporary โ it disappears when you delete the instance. To avoid losing (or re-downloading) models and data every session, keep them on a persistent disk, which exists off the machine and is reused on any new instance.
The two types of storage
When you rent a GPU by the hour, you deal with two different storages โ and confusing them is the source of almost every nasty surprise:
| Machine disk | Persistent disk | |
|---|---|---|
| What it is | the instance's local storage | an independent volume, yours |
| Survives deleting the machine? | no โ it is wiped | yes โ it stays stored |
| Good for | system, temporary session files | models, datasets, checkpoints, results |
Where each thing should live
Models (weights)
A model's weights can be tens or hundreds of gigabytes. Downloading them is what costs the most time (and therefore paid GPU). Keep them on the persistent disk and point your framework's cache there. That way the download happens once; on the next machines, the model is already available.
Datasets
Training and evaluation datasets also belong on the persistent disk, especially the large ones. For efficient reads during training, it is worth packing many small files into formats your library can stream โ it avoids the bottleneck of opening millions of individual files.
Checkpoints
Checkpoints are the heart of a long training run: they are what let you pause and continue a fine-tune later. Save them to the persistent disk at regular intervals. That way, if you terminate the machine (on purpose or not), you resume from the last checkpoint instead of starting over.
Results and logs
Generated outputs, metrics and logs you want to keep also go to the persistent disk โ otherwise they vanish with the machine.
The classic mistake: keeping everything on the temporary disk
The default on many machines is to write to a local folder that does not survive termination. It is easy to work for hours, shut down, and only then realize the checkpoint and the results are gone. The rule of thumb: what you do not want to lose lives on the persistent disk. The machine disk is just scratch.
A workflow that saves money
- Create a persistent disk once, with room for your models and data.
- Attach it to the machine and point model cache, datasets and checkpoints into it.
- Work. Save checkpoints frequently.
- When you finish, shut the machine down โ you stop paying for the GPU, and everything stays stored.
- Next session, attach the same disk and continue. No re-downloading.
That cycle trades expensive GPU hours (keeping a machine on or rebuilding setup) for cheap storage. It is the difference between paying for the card only when it is actually computing, and paying for it while it sits idle.
How to do it on GPUBrazil
On GPUBrazil this volume is the Persistent Disk, available in your account under My Disks. Create it at whatever size you need (from 50 GB to 10 TB), attach it when creating the instance โ GPU or CPU โ and reuse it on any new machine. Billing is in reais, straight from your balance, for the contracted size only.
Stop re-downloading everything each session
On GPUBrazil, the Persistent Disk keeps your models, datasets and checkpoints off the machine. Power the GPU off when you are not using it and pick up in seconds next time โ in reais, no contract.
Create my Persistent DiskFrequently asked questions
Where do my models live when I rent a GPU by the hour?
By default, on the machine's temporary disk, which is wiped when you delete it. To keep them between sessions, store them on a persistent disk.
How do I avoid losing training progress?
Save checkpoints frequently on a persistent disk. If the machine terminates, you resume from the last checkpoint instead of starting over.
What disk size do I need?
Add up the size of the model weights you use, the datasets, and a few checkpoints, with some headroom. On GPUBrazil you can choose from 50 GB to 10 TB.
Read next: what a persistent disk is and how it cuts your bill ยท how much it costs to run AI in Brazil.