Memory, not peak FLOPS, decides how many GPUs you need for most LLM work. Here is how to size it, with the configurations that fit in our catalog.
Weights: parameters × bytes
Model weights take parameters × bytes per parameter: 2 bytes in BF16, 1 byte in FP8, 0.5 byte in FP4. Add about 20% for activations, CUDA context and fragmentation when serving.
| Model size | BF16 | FP8 | FP4 |
|---|---|---|---|
| 7B | 14 GB | 7 GB | 3.5 GB |
| 13B | 26 GB | 13 GB | 6.5 GB |
| 34B | 68 GB | 34 GB | 17 GB |
| 70B | 140 GB | 70 GB | 35 GB |
| 405B | 810 GB | 405 GB | 203 GB |
The KV cache grows with context
For each token in flight, the KV cache stores 2 × layers × KV heads × head dimension × bytes. For a 70B model with 80 layers, 8 KV heads and a head dimension of 128, that is about 0.33 MB per token in BF16, so roughly 42 GB for a single 128K-token sequence. Concurrency multiplies it: plan memory for your context length × concurrent requests.
Fine-tuning needs much more
- Full fine-tuning with Adam in mixed precision needs about 16 bytes per parameter (weights, gradients, optimizer states and master weights): about 1.1 TB for a 70B model, before activations.
- LoRA keeps the base model frozen: you need the BF16 weights (140 GB for 70B) plus small adapters and activations.
- QLoRA loads the base model in 4-bit: about 35 GB for 70B, which fits on a single large GPU.
Configurations that fit
| Workload | Memory needed | Fits on |
|---|---|---|
| Serve 7B–13B in BF16 | 17–31 GB | 1× L40S (48 GB) |
| Serve 34B in BF16 or 70B in FP8 | 82–84 GB | 1× RTX PRO 6000 (96 GB) or 1× H200 |
| Serve 70B in BF16 | ~170 GB | 1× MI300X (192 GB), 1× B300 or 2× H200 |
| Serve 405B in FP8 | ~490 GB | 2× B300, 4× H200 or 4× MI300X |
| Serve 405B in BF16 | ~970 GB | 4× B300, 4× MI325X or 8× H200 |
| QLoRA on 70B | ~50 GB | 1× H200 or 1× RTX PRO 6000 |
| Full fine-tune of 70B | 1.1 TB + activations | 8× B300 or 8× MI355X (2.3 TB); 8× H200 only with CPU offload |
These are starting points for weights plus modest context. Long contexts and high concurrency need extra room for the KV cache.
Rules of thumb
- Prefer fewer, larger GPUs: tensor parallelism across NVLink is fast, but each split costs communication.
- FP8 halves weight memory with little quality loss on modern GPUs; FP4 halves it again on Blackwell and CDNA 4.
- Check memory per dollar, not only price per GPU: see the price per GB line in our comparisons.