Short answer. Match the GPU to the largest model you need to hold, then to the speed you need. For 7B to 13B models and image generation, the NVIDIA L40S ($849 per GPU per month) is the cheapest fit. Up to 70B on a single card, the RTX PRO 6000 Blackwell ($1,199) holds the model at FP8. For 70B in BF16 or long contexts on one GPU, the AMD MI300X ($1,749) has 192 GB, and the H200 ($2,699) is the CUDA choice for 70B at FP8. For training and the largest models, the B200 ($4,449), the B300 ($4,899) and the AMD MI355X ($2,449) lead, with FP4 for inference.
The quick picks
| If you need to | Rent | Why | Per month |
|---|---|---|---|
| Serve 7B to 13B models, generate images | NVIDIA L40S | 48 GB holds a 13B model in BF16, at the lowest price | $849 per GPU |
| Serve up to 70B on one card, render, edit video | NVIDIA RTX PRO 6000 | 96 GB of GDDR7 holds 70B at FP8; FP4 Tensor Cores and RT cores | $1,199 per GPU |
| Serve 70B in BF16, or with long contexts, on one GPU | AMD MI300X | 192 GB of HBM3 at 5.3 TB/s; ROCm runs vLLM and SGLang | $1,749 per GPU |
| Serve 70B at FP8 on CUDA | NVIDIA H200 | 141 GB of HBM3e at 4.8 TB/s on Hopper software | $2,699 per GPU |
| Train on a budget with NVLink | NVIDIA H100 | The cheapest 8-GPU NVLink node, the reference for training recipes | $15,992 for 8 GPUs |
| Pre-train or fully fine-tune large models | NVIDIA B200 | 4.5 PF of dense FP8, 180 GB at 8 TB/s, NVLink 5 and strong FP64 | $4,449 per GPU |
| Serve reasoning models and models above 400B | NVIDIA B300 | 288 GB of HBM3e and 13.5 PF of dense FP4 | $4,899 per GPU |
| The same on ROCm, for half the price | AMD MI355X | 288 GB of HBM3E, 5.0 PF of FP8 and 10.1 PF of FP4 | $2,449 per GPU |
| Double-precision science (FP64) | B200 or MI355X | The B300 trades FP64 throughput for AI throughput | from $2,449 per GPU |
How to choose, in four questions
- How much memory does the model need? Weights take parameters times bytes per parameter (2 in BF16, 1 in FP8, 0.5 at 4-bit), plus about 20% for activations and runtime, plus the KV cache for long contexts and many users. This decides how many GPUs you need: see how much GPU memory you need.
- Is the job limited by memory bandwidth or by compute? Token generation is mostly bandwidth-bound: an H200 (4.8 TB/s) decodes faster than an H100 (3.35 TB/s) with the same compute. Training and prompt processing are compute-bound: compare dense FP8 or FP4 throughput.
- Does the model span several GPUs? Tensor parallelism needs NVLink or Infinity Fabric between the GPUs of a server. PCIe cards (L40S, RTX PRO 6000) are best with one model per GPU.
- CUDA or ROCm? PyTorch, JAX, vLLM and SGLang run on both. CUDA-only kernels and TensorRT-LLM need NVIDIA; AMD Instinct gives more memory per dollar.
Value per dollar
Price per GB of GPU memory and per PFLOPS of dense FP8, per GPU per month at CoreGPU:
| GPU | Memory | Bandwidth | FP8 dense | Per GPU / month | Per GB of memory | Per FP8 PFLOPS |
|---|---|---|---|---|---|---|
| NVIDIA L40S | 48 GB GDDR6 | 864 GB/s | 0.73 PF | $849 | $17.69 | $1,163 |
| NVIDIA RTX PRO 6000 | 96 GB GDDR7 | 1.6 TB/s | 1 PF | $1,199 | $12.49 | $1,199 |
| AMD MI300X | 192 GB HBM3 | 5.3 TB/s | 2.6 PF | $1,749 | $9.11 | $673 |
| NVIDIA H100 | 80 GB HBM3 | 3.35 TB/s | 1.98 PF | $1,999 | $24.99 | $1,010 |
| AMD MI350X | 288 GB HBM3E | 8 TB/s | 4.6 PF | $2,249 | $7.81 | $489 |
| AMD MI325X | 256 GB HBM3E | 6 TB/s | 2.6 PF | $2,299 | $8.98 | $884 |
| AMD MI355X | 288 GB HBM3E | 8 TB/s | 5 PF | $2,449 | $8.50 | $490 |
| NVIDIA H200 | 141 GB HBM3e | 4.8 TB/s | 1.98 PF | $2,699 | $19.14 | $1,363 |
| NVIDIA B200 | 180 GB HBM3e | 8 TB/s | 4.5 PF | $4,449 | $24.72 | $989 |
| NVIDIA B300 | 288 GB HBM3e | 8 TB/s | 4.5 PF | $4,899 | $17.01 | $1,089 |
USD, excluding taxes. Prices checked Nov 4, 2026; live prices and stock on the pricing page.
AMD Instinct leads on both counts: the MI350X and MI355X cost under $9 per GB of memory and under $500 per PFLOPS of FP8. Among NVIDIA GPUs, the B200 gives the most FP8 per dollar, just ahead of the H100, and the RTX PRO 6000 the most memory per dollar.
NVIDIA or AMD?
Choose NVIDIA when your stack depends on CUDA-only kernels, TensorRT-LLM or recipes tuned for Hopper and Blackwell, or when you need the fastest link between GPUs: NVLink 5 at 1.8 TB/s on the B200 and B300. Choose AMD Instinct when your models run on PyTorch, JAX, vLLM or SGLang: more memory and more FP8 throughput per dollar mean fewer GPUs per model. Side-by-side figures are in the GPU comparisons.
When renting by the month makes sense
A monthly price beats hourly billing once a GPU is busy more than about 540 hours a month, roughly 18 hours a day: training runs, fine-tuning and inference that stays online. For a few hours of tests, an hourly instance costs less. The calculation for each GPU is in hourly vs monthly GPU rental, and every monthly price is on the pricing page.
