How to validate a new 8-GPU server before a long run

We burn in every allocation before it goes on sale, but you should still check any server before you commit a week of compute to it. The quick checks take a few minutes; full diagnostics take about half an hour.

1. Inventory and health

nvidia-smi                              # 8 GPUs, expected model and memory
nvidia-smi -q -d ECC | grep -i -A3 "Volatile"   # uncorrectable ECC must be 0
nvidia-smi -q -d PERFORMANCE            # no thermal or power throttling at idle

On AMD: rocm-smi, rocm-smi --showmeminfo vram and amd-smi metric --ecc.

2. Topology

nvidia-smi topo -m

On an HGX server every GPU pair should read NV followed by a link count. A SYS or PHB between two SXM GPUs means traffic would cross the CPU: open a ticket. On AMD, rocm-smi --showtopo should show XGMI between all GPUs.

3. PCIe links

nvidia-smi -q | grep -A2 -i "Link Width"

Current and max link widths should match (x16) for every GPU.

4. Diagnostics

dcgmi diag -r 3        # NVIDIA DCGM, 10 to 30 minutes

On AMD, the ROCm Validation Suite (rvs) runs equivalent memory, PCIe and stress tests.

5. Collective bandwidth

git clone https://github.com/NVIDIA/nccl-tests.git && cd nccl-tests && make -j
./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8

Look at the busbw column for the largest messages. It should approach the per-direction NVLink bandwidth of the platform, for example up to 450 GB/s on H100 SXM (NVLink 4) and about twice that on Blackwell (NVLink 5). Results far below that point to a topology or driver problem. On AMD, use rccl-tests with the same arguments.

6. A short burn-in

Run your real training step for 10 minutes and watch nvidia-smi dmon or rocm-smi for clocks, power and temperature. Stable clocks under load are the best sign of a healthy node.

If something fails

Email support with the hostname and the command output. If a server does not pass these checks at delivery, we replace it. See our SLA.