Complete NCCL Tests keyboard shortcuts and commands reference — 13 shortcuts across 3 categories. Quick reference cheat sheet for Windows & Mac.
NCCL Tests measure the collective operations that multi-GPU training depends on, and a run of all_reduce_perf is the standard acceptance test for a new GPU node or cluster. The table lists the binaries and the flags that shape a run; the notes explain the numbers the tests print and how to run them across nodes.
| Shortcut | Action | Description |
|---|---|---|
| all_reduce_perf -b 8 -e 128M -f 2 -g 8 | All-reduce test | Benchmark all-reduce bandwidth on a single node with 8 GPUs, scanning message sizes from 8B to 128MB. |
| all_gather_perf -b 8 -e 128M -g 8 | All-gather test | Benchmark all-gather collective bandwidth across GPUs. |
| broadcast_perf -b 8 -e 128M -g 8 | Broadcast test | Benchmark broadcast collective bandwidth from a root GPU. |
| reduce_scatter_perf -b 8 -e 128M -g 8 | Reduce-scatter test | Benchmark reduce-scatter collective bandwidth. |
| alltoall_perf -b 8 -e 128M -g 8 | All-to-all test | Benchmark all-to-all collective bandwidth, the most demanding pattern on network fabric. |
| Shortcut | Action | Description |
|---|---|---|
| -b size | Min message size | Set the starting message size for the scan, e.g. 8 for 8 bytes. |
| -e size | Max message size | Set the ending message size for the scan, e.g. 128M for 128 MiB. |
| -f factor | Step factor | Multiply the message size by this factor between each step; 2 means doubling. |
| -g n | GPUs per process | Number of GPUs each process (rank) uses. |
| mpirun -np n -N ppn all_reduce_perf -g 1 | Multi-node run | Launch the test across multiple nodes via MPI, typically one GPU per rank. |
| Shortcut | Action | Description |
|---|---|---|
| -c 1 | Check correctness | Verify collective results are numerically correct on each iteration. |
| -w n | Warmup iterations | Number of warmup iterations to run before timing begins. |
| -J out.json | JSON output | Write machine-readable results to a JSON file for automated analysis. |
The most essential NCCL Tests shortcuts are: all_reduce_perf -b 8 -e 128M -f 2 -g 8 (All-reduce test), all_gather_perf -b 8 -e 128M -g 8 (All-gather test), broadcast_perf -b 8 -e 128M -g 8 (Broadcast test).
These are command-line commands — type them in your terminal or console. Combine them with shell history search (Ctrl + R) and aliases to work even faster.
The NCCL Tests shortcut for all-reduce test is all_reduce_perf -b 8 -e 128M -f 2 -g 8. Benchmark all-reduce bandwidth on a single node with 8 GPUs, scanning message sizes from 8B to 128MB.
NCCL Tests includes 5 Flags & Multi-Node shortcuts, including -b size (Min message size) and -e size (Max message size). See the full list in the Flags & Multi-Node section above.
Yes — use My Stack to combine NCCL Tests shortcuts with any other platform on this site into one printable reference, which is useful if your daily workflow spans several tools.
all_reduce_perf -b 8 -e 128M -f 2 -g 8 runs all-reduce on eight GPUs in one process, from 8 bytes to 128 MB doubling each step. The output has one row per size with time, algorithm bandwidth and bus bandwidth; busbw is the number to compare against hardware — close to NVLink bandwidth within a node, close to the network rate across nodes. all_gather_perf -b 8 -e 128M -g 8, reduce_scatter_perf -b 8 -e 128M -g 8, broadcast_perf -b 8 -e 128M -g 8 and alltoall_perf -b 8 -e 128M -g 8 exercise the other collectives; alltoall is the one that finds a weak network.
-b size and -e size set the message range and -f factor the multiplier between steps; -g n is GPUs per process. For a multi-node run, launch one process per GPU under MPI — mpirun -np n -N ppn all_reduce_perf -g 1 — because the -g mode is single-process only. -c 1 checks results for correctness (on by default; turn off for pure timing), -w n sets warm-up iterations so the first timed run is not paying initialisation, and -J out.json writes results as JSON for comparison across runs.
Bus bandwidth far below expectation usually means NCCL fell back to a slower path: set NCCL_DEBUG=INFO to see which transport (NVLink, PCIe, IB, socket) was selected per channel. Across nodes, check that NCCL_IB_HCA points at the right adapters and that GPUDirect RDMA is in use; within a node, nvidia-smi topo -m shows whether the GPUs share NVLink or cross a PCIe host bridge.
Open your assistant with this page preloaded as the source — great for follow-up questions like "which of these work in other apps?"