17 Launching and measuring distributed work
A distributed job needs a result rule, a set of participating workers, and a physical route. The preceding chapter established the result and its possible communication cost. Here the worker launch, framework, backend, and timing interval make that model usable. A rank is the integer identity of a participating process within a process group. The group states which processes must issue compatible collective calls in the same order.
17.1 Ranks, process groups, and communication backends
After the payload, topology, collective semantics, and possible schedules are defined, the software stack can be assigned concrete responsibilities. A distributed strategy decides what must communicate, a framework API expresses the operation for a group of ranks, a backend implements it, and the selected transport carries the bytes over the installed hardware.
torch.distributed: The PyTorch API for initializing process groups, assigning rank identities, and issuing point-to-point or collective operations.- Backend: The implementation selected by a process group to execute its communication operations, such as NCCL for many CUDA tensor collectives or Gloo for common CPU/control paths.
- Distributed strategy: DDP, FSDP, tensor parallelism, pipeline parallelism, and expert parallelism decide which objects and operations are required. The API and backend execute those requests.
Backends do not all occupy one interchangeable role. NCCL is commonly selected for CUDA tensor collectives on NVIDIA GPUs, following the NVIDIA Collective Communications Library Documentation1 and the torch.distributed documentation2. Gloo is commonly used for CPU tensors and control-oriented paths. Message Passing Interface (MPI) implementations provide broader HPC process and communication facilities. Compatibility depends on the framework build, tensor location, operation, platform, and deployment configuration.
NCCL (Section 4.1) implements the collective operations and chooses algorithms over the available paths, such as PCIe, NVLink, NVSwitch, and network interfaces.
NCCL does not decide the training strategy. It implements communication operations requested by frameworks or distributed libraries. For example, DDP may need all-reduce for gradients, FSDP may need all-gather and reduce-scatter for sharded state, and expert parallelism may need all-to-all routing. NCCL tries to execute those operations efficiently on the hardware topology.
- Process group: A set of ranks that participate in communication. PyTorch creates process groups and calls backend collectives through them.
- Collective algorithm: The pattern NCCL uses to move chunks, such as ring, tree, or hierarchical variants. The selected algorithm depends on message size, rank count, and topology.
- transport: The path used underneath the collective, such as peer-to-peer GPU links, PCIe, shared memory, sockets, InfiniBand, or RoCE depending on the system.
A training job with one process per GPU needs every process to know its identity before any collective can run. The process, rank, and GPU setup in Section 3.4 supplies the mapping used here. Under the one-process-per-GPU assumption, a rank identifies a participating process. RANK is its identity across the whole job, LOCAL_RANK identifies it within its node, and WORLD_SIZE counts participating processes. A collective (Section 16.1) is called by all participating processes in a group.
Example: Reading a two-node launch
Assume two nodes, each with two GPUs visible in the same local order to its workers, and one worker process per GPU. On node A, the processes have (RANK, LOCAL_RANK) pairs (0, 0) and (1, 1). On node B they have (2, 0) and (3, 1). Every process has WORLD_SIZE = 4. The program binds each worker to the visible device indexed by its LOCAL_RANK. Rank 2 therefore uses node B’s local device 0, not a device numbered 2 on that node.
After binding their local GPUs, all four processes call torch.distributed.init_process_group(backend="nccl"). By default, this call uses the rank variables and rendezvous address that torchrun supplies. The address is the meeting point where the processes join the default group. That group can then run a collective when each rank supplies its tensor to the same operation in compatible order. A local rank is reused on different nodes, while a global rank is unique within the job. Comparing these logged pairs with the intended layout checks placement before interpreting communication timing.
torchrun creates worker processes and supplies rank-related variables such as RANK, WORLD_SIZE, and LOCAL_RANK.3 For a one-node two-worker program, the launch form is torchrun --standalone --nproc-per-node=2 script.py. The script must still initialize a process group, bind LOCAL_RANK to its visible GPU, and call matching collectives on both ranks; the command alone is not an experiment. For a multi-node job, run torchrun on each node with the same node count and rendezvous settings. The program selects or accepts the backend when it creates a process group, such as NCCL for CUDA tensor collectives or Gloo for CPU paths, while ranks must call compatible collectives in compatible order.
17.2 Communication and computation overlap
Communication/computation overlap means scheduling data movement so it occurs while independent useful computation is still running. It can reduce step time by shortening the communication interval that remains exposed, even when a tail still lies on the critical path. Resource contention can offset that gain, so the end-to-end time must be measured.
Overlap works only when there is independent computation available while communication runs. If the next layer or token needs the communicated result immediately, the communication remains exposed.
Adjacent compute and collective blocks do not prove overlap. When compute and communication lanes are aligned, they expose the end-to-end critical path. If shortening the collective would shorten the step, it lies on a completion-limiting path. Otherwise, another required path may still set completion time. The dependencies show whether independent computation covers the communication.
17.3 Profiling communication
Communication profiling asks where time is lost when ranks exchange tensors. The evidence can come from several layers: the framework call such as dist.all_reduce, the NCCL backend that chooses a collective algorithm and transport, the GPU timeline that shows copy or collective kernels, the network fabric that carries bytes between nodes, and the rank synchronization behavior that makes fast ranks wait for slow ranks.
For a communication comparison, give each rank the same operation, dtype, and payload rule. Check the collective result before interpreting its time. Separate warmup from measured calls and time completed work on each rank. Report the slowest rank-local elapsed time, or state another aggregation, because a fast rank may wait for a slow one. This interval includes work and waiting under its stated boundary; it does not by itself measure link bandwidth or align clocks across GPUs.
PyTorch Profiler can show collective operators in the framework trace. Nsight Systems shows whether collectives overlap kernels or sit on the critical path. NCCL logs help diagnose backend selection, network interface choice, peer-to-peer paths, and failures.
NCCL logs: NCCL_DEBUG=INFO and related settings expose backend decisions. Logs are diagnostic evidence, not proof of achieved performance by themselves.
The following controls change what a fresh run logs or which routes it may use; the collective result rule stays the same. Their behavior follows the torch.distributed documentation4.
NCCL_DEBUG=INFOprints NCCL initialization and communication diagnostics.NCCL_SOCKET_IFNAMEselects the network interface for socket-based NCCL paths.NCCL_IB_DISABLE=1disables NCCL’s InfiniBand/RoCE transport, allowing fallback to another available transport such as IP sockets.5NCCL_P2P_DISABLEcan disable peer-to-peer GPU communication for diagnosis.
For reproducible comparisons, set diagnostic environment variables before launching workers and use a fresh run for each configuration. Do not rely on changing an environment variable to reconfigure an existing communicator. These settings can test routing hypotheses, but they do not choose the model’s sharding strategy or prove that the enabled path reaches peak throughput.6
The diagnostic question is whether the framework requested the intended collective, NCCL selected the intended transport, and the trace shows a critical-path wait. Those are different causes: changing a framework strategy will not repair a wrongly selected network interface, while changing a transport setting will not make an overly frequent exchange disappear.
17.4 Topology-aware design
Topology-aware distributed design means choosing rank placement and a parallelism or serving layout for the traffic each creates. The physical route affects startup delay and usable transfer rate; payload, recurrence, dependent waiting, and possible overlap determine how much of that cost reaches the critical path.
Communication frequency and distance are a first placement check, not a complete cost rule. Frequent dependent exchanges make a slow route more costly, while payload and independent work can change the outcome. A transfer inside each layer can also recur for every generated token, so count both dimensions. These cases describe when data moves; later chapters identify which strategy creates the exchange.
- Per generated token: A split serving step may need a partial result or KV state before selecting the next token. Delay on that dependent path is visible in time per output token.
- Per layer use: Workers may combine partial outputs or gather a needed shard whenever a split layer runs. Repetition across layers makes startup delay matter even for small payloads.
- Per optimizer update: Workers training on different examples may combine gradients before one parameter update. Local micro-batches can contribute to that update; the synchronization rule determines the actual exchange count.
- Per request or planned phase handoff: New-request routing or moving prefill KV to a decode worker has a cost tied to that request, not automatically to every later token. Include it in first-token and whole-request time.
- Background: Checkpoints, model loading, and cache-tier movement can use slower paths when they finish before dependent work needs them. Otherwise count the exposed delay on the critical path.
Later chapters apply these frequencies to replicated and sharded training, split layers and stages, expert routing, and serving. The recurrence and actual route, rather than the strategy’s name, determine the cost.
- Scale-up domain: A tightly connected GPU group such as one server, pod, or NVLink/NVSwitch island. It is the preferred place for frequent tensor-parallel or decode-critical communication.
- Scale-out domain: Multiple nodes connected through NICs and switches. It provides capacity and aggregate throughput, but it usually adds latency, routing, and contention.
- GPU/NIC affinity: The locality relationship between a GPU and the network card used for cross-node traffic. Poor affinity can add PCIe hops, CPU-socket crossings, or host staging.
- Bisection bandwidth: The capacity across the equal-sized endpoint groups defined in Section 16.5. Concurrent transfers share this capacity, so a fast single link does not establish the aggregate rate.
- Oversubscription: A topology condition where the upstream fabric has less bandwidth than the sum of attached endpoints. It is common in cost-controlled networks and can hurt concurrent collectives.
- Placement policy: The scheduler or launch decision that maps ranks, serving groups, workers, or replicas onto physical GPUs and nodes.
17.5 Communication design checks
Reliable distributed diagnosis names the framework call, backend library, physical path, and critical-path wait. The following checklist establishes the evidence before a communication strategy changes.
Checklist: Communication design
- NVLink, a physical GPU interconnect, is distinguished from NCCL, a collective communication library.
torch.distributedis identified as the framework API that calls a backend. The backend and selected transport remain separate evidence items.- Rank placement reflects topology so a local collective does not cross an unnecessary slow path.
- Collective timing is synchronized and warmup remains separate from steady-state measurement.
A communication design connects the moving tensor, exchange frequency, rank placement, and remaining critical-path delay. Replicated and sharded training (Section 18.1) applies those relationships to model copies, gradient synchronization, and state partitions. The model-parallel, pipeline and expert, and distributed-serving chapters then add split model execution and distributed serving, where the required exchanges occur at different points in a step or request.
References
NVIDIA. (2026). NVIDIA Collective Communications Library Documentation (NCCL 2.32.3). NCCL documentation. The documentation at this link can change versions. Supports NCCL as the common GPU collective library implementing AllReduce, Broadcast, Reduce, AllGather, ReduceScatter, AlltoAll, Gather, Scatter, and point-to-point operations over paths including PCIe, NVLink, NVSwitch, and network interfaces. Limit: documents library behavior and configuration, not achieved bandwidth; algorithm choice depends on message size, rank count, topology, and settings such as NCCL_ALGO.↩︎
PyTorch. (2026). Distributed Communication Package: torch.distributed. Versioned v2.14 documentation at PyTorch distributed communication documentation, created 2017-07-12 and updated 2026-08-07. Supports process groups with rank identity, backend selection with NCCL for CUDA collectives and Gloo for CPU paths, and NCCL debugging with NCCL_DEBUG equals INFO. Limit: backend availability depends on build and platform; the page documents API behavior, not achieved performance.↩︎
PyTorch contributors. (2.14 documentation). torchrun. PyTorch elastic launch guide. Supports the launch tool’s process creation and rank-related environment variables for supported PyTorch distributed jobs. Limit: the guide documents the interface, not achieved communication performance or availability for every build.↩︎
PyTorch. (2026). Distributed Communication Package: torch.distributed. Versioned v2.14 documentation at PyTorch distributed communication documentation, created 2017-07-12 and updated 2026-08-07. Supports process groups with rank identity, backend selection with NCCL for CUDA collectives and Gloo for CPU paths, and NCCL debugging with NCCL_DEBUG equals INFO. Limit: backend availability depends on build and platform; the page documents API behavior, not achieved performance.↩︎
NVIDIA. (NCCL 2.32.3 documentation). Environment Variables. Retrieved September 28, 2026, from NCCL environment variables. Describes diagnostic settings and InfiniBand/RoCE transport disabling. The documentation at this link can change versions. Starting a fresh run for each configuration is a conservative test procedure used here, not a documented guarantee about when every setting is read or whether an existing communicator is reconfigured.↩︎
NVIDIA. (NCCL 2.32.3 documentation). Environment Variables. Retrieved September 28, 2026, from NCCL environment variables. Describes diagnostic settings and InfiniBand/RoCE transport disabling. The documentation at this link can change versions. Starting a fresh run for each configuration is a conservative test procedure used here, not a documented guarantee about when every setting is read or whether an existing communicator is reconfigured.↩︎
