Appendix B — Optional Labs

Optional GPU and runtime labs

These optional probes practice GPU and runtime measurement after the core GPU chapters. Use a supported NVIDIA GPU, driver, and CUDA-enabled PyTorch build; the Triton and Nsight Systems probes have their own extra requirements. Record the GPU, driver, PyTorch and CUDA versions, shape, dtype, and precision settings with each result. A missing device or unsupported backend stops that probe; it does not imply that a different path was measured.

The first code block checks device facts and BF16 storage. The environment logger records the local software and hardware state. The timing harness later in this entry is a fragment: it needs a bound model, inputs, positive warmup_steps, and positive num_steps. The following small setup supplies those names for a known matrix operation; it is a local practice load, not a model-serving benchmark.

Example: Device capability and dtype footprint

This probe shows that torch.cuda owns device visibility and basic CUDA runtime facts, while a tensor’s dtype determines its byte footprint. get_device_capability is a hardware probe, not a guarantee that every desired kernel path is enabled. element_size connects dtype to memory traffic: one million BF16 values occupy about two megabytes before allocator overhead.

import torch

device = torch.device("cuda", 0)

print(torch.cuda.get_device_name(device))
print(torch.cuda.get_device_capability(device))

x = torch.empty(1_000_000, dtype=torch.bfloat16, device=device)
bytes_per_element = x.element_size()
footprint_bytes = x.numel() * bytes_per_element

print(bytes_per_element, "bytes per element")
print(footprint_bytes, "total bytes")

Record the environment. The logger reports installed versions and selected settings. An unset variable or missing optional package is a recorded fact, not a reason to infer which GPU kernel ran.

Code example: Environment and version logger

import importlib.metadata
import os
import platform
import subprocess
import sys
import torch

def package_version(name):
    try:
        return importlib.metadata.version(name)
    except importlib.metadata.PackageNotFoundError:
        return "not installed"

print(f"Python: {sys.version.split()[0]}")
print(f"OS: {platform.platform()}")
print(f"PyTorch: {torch.__version__}")
print(f"CUDA runtime reported by PyTorch: {torch.version.cuda}")
print(f"cuDNN: {torch.backends.cudnn.version()}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    props = torch.cuda.get_device_properties(0)
    print(f"GPU: {props.name}; memory={props.total_memory / 2**30:.1f} GiB")

for package in ("triton", "transformers", "flash-attn", "xformers", "vllm", "deepspeed"):
    print(f"{package}: {package_version(package)}")

for name in ("CUDA_VISIBLE_DEVICES", "RANK", "LOCAL_RANK", "WORLD_SIZE", "NCCL_SOCKET_IFNAME"):
    print(f"{name}={os.environ.get(name, '<unset>')}")

try:
    driver = subprocess.run(
        ["nvidia-smi", "--query-gpu=driver_version", "--format=csv,noheader"],
        check=True, capture_output=True, text=True,
    ).stdout.splitlines()[0]
except (FileNotFoundError, subprocess.CalledProcessError, IndexError):
    driver = "unavailable"
print(f"NVIDIA driver: {driver}")

Bind a known GPU operation. This local setup checks CUDA availability, creates a fixed input and a finite reference result, and defines the names expected by the next timing harness. The harness warms the operation and times completed CUDA stream work. Its average excludes setup, data loading, and service queueing. For a changed operation, check its output against the same reference under a declared tolerance before comparing time.

Code example: Known GPU operation for the timing harness

import torch

if not torch.cuda.is_available():
    raise RuntimeError("This probe needs a CUDA-enabled PyTorch build and GPU")
device = torch.device("cuda", 0)
torch.manual_seed(7)
inputs = torch.randn(256, 256, device=device)

class KnownGpuOp(torch.nn.Module):
    def forward(self, x):
        return torch.relu(x @ x)

model = KnownGpuOp().to(device).eval()
with torch.inference_mode():
    reference = model(inputs)
assert reference.shape == inputs.shape
assert torch.isfinite(reference).all().item()
warmup_steps = 5
num_steps = 20
train_or_serve_step = lambda: model(inputs)

Code example: CUDA event timing loop

import torch

# 1. Warmup
for _ in range(warmup_steps):
    _ = model(inputs)
torch.cuda.synchronize()

# 2. Synchronized timing loop
start_event = torch.cuda.Event(enable_timing=True)
end_event = torch.cuda.Event(enable_timing=True)

start_event.record()
for _ in range(num_steps):
    _ = model(inputs)
end_event.record()

torch.cuda.synchronize()
elapsed_ms = start_event.elapsed_time(end_event)
print(f"Average per-step stream time: {elapsed_ms / num_steps:.2f} ms")

Performance probes and checks

Each template below is a controlled probe for one bottleneck question. The mechanism it exercises is taught in the section linked beside it. The notes here add the input shapes the template assumes, its correctness check, and what its output shows.

Timeline tracing

Operator totals identify expensive framework calls, but they do not show whether delay occurs in host scheduling, CUDA submission, memory copies, device kernels, or communication. A system timeline preserves that ordering. The two workflows below capture it through Nsight Systems or export it for Perfetto.

For a self-contained practice trace, save the known-operation setup and the generic timing harness above, in that order, as train_or_serve.py; run that script once normally and confirm its finite check and average-time printout. The resulting script executes only this fixed matrix operation. A real training or serving trace needs that workload’s own entry point and arguments.

Example: Recording CUDA dispatch in a system trace

This abbreviated alternative captures a system timeline. Prepare an installed nsys command-line tool that supports the selected trace options, the actual training or serving script and its arguments, a fixed workload, and a writable output directory. The displayed command uses a POSIX shell on a supported Linux capture target; its backslash line continuations are not PowerShell syntax.1

  1. Use the checked train_or_serve.py practice script or replace it with the workload’s prepared entry point and arguments. Run from the chosen output directory, with no existing run_trace.nsys-rep to overwrite.
  2. Run the command with --trace=cuda,nvtx,osrt,cublas,cudnn, --sample=none, and -o run_trace. CPU sampling is disabled; the selected trace categories still record their supported events.
  3. Confirm that the run finishes and produces run_trace.nsys-rep. Record capture errors rather than treating a missing report as a measurement.
  4. Open that report in a compatible Nsight Systems application and inspect the recorded scheduling, CUDA API, library, kernel, copy, and instrumented NVTX lanes to locate the delay.
nsys profile --trace=cuda,nvtx,osrt,cublas,cudnn \
  --sample=none \
  -o run_trace python train_or_serve.py

Example: Exporting active events for a browser trace

This abbreviated alternative exports a PyTorch trace; it does not require the Nsight capture. Prepare a CUDA-capable PyTorch installation and bind train_or_serve_step() to a no-argument callable using the intended model, inputs, device, and state. The shown fragment assumes all GPU work runs on the current CUDA device. Its no-argument torch.cuda.synchronize() completes warmup on that device. If the callable uses multiple devices, complete warmup on each relevant device before active recording begins.2

  1. Run the five scheduled calls: one wait step, one warmup step, and three active steps. Keep the warmup-completion boundary before the first active step, as shown.
  2. Confirm that trace.json exists and contains active CPU and CUDA events for the prepared workload. If expected events are missing, check the callable, devices, and capture support before interpreting the trace.
  3. Open the Perfetto browser trace viewer, which opens a browser service, and load trace.json. Inspect the recorded CPU and CUDA lanes without changing the profiler’s recorded interval.
import torch
from torch.profiler import ProfilerActivity, profile, schedule

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=schedule(wait=1, warmup=1, active=3, repeat=1),
    on_trace_ready=lambda p: p.export_chrome_trace("trace.json"),
) as prof:
    for step in range(5):
        train_or_serve_step()
        if step == 1:
            # Finish scheduled warmup before entering active recording.
            torch.cuda.synchronize()
        prof.step()
# Open trace.json in https://ui.perfetto.dev.

An empty interval shows no recorded GPU activity in the lanes covered by the trace. The selected process, streams, trace settings, and event capture need checking before that interval can be treated as device idle time. Possible causes include CPU dispatch, compilation, input starvation, a synchronization dependency, communication, memory allocation, or intentional waiting. The surrounding CPU, CUDA API, copy, and collective lanes provide the distinguishing evidence.

Roofline calculation template

Arithmetic intensity identifies the lower ceiling in the simple roofline model. FLOPs divided by measured time places the observed kernel point below or near that ceiling; the gap suggests the first bottleneck hypothesis, not its cause. Achieved bandwidth, occupancy, launch timing, and memory-transaction counters are still needed before choosing an optimization.

Example: FLOPs, bytes and measured time

This Python example turns a measured kernel into a roofline point. It includes sample numbers so the output can be interpreted immediately. Supply nonnegative FLOP and byte counts, positive elapsed time and hardware rates, and a positive HBM byte count for a finite arithmetic intensity. The arithmetic helper does not validate these input domains.

def roofline(flops, bytes_moved, seconds, peak_tflops, peak_tb_s):
    ai = flops / bytes_moved
    achieved = flops / seconds / 1e12
    memory_ceiling = ai * peak_tb_s
    bound = min(peak_tflops, memory_ceiling)
    if memory_ceiling < peak_tflops:
        limiting_roof = "memory"
    elif memory_ceiling > peak_tflops:
        limiting_roof = "compute"
    else:
        limiting_roof = "both"
    return ai, achieved, bound, limiting_roof

ai, achieved, bound, limiting_roof = roofline(
    flops=2e12,
    bytes_moved=1e12,
    seconds=0.500,
    peak_tflops=200,
    peak_tb_s=3,
)
print(f"AI={ai:.1f} FLOPs/byte, achieved={achieved:.1f} TFLOP/s")
print(f"roofline_bound={bound:.1f} TFLOP/s, limiting_roof={limiting_roof}")

With the sample inputs, arithmetic intensity is 2 FLOPs per byte, the HBM ceiling is 6 TFLOP/s, and achieved throughput is 4 TFLOP/s. The point is below the memory roof and cannot yet be called bandwidth-saturated. A trace and kernel counters are needed to determine whether launch overhead, occupancy, access efficiency, or another dependency explains the remaining gap.

Triton GEMV kernel

The next template reruns the Triton kernel from matrix-vector multiplication (Section 5.5) across tile sizes.

It applies when profiler evidence shows a layout or fusion opportunity beyond the selected vendor or compiler path.

Example: Tile size in a matrix-vector kernel

This probe compares three tile sizes for the Triton kernel, with one program instance per output row, with a torch.mm library path. Both read the same resident FP16 matrix and vector and write separate FP32 outputs. The library views the vector as an N by 1 matrix and uses a preallocated result. Its full FP16 accumulation and intermediate FP16 reductions are disabled; its internal algorithm and reduction order can still differ from Triton’s FP32 multiplication and reduction.3

Both paths must pass the same numerical rule against an FP64 reference before timing starts. A failed Triton configuration is skipped; a failed library comparison stops the benchmark. The listing needs a CUDA-enabled PyTorch build supporting torch.mm with FP16 inputs, out_dtype=torch.float32, and an FP32 output buffer, plus a supported NVIDIA GPU and compatible Triton installation. Record those versions and precision settings with the results.

import statistics

import torch
import triton
import triton.language as tl


@triton.jit
def gemv_kernel(A, x, y, M: tl.constexpr, N: tl.constexpr, BLOCK: tl.constexpr):
    row = tl.program_id(0)
    offsets = tl.arange(0, BLOCK)
    acc = tl.full((), 0.0, tl.float32)
    for start in range(0, N, BLOCK):
        cols = start + offsets
        mask = cols < N
        acc += tl.sum(
            tl.load(A + row * N + cols, mask=mask, other=0.0).to(tl.float32)
            * tl.load(x + cols, mask=mask, other=0.0).to(tl.float32),
            axis=0,
        )
    tl.store(y + row, acc)


def median_event_ms(fn, warmup=10, repeats=30, launches=20):
    """Median per-launch CUDA-stream time over repeated batches on the current stream."""
    for _ in range(warmup):
        fn()
    torch.cuda.synchronize()

    samples = []
    for _ in range(repeats):
        start = torch.cuda.Event(enable_timing=True)
        end = torch.cuda.Event(enable_timing=True)
        start.record()
        for _ in range(launches):
            fn()
        end.record()
        end.synchronize()
        samples.append(start.elapsed_time(end) / launches)
    return statistics.median(samples)


def validate_and_time(candidates, library, check):
    """Check every output before the first timing sample."""
    ready = []
    for label, fn, output in candidates:
        fn()  # Also compiles the Triton variant.
        torch.cuda.synchronize()
        try:
            check(output)
        except AssertionError as error:
            print(f"{label}: quality failed; timing skipped: {error}")
            continue
        ready.append((label, fn))

    library_label, library_fn, library_output = library
    library_fn()
    torch.cuda.synchronize()
    check(library_output)  # No timings are printed if the library fails.

    return [
        (label, median_event_ms(fn))
        for label, fn in [*ready, (library_label, library_fn)]
    ]


def main():
    if not torch.cuda.is_available():
        raise RuntimeError("This benchmark requires CUDA-enabled PyTorch")

    # Disable full FP16 accumulation and intermediate FP16 reductions.
    torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction = False
    torch.backends.cuda.matmul.allow_fp16_accumulation = False
    torch.manual_seed(0)
    M, N = 4096, 4096
    A = torch.randn((M, N), device="cuda", dtype=torch.float16)
    x = torch.randn((N,), device="cuda", dtype=torch.float16)
    y = torch.empty((M,), device="cuda", dtype=torch.float32)
    reference64 = A.double() @ x.double()

    library_y = torch.empty((M,), device="cuda", dtype=torch.float32)
    library_x2 = x[:, None]  # Views made before timing; no input copy.
    library_y2 = library_y[:, None]

    # One quality rule for both FP32 outputs, set before any timing.
    rtol, atol = 1e-4, 1e-3
    candidates = []
    for block in (256, 512, 1024):
        def candidate(block=block):
            gemv_kernel[(M,)](A, x, y, M, N, BLOCK=block)

        candidates.append((f"Triton BLOCK={block}", candidate, y))

    def library():
        try:
            torch.mm(A, library_x2, out_dtype=torch.float32, out=library_y2)
        except (TypeError, RuntimeError) as error:
            raise RuntimeError(
                "This PyTorch CUDA build must support torch.mm with "
                "FP16 inputs, out_dtype=torch.float32, and a preallocated FP32 output"
            ) from error

    def check(output):
        torch.testing.assert_close(
            output, reference64, rtol=rtol, atol=atol, check_dtype=False
        )

    results = validate_and_time(
        candidates, ("torch.mm FP16 inputs to FP32 output", library, library_y), check
    )

    print(f"FP32 output quality: rtol={rtol}, atol={atol} vs FP64 reference")
    print("Triton: FP16 resident inputs, FP32 multiply/accumulate/output")
    print("torch.mm: same FP16 inputs, FP32 output; FP16 reduction modes off")
    print("Both timings use the same input/output storage contract on this shape")
    for label, ms in results:
        print(f"{label}: median {ms:.4f} ms per call")


if __name__ == "__main__":
    main()

The first call for each tile size compiles that variant and checks its output. Each timed path then runs ten warmup calls outside the samples. A sample records CUDA events around twenty calls on the current stream, waits for the end event, and divides the interval by twenty. The printed result is the median of thirty samples, in milliseconds per call, using the event timing explained in Section 7.3.

This interval includes the GPU work and any gaps between the queued calls. Tensor setup, the FP64 reference, correctness checks, and JIT compilation remain outside it; conversions performed inside a kernel remain part of its work. It is device-stream time. A complete training or serving decision also needs a measurement of the whole operation. Reusing the arrays can change cache behavior, so this probe does not measure cold HBM traffic.

Compare configurations only after they meet the declared quality rule. The library result now supplies a measured reference for the same 4096 by 4096 input values and FP16-input/FP32-output storage. Report the precision settings and both timings; sharing a storage contract does not imply identical arithmetic order. Kernel counters can then investigate memory transactions, bandwidth, occupancy, and register pressure. Keeping a custom kernel in production still requires the actual workload’s shapes, layouts, quality requirement, and complete-operation timing.

Definition - Code-to-object mapping

A: The matrix with shape M by N. In a decode-like projection, this stands for a large weight matrix read from GPU memory.

x: The vector with length N. This stands for one token representation or a small batch slice.

row: The Triton program id. One program computes one output coordinate y[row].

offsets and mask: The tile-local column indices and boundary guard. They define which elements of A[row, :] and x are read in each loop chunk.

acc: The running dot-product accumulator for y[row], kept in registers or fast local storage before the final store.

Optional GEMV code check: In the template above, explain what tl.program_id(0), the masked tl.load, the reduction, and tl.store each control. Identify one layout or tile choice that could make the kernel slower.

Attention backend comparison

The attention comparison that follows measures the memory saving explained in FlashAttention and attention-kernel libraries (Section 5.7).

Example: Comparing attention backends

The default is B = 1, H = 4, S = 512 and D = 64 in FP16. Each explicit score or probability tensor is 2 MiB, and the output is 0.25 MiB. The entry check requires free GPU memory for the estimated intermediates and a safety allowance before allocating inputs; it does not guarantee that every backend workspace will fit. Increase shapes only after recalculating these sizes and checking headroom.

The naive path materializes B x H x S x S score and probability tensors. The optimized call requests the FlashAttention backend. The check below compares outputs, event time, and additional allocated memory for one forward-only shape. This comparison applies only on a CUDA configuration that supports the requested backend. If the forced-backend call raises RuntimeError, the check records the error and skips the comparison; the error may mean the backend is unavailable or that this call failed for another reason. It does not silently fall back to another attention backend.

import gc
import torch
import torch.nn.functional as F
from torch.nn.attention import SDPBackend, sdpa_kernel

# Small default: Batch, Heads, Sequence Length, Head Dimension.
B, H, S, D = 1, 4, 512, 64
if not torch.cuda.is_available():
    raise RuntimeError("This probe requires a CUDA-enabled PyTorch build and GPU")
score_bytes = B * H * S * S * 2  # FP16 score tensor.
# Conservative entry check, not a prediction of backend workspace needs.
required_headroom = 3 * score_bytes + 8 * B * H * S * D * 2 + 256 * 2**20
free_bytes, _ = torch.cuda.mem_get_info()
if free_bytes < required_headroom:
    raise RuntimeError("Insufficient free GPU memory; reduce B, H, or S and recalculate")
q = torch.randn(B, H, S, D, device="cuda", dtype=torch.float16)
k = torch.randn(B, H, S, D, device="cuda", dtype=torch.float16)
v = torch.randn(B, H, S, D, device="cuda", dtype=torch.float16)

# 1. Naive Attention (explicit O(S squared) intermediate matrix)
def naive_attention(q, k, v):
    scale = 1.0 / (q.shape[-1] ** 0.5)
    attn_weights = torch.matmul(q, k.transpose(-2, -1)) * scale
    attn_probs = F.softmax(attn_weights, dim=-1)
    return torch.matmul(attn_probs, v)

# 2. Optimized scaled-dot-product attention backend
def optimized_attention(q, k, v):
    with sdpa_kernel(SDPBackend.FLASH_ATTENTION):
        return F.scaled_dot_product_attention(q, k, v)

@torch.inference_mode()
def measure(fn, repeats=10):
    gc.collect()
    torch.cuda.empty_cache()
    for _ in range(3):
        out = fn(q, k, v)
    torch.cuda.synchronize()
    baseline = torch.cuda.memory_allocated()
    torch.cuda.reset_peak_memory_stats()
    start = torch.cuda.Event(enable_timing=True)
    stop = torch.cuda.Event(enable_timing=True)
    start.record()
    for _ in range(repeats):
        out = fn(q, k, v)
    stop.record()
    torch.cuda.synchronize()
    extra_peak = torch.cuda.max_memory_allocated() - baseline
    return out, start.elapsed_time(stop) / repeats, extra_peak

with torch.inference_mode():
    naive_check = naive_attention(q, k, v)
    try:
        flash_check = optimized_attention(q, k, v)
    except RuntimeError as error:
        print(f"flash: unavailable or failed; comparison skipped: {error}")
    else:
        torch.testing.assert_close(flash_check, naive_check, rtol=2e-2, atol=2e-2)
        del naive_check, flash_check
        naive_out, naive_ms, naive_peak = measure(naive_attention)
        flash_out, flash_ms, flash_peak = measure(optimized_attention)
        print(f"naive: {naive_ms:.3f} ms, extra peak {naive_peak / 2**20:.1f} MiB")
        print(f"flash: {flash_ms:.3f} ms, extra peak {flash_peak / 2**20:.1f} MiB")

For comparison, scaling to B = 4, H = 32, S = 2048 and D = 128 in FP16 would give each explicit score or probability tensor 4 × 32 × 2048² elements at 2 bytes, or 1 GiB. The output has 4 × 32 × 2048 × 128 elements, or 64 MiB. In the naive function, both 1-GiB matrices remain live while the output is produced. This explains why avoiding them can reduce memory, but the printed extra peak is not their sum: the code subtracts bytes allocated after warmup from the maximum allocated during repeated calls. That peak can include a new output and temporary storage. The Flash measurement starts while naive_out remains live, so its baseline includes that output. Allocator reuse and backend workspaces make the actual peak difference a measured result, not a guaranteed 2-GiB saving.

The event time is one mean across ten warm calls, not request latency or a latency distribution. A report across representative sequence lengths should record timings, extra peaks, unsupported shapes, and fallbacks. One successful configuration does not establish behavior for every attention workload.

Definition - Code-to-object mapping

q, k, v: Query, key, and value tensors with shape B by H by S by D.

attn_weights: The explicit B by H by S by S score tensor in the naive implementation. This is the object FlashAttention avoids materializing in HBM.

attn_probs: The normalized score tensor after softmax. In the naive path it is another large intermediate. In the tiled path its effect is computed online.

F.scaled_dot_product_attention: The PyTorch API call that may dispatch to a FlashAttention-style backend when dtype, device, mask, and shape constraints are compatible.

Inspecting a torch.compile boundary

The next probe tests the behavior described in graph breaks (Section 8.3) on the installed PyTorch version.4

Example: Tensor selection and scalar branching

This example contrasts a tensor-valued selection with a Python branch driven by a CUDA scalar. Prepare a CUDA-capable PyTorch installation that supports this compile path, its required compiler tools, and a writable working directory. The listing is a diagnostic probe, not a steady-state timing loop.

  1. Save the shown probe as compile_boundary_probe.py in that directory. The example filename below refers to this saved listing, not a supplied application.
  2. Enable TORCH_LOGS='graph_breaks,recompiles' before starting its Python process, using one of the shell-specific commands below.5
  3. Run the probe and retain the diagnostics. Successful execution creates output tensors y and z; they can exist even when capture falls back. Record errors and the observed behavior, not an assumed graph-break count.
  4. Check both outputs against their eager references. For timing, warm up each callable separately and compare repeated completed-work samples on the same x using the timing methods. Do not report the first diagnostic calls as steady-state time.
import torch

# Normal compiled path: condition remains a GPU tensor
def compiled_step(x):
    return torch.where(x.sum() > 0, x * 2, x - 2)

compiled = torch.compile(compiled_step, mode="reduce-overhead")
x = torch.randn(1024, device="cuda")
y = compiled(x)

# Host-control variation: .item() returns a Python scalar.
def host_branch(x):
    return x * 2 if x.sum().item() > 0 else x - 2

compiled_host_branch = torch.compile(host_branch)
z = compiled_host_branch(x)

The scalar-controlled branch can force capture fallback. Similar failures can arise from unsupported logging, Python branches that depend on tensor data, or scalar diagnostics inside the compiled function. Whether an operation breaks capture depends on the installed compiler and settings.

Use one launch path from the directory containing the saved probe.

PowerShell: Set the variable for this shell’s child processes, then start Python:

$env:TORCH_LOGS = 'graph_breaks,recompiles'
python .\compile_boundary_probe.py

POSIX shell: Set the variable for this Python process:

TORCH_LOGS='graph_breaks,recompiles' python compile_boundary_probe.py

The environment setting must be in place before Python starts. An installed-version diagnostic interface is an alternative if configured before the compiled calls. Record the installed version, input shapes, observed captured regions, fallbacks, recompilations, and separately measured warmed timings; diagnostics vary with compiler settings and do not promise a fixed break count.

CUDA Graph warmup and replay

The CUDA Graph template applies the capture and replay rules of the CUDA Graph lifecycle (Section 8.5) to a module.

Example: Reusing captured input and output buffers

This inference-mode pattern warms the module on a side stream, synchronizes before capture, records stable-address work, and explains the lifetime of the captured output buffer. It applies after profiling shows launch overhead and the shapes and addresses can remain stable.

import torch

if not torch.cuda.is_available():
    raise RuntimeError("This example requires a CUDA-enabled PyTorch build and GPU")
device = torch.device("cuda", 0)
module = torch.nn.Linear(1024, 1024).to(device).eval()
example = torch.randn(32, 1024, device=device)
graph = torch.cuda.CUDAGraph()
static_in = example.clone()  # Valid values before warmup and capture.
warmup_stream = torch.cuda.Stream()

with torch.inference_mode():
    warmup_stream.wait_stream(torch.cuda.current_stream())
    with torch.cuda.stream(warmup_stream):
        for _ in range(3):
            _ = module(static_in)
    torch.cuda.current_stream().wait_stream(warmup_stream)
    torch.cuda.synchronize()

    with torch.cuda.graph(graph):
        static_out = module(static_in)

@torch.inference_mode()
def replay(x, *, keep_snapshot=False):
    if x.shape != static_in.shape or x.dtype != static_in.dtype or x.device != static_in.device:
        raise ValueError("x must match static_in shape, dtype, and device exactly")
    static_in.copy_(x)
    graph.replay()
    return static_out.clone() if keep_snapshot else static_out

with torch.inference_mode():
    expected = module(example)
    first_snapshot = replay(example, keep_snapshot=True)
    torch.cuda.synchronize()
    torch.testing.assert_close(first_snapshot, expected, rtol=1e-4, atol=1e-5)
    changed_input = example + 1
    changed_expected = module(changed_input)
    second_snapshot = replay(changed_input, keep_snapshot=True)
    torch.cuda.synchronize()
    torch.testing.assert_close(second_snapshot, changed_expected, rtol=1e-4, atol=1e-5)
    torch.testing.assert_close(first_snapshot, expected, rtol=1e-4, atol=1e-5)
    print("Two replay inputs match eager results; the first snapshot is preserved")

Capture safety is workload-specific: lazy initialization, unsupported operations, dynamic allocations, changing shapes, or cross-stream dependencies can make a sequence unsafe. Each replacement input must have the captured input’s shape, dtype, and device. The template checks those properties before copy_ because copy_ itself can broadcast or convert other inputs6. The listing checks two replay inputs against eager outputs before any timing. With keep_snapshot=False, the return value is the shared static output buffer: the next replay overwrites its contents. A snapshot is an independent clone. Calls must be ordered through completion on the same stream; independent concurrent callers need separate graphs and buffers or explicit serialization.

For a timing comparison, run the completed-work harness in this collection separately for eager execution and replay, after the correctness checks above. For eager execution, bind model = module and inputs = example; for replay, bind model = replay and inputs = example. Keep the harness’s positive warmup_steps and num_steps for each run. Ordinary replay reuses the static output. If the consuming application needs a retained snapshot, bind model = lambda x: replay(x, keep_snapshot=True) so the clone cost is included. These checks do not themselves establish a speedup, and compilation or capture time remains separate from steady-state replay time.

Optional training and state labs

This optional training lab starts with a complete synthetic classifier update, then asks where its CPU submissions, transfers, and GPU operations occur. It extends the small update in the core training chapter. Run the listing only with a CUDA-enabled PyTorch installation, one supported NVIDIA GPU, and a writable output directory for its trace. It creates its own features, labels, model, and optimizer; no other lab’s fixture is required. Record the GPU, PyTorch and CUDA versions, batch shape, dtype, and profiler schedule.

PyTorch profiling template

The first PyTorch profiling pass establishes the hot operations before a change is made. PyTorch’s native profiler uses a context manager to trace host CPU thread execution, CUDA API dispatch, and GPU kernel execution timelines.

Example: Locating expensive framework operations

This complete single-GPU example trains a small classifier on fixed synthetic features and class labels. Each update transfers one batch, computes logits and loss, runs backward, and updates the parameters. It requires a CUDA-enabled PyTorch installation and one supported NVIDIA GPU. The features have shape (64, 128) and the integer labels have shape (64,); the model returns 16 class scores per example. This is a training-control example, not a measured LLM workload.

torch.profiler.record_function adds a label around the CPU code for each phase. CPU activity must be enabled for those labels to appear.7 CUDA activity connects the submitted operations with their device execution. Keep synchronization out of the individual phases: forcing each phase to finish would change the overlap being investigated.

The profile records three active updates after setup and scheduled warmup. Synchronization completes warmup before active recording. The active updates request non-blocking transfers from pinned host buffers and contain no explicit device synchronization; the trace shows the resulting ordering. The printed table and saved trace identify the labeled phases and their operations. The final finite-loss and parameter-change checks run outside the recorded phase. Passing them confirms an update occurred in this synthetic run, not that a training-method change preserves model quality.

import torch
from torch.profiler import ProfilerActivity, profile, record_function, schedule

if not torch.cuda.is_available():
    raise RuntimeError("This profiler example requires CUDA-enabled PyTorch")
torch.manual_seed(0)
device = torch.device("cuda", 0)
features = torch.randn(64, 128).pin_memory()
labels = torch.randint(16, (64,)).pin_memory()
model = torch.nn.Linear(128, 16).to(device).train()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
loss_fn = torch.nn.CrossEntropyLoss()
initial_parameters = [p.detach().clone() for p in model.parameters()]

def train_step():
    optimizer.zero_grad(set_to_none=True)
    with record_function("input_transfer"):
        batch = features.to(device, non_blocking=True)
        target = labels.to(device, non_blocking=True)
    with record_function("forward_and_loss"):
        loss = loss_fn(model(batch), target)
    with record_function("backward"):
        loss.backward()
    with record_function("optimizer_update"):
        optimizer.step()
    return loss.detach()

for _ in range(5):
    train_step()
torch.cuda.synchronize(device)

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=schedule(wait=1, warmup=1, active=3, repeat=1),
    record_shapes=True,
    profile_memory=True,
    on_trace_ready=torch.profiler.tensorboard_trace_handler("./traces"),
) as prof:
    for step in range(5):
        final_loss = train_step()
        if step == 1:
            torch.cuda.synchronize(device)
        prof.step()
torch.cuda.synchronize(device)
assert torch.isfinite(final_loss).item(), "Non-finite final loss"
changed = any(not torch.equal(before, after.detach())
              for before, after in zip(initial_parameters, model.parameters()))
assert changed, "No parameter change observed"
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=20))
print("final_loss", final_loss.item(), "parameter_changed", changed)

The table ranks framework operations within the selected profiling window. Rows named after the phase labels can also carry CUDA time. In the checked PyTorch 2.11 run, those annotation rows report the GPU span of the labeled work, including idle gaps, and can exceed the Self CUDA time total. Compare kernel and copy rows to find where device work accumulates; use the timeline to distinguish long kernels, repeated short launches, copies, synchronization, and idle gaps. Memory coalescing cannot be diagnosed from a CPU-to-GPU gap. It requires kernel-level memory-transaction evidence.

The profile table and trace diagnose where this synthetic update spends work; profiler instrumentation can change its measured duration. For a step-time comparison, run the same warmed update separately without profiling, synchronize at the timing boundary, and report the distribution and correctness checks. CPU phase labels mark submissions, not completed GPU phase durations. For ordering across CPU and GPU lanes, use the system timeline method in profiler selection (Section 7.2.1).

Optional decode and serving labs

This optional collection applies decode state, profiling, KV storage, and speculative verification to separate probes. The 32-request launch-gap case is a proposed experiment, not measured speedup. The KV calculator needs only its stated numeric inputs. The model probes need the local setup below and do not import a fixture from another lab.

Model setup for the prefill and cached-decode probes. Choose an accessible causal-model checkpoint and exact revision, then record the matching tokenizer revision, installed Transformers and PyTorch versions, one CUDA device, and a dtype supported by that model and device. Load the model in evaluation mode. Use the tokenizer to make a nonempty equal-length, unpadded batch of prompt IDs, and put its torch.long input_ids on the model’s device. Save the exact prompt IDs and batch and prompt lengths. Before timing, run one inference-mode forward with use_cache=True; check finite logits of shape [batch, prompt_length, vocabulary_size] and a returned cache. Feed one selected token of shape [batch, 1] with that cache, and check that the returned cache records one more consumed position. Inspect the chosen model’s forward signature and cache representation: if it requires an attention mask, position IDs, or cache_position, pass and update those inputs on both calls and adapt the fragments below. A checkpoint name alone does not establish this compatibility. The collection supplies no approved model and tokenizer pair, so these model probes are parameterized until the reader completes and checks this setup.

The one-forward profiler records prefill and discards its cache. The cached-decode probe uses the cache for later calls. Keep both separate from request latency, which also contains admission, queueing, and transport.

Decode launch-gap experiment design

The following illustrative protocol tests whether repeated host launch work slows a fixed decode workload. It specifies the evidence needed for a conclusion. It does not report a measured improvement.

Example: Testing a decode launch gap

Assumed workload: The proposed workload contains 32 requests, 512-token prompts, 64 generated tokens, one model revision, one GPU, and fixed sampling settings. The request arrival pattern, batch membership, and token counts remain the same in both runs. Early stopping is disabled for this fixed-token experiment, and token counts are checked after each run.

Input and state: Tokenization produces the fixed prompt IDs. Prefill builds the initial KV cache. Decode then appends one token per active request and reads the existing cache.

Evidence to collect: The launch-overhead hypothesis is worth testing if a baseline timeline shows short decode kernels separated by CPU launch gaps. Kernel reports can test whether individual kernels are close to a memory or compute ceiling. Neither observation alone proves that graph replay improves request latency.

Proposed intervention: After warmup, CUDA Graphs capture an eligible stable decode region, as developed in Chapter 8. Request admission and variable-length control remain outside capture. The comparison keeps model arithmetic and kernel selection fixed so a timing change can be connected to launch behavior.

Comparison record: Repeated baseline and replay runs without profiler instrumentation supply TPOT and 95th-percentile (p95) request latency, using the same timing boundaries. The record includes values, run-to-run spread, and the number of requests and repetitions. Separate diagnostic traces compare GPU idle gaps and kernel durations because profiling can change latency. Compile and capture time remain outside steady-state decode. Correctness checks compare logits within a declared dtype-appropriate tolerance and generated tokens under fixed deterministic selection before interpreting performance.

Conditional conclusion: Reduced launch gaps together with unchanged kernel work, passing correctness checks, and a repeatable TPOT reduction support a launch-overhead explanation. A p95 improvement supports a benefit to slower requests only if it exceeds the observed run-to-run variation. Smaller gaps without lower TPOT do not establish an end-to-end gain. Unchanged or worse p95 means tail latency has not improved. Persistent gaps call for further checks of CPU scheduling, batching, or cache management. This protocol supplies no before/after measurements, so no speedup is claimed.

Example: Profiler trace export

This abbreviated probe records one forward call, normally prompt prefill. It uses the checked model and input_ids from this collection’s setup. The shown call accepts input_ids and use_cache=True without an incoming cache. If the chosen model needs an attention mask or positions, adapt the call before using its trace.

Warmup and synchronization occur before the recorded call. The exported single_forward_trace.json shows that call’s operations, copies, and gaps. use_cache=True requests cache creation, but this listing discards the returned cache and does not record repeated decode. Cached decode state shows the required continuation: retain prefill’s past_key_values, pass one selected token of shape (batch, 1) to each later call, and keep the returned updated cache. To investigate repeated decode, profiling must enclose those later calls. Prefill remains outside that region when the question concerns decode alone. The table is diagnostic evidence, not a clean end-to-end latency measurement.

import torch
from torch.profiler import profile, ProfilerActivity

# Requires the model and unpadded prompt IDs described above.
model.eval()
with torch.inference_mode():
    for _ in range(3):
        warmup_out = model(input_ids=input_ids, use_cache=True)
        del warmup_out
    torch.cuda.synchronize(input_ids.device)
    with profile(
        activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
        record_shapes=True,
        profile_memory=True,
    ) as prof:
        out = model(input_ids=input_ids, use_cache=True)
    del out

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=20))
prof.export_chrome_trace("single_forward_trace.json")

Result to check: The table and saved trace should contain one active forward call after warmup. Use their operations and gaps to choose a later measurement; this probe alone reports no decode rate or request latency.

Cached decode state

Example: Checking generated tokens and cache positions

This callable PyTorch probe takes the checked model and nonempty prompt IDs shaped [B, P], an integer generation limit, and optionally one EOS token ID. It assumes unpadded equal-length prompts and a compatible Transformers model in evaluation mode that accepts past_key_values and infers positions from the cache, as in decode loop anatomy (Section 12.1). Models requiring explicit masks or cache-position arguments need an adapted wrapper; the setup check must pass before using this fragment.

EOS stopping is supported only for batch size one. Supplying an EOS ID with B > 1 raises ValueError before any model call. With no EOS ID, the probe generates a fixed number of tokens for every row and does not stop at EOS. A nonpositive limit returns shape [B, 0]. The output contains only new token IDs, including EOS when it triggers stopping. The first call processes the prompt and creates the cache. Later calls consume the previous selected token plus that cache, then predict the next token.

import torch
@torch.inference_mode()
def cached_decode(model, input_ids, max_new_tokens, eos_token_id=None):
    if input_ids.ndim != 2 or input_ids.shape[0] == 0 or input_ids.shape[1] == 0:
        raise ValueError("input_ids must have nonempty shape [B, P]")
    if eos_token_id is not None and input_ids.shape[0] != 1:
        raise ValueError("EOS stopping requires batch size one")
    if max_new_tokens <= 0:
        return input_ids[:, :0]
    out = model(input_ids, use_cache=True)
    past = out.past_key_values
    token = out.logits[:, -1].argmax(dim=-1, keepdim=True)
    generated = [token]
    for _ in range(max_new_tokens - 1):
        if eos_token_id is not None and token.item() == eos_token_id:
            break
        out = model(token, past_key_values=past, use_cache=True)
        past = out.past_key_values
        token = out.logits[:, -1].argmax(dim=-1, keepdim=True)
        generated.append(token)
    return torch.cat(generated, dim=1)

For example, let one prompt contain three tokens and the greedy choices be token 7 followed by EOS, with a limit of four. Prefill caches the three prompt positions and selects 7. One further call consumes 7, extends the cache to four positions, and selects EOS. The function returns [7, EOS] and makes no further model call. The final selected token has not been fed back into the model, and the function returns no cache. This is a state-flow probe, not a complete serving scheduler or sampler.

Definition - State trace

Prefill call: model(input_ids, use_cache=True) processes the prompt and returns the initial cache for all layers.

past: The persistent decode state. Its token dimension grows as generation proceeds, so cache reads still scale with context length.

token: A one-token tensor fed to later model calls. The compute for new Q/K/V is small, but attention reads the stored K/V history.

Updated cache: Each post-prefill model call adds the previous selected token’s keys and values. Selection alone does not add that newly predicted token to the cache. The EOS check can stop before another cache update.

Preserving key-value states shifts the decode workload from recomputing the whole prefix toward reading and extending stored attention state. In this case, optimization must focus on cache capacity, layout, transfer bandwidth, and launch overhead rather than treating decode as constant-cost arithmetic.

Key-value cache lower-bound calculator

The calculator implements the formula in key-value cache sizing (Section 10.6). It estimates packed scalar storage only, before block metadata, scale tensors, padding, fragmentation, allocator reserves, and temporary workspaces.

Example: Storage implied by a KV shape

This Python calculator estimates raw packed KV storage from batch size, stored sequence length, layer count, key-value heads, head dimension, and bits per element. Supply nonnegative integer batch size and sequence length, positive integer layer and head counts and head dimension, and a positive bit width. The helper does not validate those domains. It reports binary GiB and deliberately keeps storage-format overhead outside the formula.

def estimate_kv_cache_gib(batch_size, seq_len, layers, kv_heads, head_dim, bits_per_element=16):
    # 2 for separate Key and Value states
    elements_per_token = layers * kv_heads * head_dim * 2
    total_elements = batch_size * seq_len * elements_per_token
    total_bytes = total_elements * bits_per_element / 8
    return total_bytes / (1024 ** 3)

gib = estimate_kv_cache_gib(
    batch_size=32,
    seq_len=4096,
    layers=32,
    kv_heads=8,      # Grouped-Query Attention (GQA)
    head_dim=128,
    bits_per_element=16,  # FP16/BF16 scalar payload
)
print(f"Raw packed KV payload: {gib:.2f} GiB")

The shown values give 16.00 GiB of raw packed KV payload; metadata, padding, and workspaces need additional space.

Definition - Code-to-formula mapping

batch_size: B, the number of active sequences admitted by the scheduler.

seq_len: S, the number of stored prompt plus generated tokens per sequence.

layers: L, the number of transformer layers that keep KV state.

kv_heads and head_dim: The key-value head count and per-head dimension in the KV-cache formula.

bits_per_element: The nominal packed payload width of each cached scalar. It is not a complete quantized-storage model.

gib: The raw KV-cache payload converted to GiB. It excludes block metadata, scales, padding, fragmentation, and temporary workspaces.

The next reference implementation follows the acceptance and correction rules of speculative decoding (Section 14.1) for one sequence, without KV-cache reuse.

Distribution-correct speculative step

Example: Verifying a draft and sampling a correction

This pedagogical implementation drafts k tokens, verifies them in one target-model pass, applies the standard acceptance rule, and commits either a residual-sampled correction or one additional target token. It deliberately recomputes model state and uses host-visible decisions, so it is a correctness reference rather than a production speed implementation. Its draft and target checkpoints are a separate setup from the single-model probes: replace both placeholder IDs with accessible pinned revisions; verify a shared tokenizer and token-to-ID mapping, matching vocabulary dimensions, supported dtype/device, a nonempty tokenized prompt, and a positive integer k. No compatible pair is supplied or tested here. It uses plain categorical sampling with FP32 probability calculations. Finite precision can still differ from the exact-arithmetic distribution proof.

import time
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

device = torch.device("cuda", 0)
target_name = "target-checkpoint"
draft_name = "draft-checkpoint"

tokenizer = AutoTokenizer.from_pretrained(target_name)
draft_tokenizer = AutoTokenizer.from_pretrained(draft_name)
target = AutoModelForCausalLM.from_pretrained(
    target_name, torch_dtype=torch.float16
).to(device).eval()
draft = AutoModelForCausalLM.from_pretrained(
    draft_name, torch_dtype=torch.float16
).to(device).eval()
if (
    tokenizer.get_vocab() != draft_tokenizer.get_vocab()
    or tokenizer.special_tokens_map != draft_tokenizer.special_tokens_map
):
    raise ValueError("Draft and target require the same token-to-ID mapping")

@torch.inference_mode()
def speculative_step(prompt, k=4):
    if isinstance(k, bool) or not isinstance(k, int) or k < 1:
        raise ValueError("k must be a positive integer")
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
    prompt_len = input_ids.shape[1]
    if prompt_len == 0:
        raise ValueError("prompt must contain at least one token")
    draft_ids = input_ids
    draft_probs = []

    torch.cuda.synchronize()
    start = time.perf_counter()
    for _ in range(k):
        q_logits = draft(draft_ids, use_cache=False).logits[:, -1]
        q_probs = torch.softmax(q_logits.float(), dim=-1)
        next_id = torch.multinomial(q_probs, num_samples=1)
        draft_probs.append(q_probs)
        draft_ids = torch.cat([draft_ids, next_id], dim=1)

    target_logits = target(draft_ids, use_cache=False).logits
    accepted = 0
    committed = None
    for i, pos in enumerate(range(prompt_len, draft_ids.shape[1])):
        token = draft_ids[:, pos]
        p = torch.softmax(target_logits[:, pos - 1].float(), dim=-1)
        q = draft_probs[i]
        ratio = p.gather(1, token[:, None]) / q.gather(1, token[:, None])
        accept = (torch.rand_like(ratio) < torch.clamp(ratio, max=1.0)).item()
        if accept:
            accepted += 1
        else:
            residual = torch.clamp(p - q, min=0)
            residual_mass = residual.sum(dim=-1, keepdim=True)
            if not torch.isfinite(residual_mass).all() or (residual_mass <= 0).any():
                raise RuntimeError("Nonpositive or nonfinite residual mass after rejection")
            residual = residual / residual_mass
            replacement = torch.multinomial(residual, num_samples=1)
            committed = torch.cat([draft_ids[:, :pos], replacement], dim=1)
            break

    if committed is None:
        extra_p = torch.softmax(target_logits[:, -1].float(), dim=-1)
        extra_token = torch.multinomial(extra_p, num_samples=1)
        committed = torch.cat([draft_ids, extra_token], dim=1)

    torch.cuda.synchronize()
    wall_ms = (time.perf_counter() - start) * 1000
    return committed, accepted, wall_ms

The function returns committed token IDs, accepted-prefix length, and synchronized step time. The IDs and retained length permit correctness checks; speed needs a matched target-only run with the same stopping and token-count rules.

Definition - Code-to-object mapping

draft_ids: The growing candidate sequence proposed by the draft model.

draft_probs: The saved draft distributions q(x) at the exact positions where candidate tokens were proposed.

target_logits and p: The target-model verification output and target distribution p(x).

ratio: The p(x) / q(x) term used by the acceptance rule for the proposed token.

accepted: The retained draft-prefix length, an integer from 0 through k.

committed: The prompt plus the accepted prefix and either a residual correction or an extra target-sampled token.

wall_ms: Synchronized wall time for this unoptimized reference step. Compare it with a target-only reference on the same prompts before claiming speedup.

Example: Counting committed output tokens

Consider three illustrative calls with k = 4 and returned accepted-prefix lengths 0, 2, and 4. Their mean is (0 + 2 + 4) / 3 = 2 draft tokens per call. Each call also commits a residual correction or an extra target token, so the corresponding new-token counts are 1, 3, and 5. These counts can be checked directly by subtracting the original prompt length from each returned sequence length.

Suppose the synchronized step times are 12, 15, and 18 ms. The combined result is nine new tokens in 45 ms, or 9 / 0.045 = 200 tokens/s across these measured step intervals. This is illustrative arithmetic, not an observed benchmark. The ratio of total tokens to total time weights the intervals correctly. An unweighted mean of per-call rates would answer a different question.

The timing excludes loading and tokenization but includes this reference’s draft work, target verification, sampling, and tensor updates. It is not endpoint goodput. The reference has no EOS or output-budget truncation. A serving implementation must count only tokens actually committed before stopping. Separate reports by k and prompt class make changes in workload visible. A matched target-only run must generate the same number of new tokens under the same output rules and timing boundary before a speedup ratio is meaningful.

The Section 14.1.1 cost model uses a theoretical alpha with independent acceptance assumptions. No estimator of that parameter is inferred from these stopped-prefix counts. This reference omits KV-cache reuse, rollback, batching, and device-side acceptance, so its elapsed time does not establish production speed.

The final template batches that verification across sequences, as production decoders do (speculative decoding, Section 14.1).

Vectorized speculative verification

Example: Batched verification with per-row corrections

This pseudocode fixes the data contract for batched verification. It assumes B nonempty, unpadded prompts of common length P, k >= 1 proposals, identical draft/target token IDs, and plain categorical sampling from normalized probabilities over vocabulary size V. It recomputes full prefixes without persistent caches to make position alignment explicit. No EOS stopping or output-budget truncation is applied here.

draft_ids contains the prompt plus proposals, shape [B, P+k]. proposal_ids is only the [B, k] suffix. draft_probs, shape [B, k, V], stores the full distribution used at each proposal position. The target output has shape [B, P+k, V]. Its logits at P-1+i predict proposal i, and its final logits predict an extra token after all k proposals. The gathered [B, k] proposal probabilities are sufficient for acceptance decisions, but residual sampling needs both full vocabulary distributions at the first rejection.

For a row retaining n < k proposals, the commit helper uses target_probs[row, n, :] and draft_probs[row, n, :], normalizes max(p - q, 0), and samples one correction. For n = k, it samples from extra_target_probs[row, :]. It returns that row’s original prompt, the n retained proposals, and this one final token as a variable-length sequence. A zero residual mass on an actual rejection is a numerical error to report, not permission to substitute ordinary target sampling. Production integration additionally needs per-request stopping, output limits, and cache rollback.

# Pseudocode: B equal-length unpadded prompts, plain categorical sampling.
# propose returns full prompt + k tokens and the full q used for each draw.
B, P = input_ids.shape                         # P >= 1; k >= 1
draft_ids, draft_probs = draft_model.propose(input_ids, k=k)
proposal_ids = draft_ids[:, P:]                # [B, k]
# draft_ids: [B, P+k]; draft_probs: [B, k, V]
# Full-prefix computation here; no persistent KV state is passed or returned.
target_logits = target_model(draft_ids, use_cache=False).logits
target_probs = target_logits[:, P-1:P+k-1, :].float().softmax(dim=-1)
extra_target_probs = target_logits[:, P+k-1, :].float().softmax(dim=-1)
# target_probs[:, i, :] predicts proposal_ids[:, i] after its causal prefix.
p_selected = target_probs.gather(-1, proposal_ids[..., None]).squeeze(-1)
q_selected = draft_probs.gather(-1, proposal_ids[..., None]).squeeze(-1)
# q_selected is positive because these tokens were sampled from draft_probs.
acceptance_prob = (p_selected / q_selected).clamp(max=1.0)  # [B, k]
accept_mask = torch.rand_like(acceptance_prob) < acceptance_prob
accepted_prefix_len = first_false_index_or_k(accept_mask)  # [B]
# Helper returns B variable-length sequences, each of length P+n+1.
# At n < k: normalize max(target_probs[b,n,:] - draft_probs[b,n,:], 0).
# At n == k: sample extra_target_probs[b,:]. Append after the retained prefix.
committed_ids = commit_prefix_residual_or_extra_target(
    input_ids, proposal_ids, accepted_prefix_len,
    target_probs, draft_probs, extra_target_probs
)

Result to check: Each returned row contains its original prompt, the accepted prefix, and exactly one correction or extra target token. This is pseudocode: first_false_index_or_k and commit_prefix_residual_or_extra_target are specified by their behavior here, not implemented in this listing.

A performance result is usable only when the workload and environment are recorded, warmup is separated from measured work, the optimized output is checked, and the timing boundary matches the reported metric. Distributed experiments add rank identity, topology, synchronized timing, and cross-rank validation to the same discipline.

When work spans GPUs, the next question is what data another worker needs and how that dependency delays progress. Communication paths and costs (Section 16.4) separates transfer time from rank waiting before the group operations and distributed layouts are examined.

Optional distributed labs

These optional exercises turn two core mechanisms into runnable checks. Each starts from its own program and can be done alone after the distributed chapters. They require a compatible CUDA-enabled PyTorch build, working NVIDIA GPUs, and a torchrun command available in the same environment. A local run here was not possible; expected outputs are checks for a reader to verify, not observed results.

Collective result and timing probe

Example: NCCL collective timing probe

This distributed probe shows the objects involved in a simple all-reduce: process group, rank-assigned GPU, tensor payload, synchronization, and elapsed time. It requires a CUDA/NCCL-enabled PyTorch installation and at least two available GPUs. Save the listing as nccl_probe.py, then launch it on one node with torchrun --standalone --nproc-per-node=<gpu_count> nccl_probe.py, where <gpu_count> is at least 2 and no larger than the number of available GPUs. The launch setup in Section 17.1 explains the rank variables. Expect rank 0 to print slowest_rank_mean_ms and estimated_ring_payload_GBps after the checks pass; a one-rank run has no useful inter-rank rate. Its tensor payload is 512 MiB per rank before communication and checking buffers. Each warmup and timed all-reduce starts from a known payload value. The final result of each loop is checked for finite values and the expected world-size sum outside the recorded timing intervals. Ranks combine their pass/fail status before raising, so one bad result stops the group before the next collective.

import os
import torch
import torch.distributed as dist

dist.init_process_group("nccl")
try:
    local_rank = int(os.environ["LOCAL_RANK"])
    torch.cuda.set_device(local_rank)
    device = torch.device("cuda", local_rank)
    payload = torch.zeros(256 * 1024 * 1024, device=device, dtype=torch.float16)
    world = dist.get_world_size()

    def check_all_ranks(stage):
        status = torch.tensor([
            int(torch.isfinite(payload).all().item()),
            int((payload == world).all().item()),
        ], device=device, dtype=torch.int32)
        dist.all_reduce(status, op=dist.ReduceOp.MIN)
        if not status.all().item():
            raise AssertionError(
                f"{stage}: at least one rank failed "
                f"finite={bool(status[0].item())}, expected_sum={bool(status[1].item())}"
            )

    for _ in range(5):
        payload.fill_(1.0)
        dist.all_reduce(payload, op=dist.ReduceOp.SUM)
    torch.cuda.synchronize()
    check_all_ranks("warmup")
    dist.barrier()

    samples_ms = []
    for _ in range(20):
        payload.fill_(1.0)
        start = torch.cuda.Event(enable_timing=True)
        stop = torch.cuda.Event(enable_timing=True)
        start.record()
        dist.all_reduce(payload, op=dist.ReduceOp.SUM)
        stop.record()
        stop.synchronize()
        samples_ms.append(start.elapsed_time(stop))
    check_all_ranks("measured loop")

    mean_ms = torch.tensor(sum(samples_ms) / len(samples_ms), device=device)
    dist.all_reduce(mean_ms, op=dist.ReduceOp.MAX)
    ring_bytes = 2 * (world - 1) / world * payload.numel() * payload.element_size()
    effective_gb_s = ring_bytes / (mean_ms.item() / 1000) / 1e9

    if dist.get_rank() == 0:
        print(f"slowest_rank_mean_ms={mean_ms.item():.3f}")
        print(f"estimated_ring_payload_GBps={effective_gb_s:.1f}")
finally:
    dist.destroy_process_group()

The probe isolates one payload and one collective, so it can compare rank placement or transport configurations without model noise. On each node, run nvidia-smi topo -m to read the installed GPU-to-GPU paths and connections to qualified NICs shown in the matrix. It describes possible paths, not achieved bandwidth. For a diagnostic run, set NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH in the launching shell, then rerun the probe; inspect the transport and interface reported by the installed NCCL version’s logs (NCCL logging guide). Log fields vary by version, and a selected transport alone does not prove which link limited the run. The probe reports estimated per-rank ring payload divided by measured time, not an observed link rate. Use Table 3.2 to identify a specified peak or separately measured rate for the selected route, with matching direction and per-link, per-port, or aggregate scope. Compare only after those accounting scopes match. Inspect an aligned Nsight Systems trace before assigning a gap to the fabric. Do not treat a specified peak as a measured rate.

Replicated update with DDP

Start this exercise independently. Its listing creates a synthetic dataset and model, initializes its own process group, maps each rank to a GPU, and closes the group when it finishes. Check that the same CUDA/NCCL-enabled environment can see one GPU per worker and that no other job holds them. Compare only runs with the same rank count, batch rule, model, and timing boundary.

Example: DDP training template

This synthetic template requires a CUDA/NCCL-enabled PyTorch installation and one available GPU per worker. Save the listing as script.py and run torchrun --standalone --nproc-per-node=<gpu_count> script.py with 2 to 16 GPUs. It shows rank-to-device mapping, data sharding, backward synchronization, and optimizer updates. The first five steps warm up the run; after a rank barrier, each rank times 20 complete forward, backward, and optimizer steps. Data loading and host-to-GPU copies happen before each timer starts. Rank 0 reports the largest rank-local mean, without comparing clocks across GPUs. After training, each rank checks its final loss and parameters for finite values, a nonzero parameter change, and its parameters against rank 0’s copied values with rtol=1e-5 and atol=1e-6 (PyTorch tensor comparison). A successful run prints the timing and replicas_close=True. This one-layer model does not demonstrate overlap between multiple gradient buckets. If the dataset length is not divisible by the rank count, the sampler’s default padding repeats some examples to give ranks equal-length streams.

import os
import time
import torch
import torch.distributed as dist
import torch.nn as nn
from torch.utils.data import DataLoader, DistributedSampler, TensorDataset
from torch.nn.parallel import DistributedDataParallel as DDP

def train_one_rank():
    dist.init_process_group("nccl")
    try:
        local_rank = int(os.environ["LOCAL_RANK"])
        torch.cuda.set_device(local_rank)
        device = torch.device("cuda", local_rank)

        feature_count = 1024
        class_count = 100
        num_epochs = 2
        generator = torch.Generator().manual_seed(0)
        dataset = TensorDataset(
            torch.randn(4096, feature_count, generator=generator),
            torch.randint(class_count, (4096,), generator=generator),
        )
        sampler = DistributedSampler(dataset, shuffle=True)
        loader = DataLoader(
            dataset, batch_size=8, sampler=sampler, num_workers=4, pin_memory=True
        )
        warmup_steps, timed_steps = 5, 20
        assert len(loader) >= warmup_steps + timed_steps
        model = DDP(nn.Linear(feature_count, class_count).to(device), device_ids=[local_rank])
        initial = [p.detach().clone() for p in model.module.parameters()]
        loss_fn = nn.CrossEntropyLoss()
        optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)

        samples_ms = []
        last_loss = None
        for epoch in range(num_epochs):
            sampler.set_epoch(epoch)
            for step, (features, labels) in enumerate(loader):
                features = features.to(device, non_blocking=True)
                labels = labels.to(device, non_blocking=True)
                if epoch == 0 and step == warmup_steps:
                    torch.cuda.synchronize(device)
                    dist.barrier()
                timed = epoch == 0 and warmup_steps <= step < warmup_steps + timed_steps
                if timed:
                    torch.cuda.synchronize(device)
                    start = time.perf_counter()
                loss = loss_fn(model(features), labels)
                loss.backward()
                optimizer.step()
                optimizer.zero_grad(set_to_none=True)
                if timed:
                    torch.cuda.synchronize(device)
                    samples_ms.append((time.perf_counter() - start) * 1000)
                last_loss = loss.detach()

        assert len(samples_ms) == timed_steps
        finite_loss = last_loss is not None and torch.isfinite(last_loss).item()
        finite_parameters = all(
            torch.isfinite(p).all().item() for p in model.module.parameters()
        )
        max_change = max(
            (p.detach() - before).abs().max().item()
            for p, before in zip(model.module.parameters(), initial)
        )
        replica_error = None
        for parameter in model.module.parameters():
            reference = parameter.detach().clone()
            dist.broadcast(reference, src=0)
            try:
                torch.testing.assert_close(
                    parameter.detach(), reference, rtol=1e-5, atol=1e-6
                )
            except AssertionError as error:
                replica_error = str(error)

        local_ok = torch.tensor(
            int(finite_loss and finite_parameters and max_change > 0
                and replica_error is None), device=device
        )
        dist.all_reduce(local_ok, op=dist.ReduceOp.MIN)
        assert local_ok.item() == 1, (
            f"Training check failed on rank {dist.get_rank()}: "
            f"finite_loss={finite_loss}, finite_parameters={finite_parameters}, "
            f"max_change={max_change}, replica_error={replica_error}"
        )

        rank_mean_ms = torch.tensor(sum(samples_ms) / timed_steps, device=device)
        dist.all_reduce(rank_mean_ms, op=dist.ReduceOp.MAX)
        if dist.get_rank() == 0:
            print(f"slowest_rank_mean_step_ms={rank_mean_ms.item():.3f}")
            print("replicas_close=True")
    finally:
        dist.destroy_process_group()

if __name__ == "__main__":
    train_one_rank()

Backward propagation triggers distributed communication in this template: DDP hooks mark gradient buckets ready and submit their reductions while other backward work may still run. The reported time includes rank waiting inside the timed step but excludes data loading and transfers, so compare only runs with the same boundary. The copied rank-0 tensors give a direct replica check; equal checksums alone could hide different tensors. This one-layer run establishes the training and checking path, while a separate model with several gradient buckets is needed to study bucket overlap.


  1. NVIDIA. (n.d.). Nsight Systems User Guide. Retrieved October 11, 2026, from the Nsight Systems command-line and report guide. Documents trace categories, disabled CPU sampling, .nsys-rep report generation, and report opening in the GUI. Trace support depends on the target platform and installed release.↩︎

  2. PyTorch contributors. (2.14 documentation). torch.profiler and torch.cuda.synchronize. Profiler scheduling and Chrome-trace export and CUDA device synchronization. The schedule separates wait, warmup, and active steps; synchronization without a device argument uses the current device. Multiple-device completion needs the relevant device boundaries.↩︎

  3. PyTorch contributors. (2.14 documentation). torch.mm and CUDA semantics. FP32 output from CUDA FP16 matrix inputs and CUDA precision and timing controls. Support out_dtype=torch.float32 on CUDA for FP16/BF16 inputs, precision settings, and event timing. The benchmark fixes its own input/output storage, tolerances, repetitions and summary statistic; it does not claim identical reduction order or a measured production speedup.↩︎

  4. PyTorch contributors. (2026). Common Graph Breaks (PyTorch 2.14 documentation, Data-dependent operations). PyTorch 2.14 Common Graph Breaks. Distinguishes data-dependent Python control flow from tensor operations. Scalar capture settings can change .item() behavior, so the example records diagnostics rather than assuming a fixed break count.↩︎

  5. PyTorch contributors. (2.14 documentation). torch._logging.set_logs. PyTorch compiler logging controls. Documents graph_breaks and recompiles diagnostics and the precedence of the TORCH_LOGS environment variable over this programmatic logging interface.↩︎

  6. PyTorch. (2.14 documentation). Tensor.copy_. Tensor.copy_. Allows a broadcastable source tensor with a different dtype or device. The replay example imposes a narrower same-shape, same-dtype, same-device input contract to keep the measured workload fixed.↩︎

  7. PyTorch contributors. (2.14 documentation). torch.autograd.profiler.record_function. Profiler phase labels. Adds a label to a block or function while profiling; CPU activity must be enabled for the label to appear. The label marks CPU execution and submission, not a separately synchronized GPU duration.↩︎