Appendix A — References
Reference Material
Provide compact formulas, software-control ownership, practice checks, and source mappings for implementing and checking the methods taught in the chapters.
The main chapters explain mechanisms and trade-offs. This appendix provides formulas, software controls, and supporting research and documentation for quick reference.
Core formulas
These equations summarize the guide’s first-order models. Byte counts refer to the memory tier being studied, storage depends on the numerical format, and theoretical peaks differ from achieved rates.
The KV-cache product assumes equally long retained sequences, the same key/value-head dimensions in every counted layer, and separate cache storage for each sequence. Shared prefixes, sliding windows, and other cache representations require different counts. For more than one output token, TPOT is the mean interval after the first token. With one output token, end-to-end latency is approximately TTFT. Zero output tokens are outside this model. Ring traffic counts bytes sent by one rank, not sent and received bytes added together. The full-mesh expression counts one undirected link per pair of distinct endpoints. Activation storage is an approximation for the saved tensors of the particular training implementation.
\[ \text{AI} = \frac{\text{FLOPs}}{\text{bytes moved}} \tag{A.1}\]
\[ \text{achieved throughput} = \frac{\text{FLOPs}}{\text{seconds}} \tag{A.2}\]
\[ \text{roofline bound} = \min(\text{peak compute}, \text{AI} \times \text{peak bandwidth}) \tag{A.3}\]
\[ \text{AI}_{\text{ridge}} = \frac{\text{peak compute}}{\text{peak memory bandwidth}} \tag{A.4}\]
\[ \text{KV bytes} = 2 \times B \times T \times L \times H_{\text{kv}} \times d_h \times b \tag{A.5}\]
\[ \text{E2E latency} \approx \text{TTFT} + (\text{number of output tokens} - 1) \times \text{TPOT} \tag{A.6}\]
\[ \text{communication time} \approx \text{latency} + \frac{\text{bytes transferred}}{\text{effective bandwidth}} \tag{A.7}\]
\[ \text{ring all-reduce bytes sent per rank} \approx \frac{2(N - 1)}{N} \times \text{tensor}_{\text{bytes}} \tag{A.8}\]
\[ \text{full-mesh links} = \frac{N(N - 1)}{2} \tag{A.9}\]
\[ \text{pipeline bubble fraction} \approx \frac{K - 1}{M + K - 1} \tag{A.10}\]
\[ \text{training activation bytes} \approx b L \left(n_{\text{hidden}} B T d + n_{\text{score}} B H T^2\right) \tag{A.11}\]
\[ \text{expected speculative advance} = \sum_{i=0}^{\gamma} \alpha^i \tag{A.12}\]
\[ \text{speculative speedup estimate} = \frac{\sum_{i=0}^{\gamma} \alpha^i}{\gamma c + 1} \tag{A.13}\]
The symbols below use the same meanings and units as their first derivations in the main chapters.
| Symbol | Meaning and unit | Defined in |
|---|---|---|
| AI | Arithmetic intensity, measured in floating-point operations per byte moved at the stated memory tier | 6.2, 6.3 |
| B | Batch size or number of active sequences, according to the formula | 9.2, 10.6 |
| T | Sequence length in tokens (stored tokens per sequence in the KV-cache formula) | 9.2, 10.6 |
| L | Number of model layers represented by the estimate | 9.2, 10.6 |
| H_kv | Number of key-value heads per attention layer | 10.6 |
| d_h | Elements in one key or value head vector | 10.6 |
| b | Bytes per stored element | 9.2, 10.6 |
| d | Model hidden dimension in elements | 9.2 |
| H | Number of attention heads in the saved attention-score tensors | 9.2 |
| n_hidden | Implementation-dependent count of retained hidden-sized tensors per layer | 9.2 |
| n_score | Implementation-dependent count of retained attention-score-like tensors per layer | 9.2 |
| N | Number of communication ranks or topology endpoints | 16.2, 16.5 |
| K | Number of pipeline stages | 20.1 |
| M | Number of micro-batches in the GPipe estimate | 20.1 |
| alpha | Average speculative-token acceptance probability | 14.1.1 |
| gamma | Number of draft tokens proposed per speculative iteration | 14.1.1 |
| c | Draft-step cost divided by target-step cost in the simplified model | 14.1.1 |
The pipeline expression is a balanced GPipe-style fill/drain estimate.1 The activation expression repeats Section 9.2: n_hidden counts retained hidden-sized tensors per layer, and n_score counts retained attention-score-like tensors. The latter contributes a quadratic sequence-length term when those scores are saved. An implementation that does not retain these score tensors has n_score = 0; its saved-activation estimate is linear in sequence length under otherwise fixed counts. These counts depend on the implementation and recomputation policy, not just the architecture.
The speculative expressions assume constant independent acceptance and a simple sequential draft-cost model, with \(0 \leq \alpha \leq 1\) and \(c \geq 0\).2 Writing the geometric series as a finite sum also covers perfect acceptance: at \(\alpha=1\), expected advance is \(\gamma+1\) tokens and estimated speedup is \((\gamma+1)/(\gamma c+1)\).
Each formula answers a limited question. The roofline estimate compares arithmetic work with memory traffic. The KV-cache formula estimates storage. The latency and communication formulas estimate time for the stated workload. None accounts for every delay in a running system. Profiles reveal waiting between launches, competing work, and transfers that overlap with computation.
Software layers and controls
Each group identifies the component that interprets a setting and the behavior it can change. A setting can also affect other software layers.
PyTorchmodel and tensor layer:PyTorchdefines tensors, modules, automatic differentiation, optimizers, and high-level device operations.torch.cuda: Device, stream, event, memory-statistic, synchronization, and CUDA Graph APIs select devices and expose execution controls. Use them to select or confirm a device, time device work, and inspect memory; use traces to identify the kernel path.torch.profiler: RecordsPyTorchoperations and their associated CPU, CUDA, tensor-shape, and memory events.torch.compile(...): Requests compiled execution through thePyTorchAPI.TorchDynamo,AOTAutograd, andTorchInductorimplement compatible parts of that request. Guards, graph breaks, and backend support determine the result.torch.distributed: Creates process groups and submits point-to-point or collective communication through a selected backend.
- Model and checkpoint layer: Model libraries construct tokenizers and architectures and interpret checkpoint metadata. Serialization libraries store tensor payloads.
Transformers: Provides model configuration, tokenizer factories, checkpoint model classes, and generation baselines for supported architectures.safetensors: Stores tensor payloads with explicit metadata and avoids executable pickle payloads. Loading speed still depends on storage and placement.
- CUDA and kernel layer: The CUDA runtime and driver manage contexts, memory, streams, events, modules, and launches. Vendor libraries and generated kernels perform the device work.
cuBLASandcuBLASLt: Tuned dense linear-algebra kernels are selected according to dimensions, layout, dtype, alignment, and hardware support.cuDNN: Provides tuned implementations for supported neural-network operations.- Triton: Expresses custom tile-oriented GPU kernels with program IDs, address calculations, masks, loads, computation, and stores.
CUDA_VISIBLE_DEVICES: Changes which GPUs a process can see. It does not move tensors or improve a kernel by itself.PYTORCH_CUDA_ALLOC_CONF: Changes selectedPyTorchcaching-allocator policies when allocation behavior or fragmentation is being investigated.
- Compiler and representation layer: Compiler components transform compatible tensor programs, while quantization tools prepare lower-precision representations that still require a supported execution path.
TorchDynamo,AOTAutograd, andTorchInductor: These components capture Python/PyTorchregions, stage differentiable graphs, and lower them to generated kernels or library calls.bitsandbytesandtorchao: Provide selected low-precision weight, optimizer, sparsity, or quantization paths. Exact model, format, GPU, and kernel support is version-dependent.
- Serving layer: A serving engine admits requests, schedules prefill and decode work, allocates KV-cache blocks, and records service metrics.
- vLLM and SGLang: Examples of serving engines with request scheduling, KV-cache management, and prefix-reuse capabilities whose exact interfaces change by release.
TensorRT-LLM: NVIDIA’s inference stack for supported models and hardware, including multi-GPU configurations. Documentation retrieved September 28, 2026 describesPyTorch-native execution; older releases had an engine-build workflow (Section 15.2.2).max_num_batched_tokens: A vLLM-style token budget that limits how much work the scheduler admits in an iteration.3--gpu-memory-utilization: A vLLM-style reservation target used when dividing device capacity among model state, workspaces, and KV cache.--max-model-len: A vLLM-style context limit that changes the worst-case KV-cache requirement.--kv-cache-dtype: Selects KV-cache storage precision only when the engine and attention kernels support that format.
- Communication layer: NVIDIA Collective Communications Library (
NCCL) selects collective algorithms and transports for CUDA tensors on NVIDIA systems. The physical path still depends on topology and configuration.NCCL_DEBUG=INFO: Enables informational logging for the selectedNCCLsubsystems. The separateNCCL_DEBUG_SUBSYSfilter determines which records appear. In the cited documentation, the default isINIT,BOOTSTRAP,ENV; topology, tuning, and network diagnosis can requireGRAPH,TUNING,NETorALL. The available categories and defaults depend on the installedNCCLversion.4NCCL_SOCKET_IFNAME: Selects the network interface used byNCCLsocket transports.NCCL_IB_DISABLE: Disables InfiniBand/RDMA paths for a controlled fallback comparison.NCCL_P2P_DISABLE: Disables direct peer-to-peer GPU paths for a controlled topology comparison.
- Measurement layer: Profiler selection follows the object under inspection:
PyTorchoperations, gaps in the CPU/GPU timeline, or the instructions and memory accesses of one kernel. The tools below provide these different views.PyTorchProfiler: Maps framework operations to CPU and CUDA activity and can export a trace.- Nsight Systems: Shows process, CUDA, copy, kernel, and collective timing on a system timeline.
- Nsight Compute: Inspects one GPU kernel’s instructions, occupancy, memory transactions, and stalls.
- Perfetto: Displays exported Chrome-trace timelines for interactive inspection.
Version, GPU architecture, dtype, shape, and topology can change whether a control is supported or useful. Logs and traces establish which kernel or transport actually ran. A setting alone does not establish its effect. The corresponding explanations and source notes are in Chapter 4, Chapter 8, Chapter 15, and Chapter 16.
Compatibility checks
A package can install successfully and the program can run without using the expected optimized kernel. The attention kernel, compiled region, or communication path that actually ran is established from runtime evidence. The runtime may have selected a slower supported alternative.
Checklist: Compatibility evidence
- Hardware and driver: The record contains the GPU model, memory capacity, driver, framework-visible CUDA runtime, and interconnect topology.
- Software versions: The record contains
PyTorch,Triton, attention backend,Transformers, quantization library, and serving-engine versions. - Attention path: Support covers dtype, head dimension, causal or other masks, grouped-query layout, sequence length, and training or inference mode.
- Quantization path: The weight and KV formats have kernels for the target GPU, and accuracy is evaluated on the intended workload.
- Compiler and graph path: Keep compilation and warmup out of steady-state timing. Establish graph breaks, recompilations, and fixed-address requirements before CUDA Graph capture.
- Communication path:
NCCLor another backend selects the intended local or network transport, and profiler traces agree with the assumed topology.
A compatibility check is complete only when the observed execution path matches the intended path. Package versions are necessary evidence, but kernel names, compiler logs, memory accounting, and communication traces show what actually ran.
Practice checks
The twelve questions below mix numerical estimates, conceptual explanations, and code reading. Work each problem without the answer key, state assumptions and units, and name the measurement that would confirm the result.
- P1 KV-cache budget: For 4 active sequences, 2,048 stored tokens, 32 layers, 8 key-value heads, head dimension 128, and 2-byte elements, compute the raw KV-cache size in GiB.
- P2 Model-scale units: Using the Llama 3.3 70B configuration in Section 10.6, compare one sequence at 32,000 tokens with one sequence at 32,768 tokens. Report both cache sizes in GiB. Explain why writing only “32K tokens” can make the estimate ambiguous, and distinguish GB from GiB.
- P3 Roofline: A kernel performs 2e12 FLOPs, moves 1e12 bytes, and runs in 0.020 seconds on a device rated at 200 TFLOP/s and 3 TB/s. Compute arithmetic intensity, the memory ceiling, and achieved throughput. Then explain the result.
- P4 Ring all-reduce: Eight ranks run ring all-reduce on a 1 GiB tensor. Use the first-order traffic formula to estimate bytes sent per rank, then list two reasons a trace can differ from the estimate.
- P5 Pipeline bubble: A pipeline has four stages and eight micro-batches. Use the bubble-fraction formula to estimate the idle fraction before considering load imbalance or communication.
- P6 Prefill and decode: Why can transformer projections be GEMM-like during prefill and GEMV-like during small-batch decode? Which metric may improve and which may worsen when batching increases?
- P7 FlashAttention: Which mathematical result does FlashAttention preserve, which intermediate does it avoid materializing in HBM, and which performance resource does it target? Which serving concern remains unresolved?
- P8 PagedAttention: A request maps logical KV blocks [0, 1, 2] to physical blocks [7, 2, 15]. Explain what remains ordered, what becomes non-contiguous, and why this helps admission capacity.
- P9 Software controls: For each setting, identify the component that interprets it and describe the work or data it can change:
torch.compile(...),--kv-cache-dtype,tl.load(..., cache_modifier=...), andNCCL_DEBUG=INFO. - P10 Speculative decoding: Give two conditions that can reduce latency and two conditions that can erase the gain. Include target-model verification and rejected-token correction.
- P11 Triton addresses and masks: A contiguous row-major matrix has shape
(3, 10). A program handles row 1 and the last tile has column positions[8, 9, 10, 11]. Find the element offsets and the valid-load mask. Explain howtl.loaduses this mask to prevent reads from the next row, withother=0supplying zero for invalid positions. State why a different layout needs its actual strides. - P12 Distributed Data Parallel: Two ranks start with equal parameters and optimizer state, use the same optimizer rule and settings, process different local batches, and average their gradients. Explain why they can apply the same parameter update. State what happens before, during and after gradient synchronization. Identify one local cause and one cross-rank cause of a slow step.
Several answers are first-order diagnoses, so a real experiment still tests cache effects, overlap, fragmentation, graph breaks, or topology.
Answer key
A different answer can still be correct when it states its assumptions, units, and supporting measurement clearly.
- P1 KV-cache budget: The raw size is 1,073,741,824 bytes, exactly 1 GiB. This counts separate, equally long caches for all four sequences. Block metadata, fragmentation, and workspaces require additional memory. Prefix sharing is excluded.
- P2 Model-scale units: At 32,000 tokens the result is 10,485,760,000 bytes, about 9.77 GiB. At 32,768 tokens it is 10,737,418,240 bytes, exactly 10 GiB. State whether K means decimal thousands or 1,024-token units.
- P3 Roofline: Arithmetic intensity is 2 FLOPs per byte. The memory ceiling is 6 TFLOP/s, but the stated runtime implies 100 TFLOP/s. The values contradict the simple model, so recheck the byte count, cache assumption, FLOP count, and timing boundary.
- P4 Ring all-reduce: Each rank sends 1.75 GiB and receives 1.75 GiB. The formula counts one direction, not the sum of sent and received bytes. Algorithm choice, chunking, topology, protocol overhead, overlap, contention, and rank skew can change a trace.
- P5 Pipeline bubble: The first-order idle fraction is 3/11, or about 27.3 percent. Stage imbalance, communication, and scheduling overhead change the measured result.
- P6 Prefill and decode: Prefill exposes many token positions and reuses weights across larger matrix operations, so it can raise compute use and mainly affects TTFT. Small-batch decode adds one token per request while rereading weights and KV history. Batching can improve throughput while increasing queueing or TPOT.
- P7 FlashAttention: It preserves the attention result while avoiding a full quadratic score or probability matrix in HBM. It reduces off-chip memory traffic. It does not allocate KV blocks, admit requests, or choose a serving schedule.
- P8 PagedAttention: Logical token order remains [0, 1, 2], while physical blocks can be [7, 2, 15]. On-demand fixed-size allocation reduces reserved waste and fragmentation at the cost of block-table lookup and scheduler/kernel complexity.
- P9 Software controls:
PyTorch’s compiler path interpretstorch.compile(...), captures compatible regions, and may lower them to generated kernels or library calls. The serving engine interprets--kv-cache-dtype: it changes stored KV bytes and can change which attention kernels are supported.Tritonkernel code suppliescache_modifier, and theTritoncompiler turns it into a supported load-cache hint fortl.load.NCCLreadsNCCL_DEBUG=INFOand emits diagnostic logs; that setting alone does not choose a collective or transport. Use logs or traces to confirm the execution path. - P10 Speculative decoding: It helps when the draft is cheap, acceptance is high, and target verification is efficient. Low acceptance, a costly draft, a saturated target batch, or rollback, cache, sampling, and scheduling overhead can erase the gain. After a rejection, the method samples a correction from the residual distribution and discards the remaining draft suffix; frequent rejection wastes draft and verification work.
- P11 Triton addresses and masks: The candidate offsets are
1 * 10 + [8, 9, 10, 11] = [18, 19, 20, 21]. The mask is[True, True, False, False]. Only offsets 18 and 19 belong to this row. A masked load with zero for the other positions prevents those positions from reading row 2. The formula uses contiguous row-major strides; another layout must use its row and column strides. - P12 Distributed Data Parallel: Each rank first clears old gradients, computes its local prediction and loss, and runs backward. Gradient buckets can be reduced as they become ready. After the reductions complete, every rank has the same averaged gradients, so equal parameters and optimizer state with the same update rule produce equal new parameters. The next forward uses those updated values. A slow local kernel is a local cause; rank skew, topology contention or exposed all-reduce time are cross-rank causes.
Source map
The source map records where lecture, summary, notebook, and externally researched material contributes to the guide.
| Source | Mapped units | Coverage status | Guide sections |
|---|---|---|---|
| [Py4DP-L3] [Optional] PyTorch on GPU.ipynb | cell 1-58 (58 mapped) | 58 cells combined into the explanation | 4.1 |
| [Py4DP-L3] Gradients & Logistic regression in PyTorch.ipynb | cell 1-56 (56 mapped) | 56 cells combined into the explanation | 2, 9, 18 |
| [Py4DP-L3] PyTorch basics.ipynb | cell 1-78 (78 mapped) | 78 cells combined into the explanation | 1, 2, 9, 10 |
| 04 _ Performance Engineering - Week 01 _ Systems, GPUs & Networking - Summary.docx | unit 1-82 (74 mapped) | 70 units covered; 4 omitted | Optional decode and serving labs, Optional GPU and runtime labs, 3, 3.1, 3.2, 3.3, 3.4, 3.5, 4, 4.2, 4.5, 4.6, 6, 6.4, 6.5, 7, 7.1, 7.2, 7.4, 8, 9, 10.2, 10.3, 13, 15, 16.1, 16.6, 17, 18, 19 |
| 04 _ Performance Engineering - Week 02 _ Efficient Inference- Architecture, Quantization & Compilers - Summary.docx | unit 1-57 (53 mapped) | 51 units covered; 2 omitted | 7.1, 10.2, 10.3, 10.6, 12.4, 12.5, 13.2, 13.2.1, 13.2.2 |
| hw1_roofline.ipynb | unit 1-18 (18 mapped) | 18 units covered | Performance probes and checks, 6, 6.4, 6.5, 7.2, 7.4, 18 |
| hw2_decode_optimization.ipynb | unit 1-17 (17 mapped) | 17 units covered | Performance probes and checks, 7.4, 12.1, 12.2, 12.3 |
| hw3_compile_cuda_graphs.ipynb | unit 1-21 (21 mapped) | 21 units covered | Performance probes and checks, 7.4, 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.8 |
| lecture 1 - foundations of supervised training [Autosaved].pptx.pdf | unit 1-35 (35 mapped) | 14 units covered; 21 omitted | 2, 6, 9, 10.5, 18, 18.4 |
| lecture 2 - Data Parallel Training (DDP) [Autosaved].pptx.pdf | unit 1-41 (41 mapped) | 32 units covered; 9 omitted | Optional distributed labs, 7, 7.4, 16, 17, 17.1, 18, 18.1, 18.2, 19, 20, 21 |
| lecture 3 - Modern Large-Model Training Techniques.pptx.pdf | unit 1-61 (61 mapped) | 34 units covered; 27 omitted | 3.6, 10.4, 10.5, 17.1, 18.4, 19, 19.3, 20 |
| Performance Engineering - L1.pdf | unit 1-89 (89 mapped) | 83 units covered; 6 omitted | Optional decode and serving labs, Optional GPU and runtime labs, 3, 3.1, 3.2, 3.3, 3.4, 3.5, 4, 4.2, 4.5, 4.6, 6, 6.4, 6.5, 7, 7.1, 7.2, 7.4, 8, 9, 10.2, 10.3, 13, 15, 16.1, 16.6, 17, 18, 19 |
| Performance Engineering - L2-2.pptx.pdf | unit 1-139 (139 mapped) | 133 units covered; 6 omitted | A.2 Software layers and controls, 8.2, 8.3, 8.4, 8.5, 8.6, 8.8, 8.8.2, 10.2, 10.3, 10.6, 12.4, 12.5, 12.6, 13, 13.1, 13.2, 13.2.1, 13.2.2, 13.3, 13.5, 13.5.1, 13.5.2, 13.5.3, 14.2, 15.1, 15.2, 16.1, 16.5, 21, 21.1 |
| Performance Engineering - L4.pdf | unit 1-61 (61 mapped) | 56 units covered; 5 omitted | Performance probes and checks, Roofline calculation template, Attention backend comparison, 4.2, 4.5, 4.6, 4.7, 5, 5.2, 5.3, 5.5, 5.6, 7.2, 8, 8.4, 8.7 |
| Performance Engineering - L6.pdf | unit 1-81 (81 mapped) | 79 units covered; 2 omitted | 9.4, 11.1, 11.2, 11.4, 12.6, 13, 13.5, 13.5.1, 13.5.2, 13.5.3, 14.2, 15.1, 15.2, 16.1, 16.5, 21, 21.1 |
| solution_1.1.ipynb | unit 1-13 (13 mapped) | 11 units covered; 2 omitted | Distribution-correct speculative step, Performance probes and checks, 14.1, 14.1.1 |
| solution_1.4.ipynb | unit 1-28 (28 mapped) | 27 units covered; 1 omitted | Distribution-correct speculative step, Vectorized speculative verification, Performance probes and checks, 14.1, 14.1.1 |
| Speculative Decoding Paper Summary.docx | unit 1-58 (54 mapped) | 45 units covered; 9 omitted | Distribution-correct speculative step, Performance probes and checks, 10.2, 12.1, 14.1, 14.1.1 |
| Speculative decoding workshop (June 19, 2026) | unit 1-25 (25 mapped) | 9 units covered; 16 omitted | Distribution-correct speculative step, Performance probes and checks, 14.1, 14.1.1 |
| workshop.ipynb | unit 1-8 (8 mapped) | 7 units covered; 1 omitted | Distribution-correct speculative step, Performance probes and checks, 14.1, 14.1.1 |
| master_outline.txt | document 1 | 1 omitted | No mapped guide section; see source notes |
The source map records the extracted inventory: PDF pages, nonempty notebook source cells, nonempty DOCX body paragraphs, and whole-document outline records. Each record identifies where the material is used and why it was included, combined, treated as a duplicate, or omitted. Blank cells and paragraphs and text inside DOCX tables are not separate inventory units; the record count does not establish that every physical element was reviewed. The selected external sources below connect major methods to their teaching sections. The References section collects the cited works.
| Primary reference | Date | Guide section | URL |
|---|---|---|---|
| Dao et al. (2022) FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness | 2022-05-27 | 5.7 | https://arxiv.org/abs/2205.14135 |
| Kwon et al. (2023) Efficient Memory Management for Large Language Model Serving with PagedAttention | 2023-09-12 | 12.5 | https://arxiv.org/abs/2309.06180 |
| Zheng et al. (2024) SGLang: Efficient Execution of Structured Language Model Programs | 2024-06-06 | 13.5.2 | https://arxiv.org/abs/2312.07104 |
| Leviathan et al. (2023) Fast Inference from Transformers via Speculative Decoding | 2023-05-18 | 14.1 | https://arxiv.org/abs/2211.17192 |
| Rajbhandari et al. (2020) ZeRO: Memory Optimizations Toward Training Trillion Parameter Models | 2020-05-13 | 18.4.2 | https://arxiv.org/abs/1910.02054 |
| Shoeybi et al. (2019) Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | 2019-09-17 | 19.1 | https://arxiv.org/abs/1909.08053 |
| Huang et al. (2019) GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism | 2019-07-25 | 20.1 | https://arxiv.org/html/1811.06965v5 |
| Williams et al. (2009) Roofline: An Insightful Visual Performance Model for Multicore Architectures | 2009-04-01 | 6.5 | https://dl.acm.org/doi/10.1145/1498765.1498785 |
| Tillet et al. (2019) Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations | 2019-06-22 | 5.1 | https://dl.acm.org/doi/10.1145/3315508.3329973 |
Software interfaces and hardware behavior can change after the dates shown here. Check the installed version and official documentation before relying on a setting, backend, or limit.
Techniques by lever class
Section 7.5 names seven optimization levers, meaning seven kinds of change a technique can make. The table groups the guide’s techniques by the lever each one pulls, so that a measured bottleneck can be turned into a short list of candidate changes. The entries highlight the main levers discussed in the sections listed in the table. A technique can affect more than one.
| Lever | Technique | Section |
|---|---|---|
| Do less work | Key-value cache | 12.1 |
| Do less work | Prefix caching and radix-style reuse | 13.5 |
| Do less work | Grouped-query attention | 10.6 |
| Do less work | Pruning and structured sparsity | 11.5 |
| Do less work | Distillation | 11.6 |
| Do less work | Sliding-window attention and attention sinks | 12.7 |
| Move fewer bytes | Kernel fusion | 8.7 |
| Move fewer bytes | Tiled attention, such as FlashAttention | 5.7 |
| Move fewer bytes | Memory coalescing | 4.6 |
| Move fewer bytes | Tile order and cache hints | 5.4 |
| Move fewer bytes | Weight quantization | 11.1 |
| Move fewer bytes | Key-value cache quantization and latent cache | 12.7 |
| Use faster hardware paths | Tensor Cores and shape alignment | 3.3 |
| Use faster hardware paths | Vendor kernel libraries | 4.7 |
| Use faster hardware paths | Mixed-precision training | 9.4 |
| Use faster hardware paths | Compiled inference engines | 15.2.2 |
| Keep the hardware busy | Continuous batching | 13.2 |
| Keep the hardware busy | Speculative decoding (parallel target verification) | 14.1 |
| Keep the hardware busy | Chunked prefill | 13.3 |
| Keep the hardware busy | Streams and copy-compute overlap | 4.4 |
| Keep the hardware busy | Communication and computation overlap | 17.2 |
| Keep the hardware busy | Gradient bucket overlap in DDP | 18.3 |
| Cut fixed overhead | torch.compile capture and fusion |
8.2 |
| Cut fixed overhead | CUDA Graph capture and replay | 8.5 |
| Cut fixed overhead | Scheduler work off the decode path | 13.2.2 |
| Fit in memory | Activation checkpointing | 9.2 |
| Fit in memory | PagedAttention and block tables | 12.5 |
| Fit in memory | ZeRO and FSDP sharding | 18.4 |
| Fit in memory | Offload to CPU memory | 18.4.2 |
| Fit in memory | External key-value cache layers | 13.5.3 |
| Scale out | Data parallelism | 18.2 |
| Scale out | Tensor parallelism | 19.1 |
| Scale out | Pipeline parallelism | 20.1 |
| Scale out | Sequence and context parallelism | 19.4 |
| Scale out | Expert parallelism and mixture-of-experts serving | 21.1.2, 20.2 |
| Scale out | Separate prefill and decode workers | 13.4 |
A lever describes what a change does. The changed region’s share of runtime limits the possible speedup of the whole run, as described by Amdahl’s law in Section 6.1.
Glossary
Define the guide’s formal terms and provide links to their teaching, with separate API and notation references.
Model Workloads and Performance Limits
Teaching sources: Chapter 1, Chapter 2, Chapter 3, Chapter 6, Chapter 7, Chapter 10 and Chapter 13.
- tensor: A scalar, vector, matrix or higher-dimensional array used in model computation, with logical dimensions, a numeric format and a storage location.
- training: Computing predictions from examples, comparing them with targets through a loss, and updating the model’s weights using the resulting gradients.
- loss: A value that measures disagreement between model predictions and training targets and supplies the starting point for backward propagation.
- backward propagation: Computing derivatives of the loss through the operations that produced it, including derivatives with respect to model weights.
- weights: The model’s learned numeric values, also called parameters. Training updates them; inference uses them without updating them.
- model: A computation with learned numerical parameters that transforms inputs into outputs. Model and operating modes.
- forward pass: Applying a model to an input to compute predictions. Model and operating modes.
- inference: Producing an output from an input with fixed model parameters. Model and operating modes.
- evaluation: Comparing predictions with held-out reference answers to measure quality without updating model parameters. Model and operating modes.
- gradients: Derivatives of the loss with respect to model parameters, used by the optimizer to calculate an update. Model and operating modes.
- Shape: The logical dimensions of a tensor. Tensor descriptors.
- Dtype: The numerical representation of a tensor’s elements, determining their stored size and supported arithmetic. Tensor descriptors.
- Device: The CPU, GPU or other execution location holding a tensor’s storage. Tensor descriptors.
- Strides and layout: The mapping from tensor indices to storage positions. A stride counts stored elements skipped when one index increases by one. Tensor descriptors.
- Activations: Intermediate tensors produced during a model’s forward pass; training may retain selected values for backward propagation. Live state.
- Optimizer state: Persistent buffers an optimizer keeps to compute updates, such as Adam’s gradient and squared-gradient running averages. Live state.
- Temporary workspaces: Short-lived buffers used by kernels, libraries or runtimes while performing an operation. They still count toward peak live memory. Live state.
- Communication buffers: Intermediate tensor storage used for collective or point-to-point transfers. Live state.
- optimizer: An update rule that uses gradients, and sometimes retained state, to change model parameters.
- token: A unit from a tokenizer’s vocabulary represented by one token ID, such as a word, word part, punctuation, whitespace, byte sequence, or special control marker.
- prefill: The initial processing of a request’s prompt tokens, which creates the attention state used by later generation steps.
- decode: Repeated generation steps that advance an active sequence by one token at a time, using retained attention state when caching is enabled.
- Attention: An operation that compares a token’s query with permitted positions’ keys and uses the resulting weights to combine their values. See Section 1.1 and Section 5.6.
- key-value (KV) cache: Retained attention keys and values from earlier token positions, reused during decode to avoid recomputing that prefix state. See Section 2.1 and Section 12.1.
- bottleneck: The resource or dependency currently limiting one measured execution region. The limit can change with the workload phase and configuration.
- profiler: A tool that records evidence about where execution time, memory or hardware resources are used.
- CUDA events: Markers recorded in a GPU stream. Waiting for the end marker to complete lets an event timer report the GPU time between two markers. GPU timing.
- tokenizer: Software that splits text into tokens and maps each token to an integer ID.
- token ID: An integer position identifying one token in a tokenizer’s vocabulary.
- vocabulary: The fixed collection of tokens that a tokenizer represents with token IDs.
- embedding table: A learned matrix containing one vector per vocabulary entry. A token ID selects its corresponding row.
- hidden width: The number of numeric features in a token’s hidden vector, also called the model dimension.
- hidden state: The current feature vectors passed between model operations. For T tokens with hidden width d, the representation is a T by d matrix.
- layers: Transformer blocks with separately learned weights through which the hidden state passes. In a Llama-style decoder, each has attention and feed-forward sub-layers, each with normalization and a residual connection. The model workflow.
- attention sub-layer: The part of a transformer layer that combines information across permitted token positions. Section 1.1.
- feed-forward sub-layer: The part of a transformer layer that transforms each token’s vector independently, using the same learned matrices at every position. Section 1.1.
- RMSNorm: A model normalization operation that divides each token’s feature vector by its root-mean-square magnitude with a small stabilizing term, then applies learned per-feature scaling. The unweighted RMS calculation is a separate example in Section 8.8.1.
- projections: Multiplications by learned weight matrices that transform feature vectors, such as producing attention queries, keys and values.
- rotary position embedding (RoPE): A position-dependent rotation of attention queries and keys before their comparison, making that comparison depend on relative token positions. Rotated keys are retained for decode. The Q/K/V path.
- causal: Allowing a token position to use itself and earlier positions, but not later positions, in the attention calculation.
- attention heads: Separate attention calculations on head-sized query, key, and value vectors split from the projected vectors. Each projection reads the whole input vector. Query heads can share key and value heads in grouped-query attention.
- head dimension: The number of scalar components in an attention head’s vector, distinct from the number of heads and the full hidden width.
- multilayer perceptron (MLP): A feed-forward sub-layer applied independently to each token. The basic form uses two learned projections with a nonlinear operation between them; gated forms can use three projections.
- SwiGLU: A gated feed-forward form that applies SiLU to one projection, multiplies it element by element with another projection, and projects the result back to hidden width. See the gated computation in Section 19.1.
- residual connection: Adding a sub-layer’s output to its input. In the transformer architecture described in Section 1.1, the hidden state keeps its T by d shape from layer to layer.
- language-model head: The output projection from hidden features to vocabulary scores, producing one logit per vocabulary entry.
- logits: Unnormalized scores for the possible output tokens before softmax or a decoding rule selects a token.
- grouped-query attention: Attention in which several query heads share each key and value head, reducing the number of distinct key and value vectors stored per token.
- Multi-head attention (MHA): Attention with a separate key and value head for each query head.
- Multi-query attention (MQA): Attention in which all query heads share one key and value head.
- Softmax: Converting scores to probabilities by exponentiating each score and dividing by the sum of those exponentials.
- Greedy decoding: Selecting the most probable next token. It is deterministic for fixed inputs and arithmetic, but rounding differences can change a close choice.
- Temperature: A positive value used to divide logits before softmax. Values between zero and one concentrate probabilities; values above one spread them out. Greedy decoding is a separate highest-probability selection rule, not evaluation of the formula at zero.
- Top-k sampling: Sampling the next token from the k most probable tokens in proportion to their probabilities renormalized to sum to one.
- Top-p (nucleus) sampling: Keeping the shortest highest-probability prefix whose total probability reaches p, for 0 < p <= 1, then sampling from that prefix after renormalization.
- Beam search: Keeping several candidate continuations and extending the most probable ones, so a request runs several sequences.
- Mathematical tensor: The scalar, vector, matrix or higher-dimensional array used in a computation, as distinct from the bytes that store it.
- Tensor storage: The underlying allocated bytes holding a tensor’s values in CPU memory, GPU memory or another device-visible memory tier.
- High Bandwidth Memory (HBM): A type of memory used as main GPU memory in devices such as the H100 (Section 3.6). On those devices, all SMs share it, and it is separate from their on-chip storage. GPU models can use other memory types.
- pinned host memory: Host memory held in place for device transfers. Starting from a pinned buffer can avoid a pageable-to-pinned staging copy. Pinning alone does not guarantee overlap.
- autograd:
PyTorch’s automatic differentiation engine, which tracks operations and computes gradients through them. - fragmentation: Free memory split into pieces that cannot satisfy a required allocation even when their total size appears sufficient.
- Time To First Token (TTFT): Elapsed time from request arrival to its first emitted token.
- Inter-Token Latency (ITL): The elapsed gap between two consecutive emitted tokens of a request.
- Time Per Output Token (TPOT): For a request with n > 1 output tokens, the mean of its n - 1 inter-token gaps. It excludes first-token latency.
- End-to-end latency: Elapsed time from a request’s arrival to its completion.
- Throughput: Completed requests, tokens or other specified work per second for the workload being measured.
- Goodput: Completed work per second that meets the specified service targets, such as latency and quality limits.
- percentile: A value determined by a percentage position in ordered measurements. p50 and p95 use the 50% and 95% positions; selection or interpolation depends on the stated method. Request measures.
- Adam: An optimizer that keeps running averages of each gradient and of its square to guide parameter updates. Its state adds memory beyond parameters and gradients.
- KV block tables: Per-request mappings from logical blocks of token positions to the physical GPU-memory blocks holding their keys and values. See Section 12.5.
- prefix-cache: Stored key-value state reused across requests with matching prefix tokens, model weights, adapter, tokenizer output, position handling, and cache format. See prefix reuse.
- Prefix-cache entries: Bookkeeping records that identify retained key-value state a compatible later request can reuse. Reuse requires matching prefix tokens, model weights, adapter, tokenizer output, position handling, and cache format. See prefix reuse.
- KV head: A head that supplies one key vector and one value vector per token position in a layer. Several query heads can share that pair.
- Latency: Elapsed time for one piece of work between stated start and completion events.
- Bandwidth: The rate at which a memory or communication path moves bytes. Achieved bandwidth is measured bytes divided by measured time.
- Utilization: The fraction of a specified resource’s capacity or time that is in use. Its meaning depends on the resource and measurement definition.
- Saturation: A resource operating at its limit, so extra demand waits rather than being served faster.
- Critical path: The longest chain of dependent work in a step, for fixed operation times and dependencies that include required serialization and resource waits. A graph of data dependencies alone gives a lower bound when limited resources add waiting. Shortening it helps until another required chain becomes limiting. If longest chains tie, shortening only one may leave completion time unchanged. Overlapping work is hidden only when it neither extends the required chain nor slows it through competition for resources.
- Steady state: Repeated execution after one-time work such as compilation, allocation and cache filling has finished.
- Training forward: Computing model outputs from parameters and inputs, producing intermediate activations that training may retain for backward propagation.
- Training backward: Using parameters and saved or recomputed activations to calculate gradients of the loss.
- Optimizer step: Reading gradients and any optimizer state, then updating parameters and that state.
- Distributed synchronization: Exchanging required state or partial results between ranks so dependent distributed work can proceed.
- collective operation: A communication step in which every participating rank takes part, such as all-reduce. Communication buffers.
- floating-point operations (FLOPs): Arithmetic operations counted for a specified computation. Under the counting convention used here, a multiplication plus an addition counts as two FLOPs, even when performed by one fused instruction.
- fusion: Combining compatible operations into one GPU kernel to reduce launches or intermediate memory traffic. The compiler and resource limits determine where intermediates live. Kernel fusion.
- bottleneck class: A category describing the resource or wait that determines the duration of one measured region under its current conditions.
- optimization lever: The kind of change a technique makes, such as reducing work, moving fewer bytes or reducing launch overhead.
- Amdahl’s law: If a non-overlapped fraction f of the original time is sped up by k and the remaining work and scheduling stay unchanged, total speedup is 1 / ((1 - f) + f / k).
- Compute-bound: Having a measured region’s duration limited by arithmetic throughput for the selected instructions and kernel path.
- Memory-bound: Having a measured region’s duration limited by bytes moved through the relevant memory path at its bandwidth limit.
- Latency-bound: Having too little parallel work to hide memory or instruction delays, while neither arithmetic throughput nor bandwidth approaches its limit.
- occupancy: The ratio of resident active warps on an SM to its maximum supported resident warps. Higher occupancy can hide latency but does not guarantee higher throughput. See Section 3.1.
- Launch-bound: Having a measured region’s duration limited by CPU dispatch and kernel launch overhead, with gaps between GPU kernels.
- Input-bound: Having execution wait for data loading, tokenization or other host input preparation.
- Capacity-bound: Being limited by memory size because the work does not fit, or fits only with a smaller batch or concurrency.
- Communication-bound: Having a measured region’s duration limited by transfers or synchronization between GPUs or nodes.
- Scheduler-bound: Having request latency dominated by waiting for admission or batching rather than GPU execution.
GPU Hardware and Memory
Teaching sources: Chapter 3 and Section 6.2.
- VRAM: GPU device-memory capacity available for tensors, model weights, activations, workspaces, and caches. On the data-center GPUs described in GPU memory capacity, it is HBM.
- Streaming Multiprocessors (SMs): GPU processing units containing local scheduling, arithmetic, register and shared-memory resources. L2 cache and HBM serve the whole device.
- L2 cache: An on-chip cache shared by all SMs on one GPU, retaining some data to reduce slower memory accesses.
- warp: A scheduling group of 32 threads within a CUDA block on NVIDIA GPUs.
- warp scheduling: An SM selects ready resident warps to issue instructions while other warps wait for memory, dependencies or synchronization. GPU execution hardware.
- NVLink: NVIDIA’s GPU-to-GPU interconnect for peer traffic within a supported group of GPUs. The group can span more than one server.
- threads: Logical instruction sequences, each working on its own data. A GPU kernel exposes many of them for parallel execution.
- kernel: A GPU function launched to execute over many logical threads.
- blocks: Groups of CUDA threads that can share on-chip storage and synchronize. In the basic CUDA execution model, each block runs on one SM.
- Register pressure: A kernel’s demand on an SM’s finite register storage. High demand can reduce the number of resident warps or cause spilling.
- Shared memory: On-chip storage shared by threads in a CUDA block and explicitly managed by the kernel. Its per-block use can limit resident blocks. GPU storage hierarchy.
- PCIe: The expansion bus connecting GPUs, network cards and other devices to the host, and carrying some peer-transfer paths.
- cache line: A fixed-size aligned region of memory retained in a cache together with information identifying its address.
- set: The group of cache locations to which the cache’s address-mapping rule assigns a line.
- way: One candidate location for a cache line within a cache set.
- Associativity: The number of ways in each cache set, limiting how many lines assigned to that set can be retained together.
- L1 or unified data cache: A per-SM cache path for local load and store traffic. It is not a general store for state shared across SMs.
- Cache sector: A unit of cache traffic visible in profiler counters. The Nsight Compute views discussed in Section 3.2 report 32-byte sectors and four sectors per 128-byte L1 or L2 line.
- hit rates: Fractions of measured memory accesses that find their data in the cache being measured.
- Tensor Cores: Specialized matrix multiply-accumulate units inside SMs. Their supported formats, instruction shapes and throughput depend on the GPU and kernel path.
- Multiply-accumulate: Multiplying operand values and adding their products into a running result. Matrix operands of shapes m by k and k by n update an m by n result.
- Network Interface Cards (NICs): Hardware devices connecting a server to the cluster network.
- process: A running operating-system program instance.
- rank: A process’s integer identity within a distributed group, distinct from the GPU device it uses.
- GPU device: The selected GPU to which a process submits device work.
- CPU affinity: A restriction on which CPU cores may execute a process. Host-memory placement is a separate policy.
- data-loading workers: Optional helper processes that prepare input batches. They do not automatically receive distributed ranks or GPUs.
- NVSwitch: A switch chip connecting multiple GPUs through NVLink.
- Pageable memory: Ordinary host memory whose physical pages the operating system may move or page out. A device transfer may first stage its payload in pinned memory.
- Direct Memory Access (DMA): Device transfer of a memory payload without the CPU copying each byte. Software still submits work and tracks when the transfer finishes before dependent work uses the data.
- Remote DMA (RDMA): Network-hardware transfer to or from memory registered with the network card for direct access on another machine. Software prepares accessible buffers and coordinates communication.
- GPUDirect RDMA: A supported direct path between a NIC and GPU memory that avoids an intermediate host payload buffer. It requires compatible devices, drivers, registration and topology.
- InfiniBand: A high-performance cluster network fabric commonly used for communication between GPU servers.
- RoCE: RDMA over Converged Ethernet, whose performance depends on the Ethernet fabric’s configuration.
- Ethernet: A general-purpose network fabric. Untuned Ethernet paths can perform poorly for frequent GPU collectives.
- peer transfer: A direct copy between GPUs over a supported NVLink, NVSwitch or PCIe path.
- Arithmetic intensity: FLOPs performed per byte moved through a specified memory level, measured in FLOP/byte for the same execution region. Work, bytes, and time.
- Ridge point: The arithmetic intensity at which the estimated compute time and memory-transfer time are equal: peak compute rate divided by the selected memory bandwidth. Below it, the bandwidth ceiling is lower; above it, the compute ceiling is lower. Other bottlenecks can keep execution below either ceiling. Work, bytes, and time.
Software to CUDA
Teaching source: Chapter 4.
- Compute Unified Device Architecture (CUDA): NVIDIA’s GPU programming model and platform, providing kernel execution, memory operations and host interfaces.
- CUDA runtime: The host API for device selection, memory allocation, kernel launches, streams, events, copies and synchronization. CUDA runtime and vendor libraries.
- CUDA driver: The lower-level interface that loads GPU code and manages device interaction through the installed operating-system GPU driver.
- Device context: Process-visible execution state for a GPU, including loaded code modules, allocations, streams and runtime resources.
cuBLAS: An NVIDIA matrix and linear-algebra library used for matrix multiplication. Supported dimensions, dtype, layout, and alignment determine access to Tensor Core implementations. Matrix operations in vendor libraries.cuBLASLt: An NVIDIA matrix and linear-algebra library used for matrix multiplication. Supported dimensions, dtype, layout, and alignment determine access to Tensor Core implementations. Matrix operations in vendor libraries.cuDNN: NVIDIA’s library of supported neural-network operators, including convolution primitives.- Vendor library kernel: A compiled GPU implementation selected through a library API, serving as a measured baseline for a custom kernel.
- Host: The CPU side of the program, which prepares data, allocates tensors, submits device work and may wait for completion.
- Grid: All CUDA blocks created by one kernel launch.
- Warp divergence: Different lanes (the threads of one warp) following different control paths, which the hardware executes under masks rather than as fully shared work.
- Predication: Executing conditional work under lane masks instead of a full branch. Masked paths may still consume instructions.
- Latency hiding: Scheduling ready warps while others wait for memory, dependencies or synchronization, keeping execution units busy when enough work is available.
- Mapped host memory: A host allocation that a GPU kernel can access through the system interconnect.
- Managed memory: Memory whose pages can migrate or be prefetched between processors. Hints such as
cudaMemAdviseandcudaMemPrefetchAsyncguide where pages live. L1 and L2 caching stays under hardware control. - stream: An ordered queue of submitted CUDA operations. Copies and kernels in the same stream execute in submission order.
- Device tensor: A tensor with storage in GPU memory. Its Python object is host metadata, while its data pointer refers to device storage.
- caching allocator:
PyTorch’s CUDA allocator that reserves memory blocks and reuses them for later tensor allocations, reducing repeated allocation requests to the CUDA driver. - L2 persisting access window: A CUDA stream or graph setting requesting preferred L2 retention for a global-memory range, subject to hardware limits and the L2 capacity reserved for persisting accesses.
- Memory coalescing: Combining nearby memory requests from warp lanes into fewer transactions, reducing traffic spent on unused data.
- General Matrix-Matrix Multiplication (GEMM): An M by K matrix multiplied by a K by N matrix produces an M by N result.
- General Matrix-Vector Multiplication (GEMV): Multiplying a matrix by a vector to produce an output vector.
- shuffle: Transferring a register value between participating lanes of a warp.
- epilogue: Processing applied to a matrix operation’s result, such as adding a bias. Matrix operations.
Triton and Custom Kernels
Teaching source: Chapter 5.
Triton: A Python-based language and compiler for GPU kernels, where each logical program instance handles a tile of data. Kernel diagnosis; Triton program structure.- LLVM: Reusable compiler tools for code optimization and generation, used in Triton’s lowering path from intermediate representations toward PTX. Kernel implementation choices.
- CUDA C++ kernel or extension: Custom GPU code written with CUDA C++ controls for threads, blocks, shared memory and synchronization. GPU kernel path selection.
- Program instance: One logical tile of Triton work selected by the launch grid.
- Block pointers and strides: Descriptions of tiled memory regions and their physical indexing, including how logical positions map to storage and the tile’s valid tensor limits.
- Masks: Conditions disabling selected loads or stores, such as accesses outside a partial tile’s valid region.
- Reductions: Operations combining multiple tile values into fewer results, such as a sum.
- Just-in-time (JIT) compilation: Compiling a function when first needed for a device, dtype, and compile-time configuration. First-call compilation belongs outside steady-state timing. Triton program structure.
- Specialization: Generating different compiled code for different compile-time values, shapes, strides or selected configurations.
- Autotuning: Benchmarking candidate kernel configurations and selecting one for the target workload. It is optional. Kernel implementation choices.
- Spilling: Placing values intended for registers in CUDA local memory when register demand is too high, potentially adding device-memory traffic.
- Program order: The mapping from program IDs to visited tiles. A different order can change cache reuse without changing the intended mathematical result.
- Cache modifier: A load or store option requesting cache behavior supported by the operation and backend.
- Eviction policy: A preference guiding cache replacement, such as evict-first or evict-last. It does not guarantee data remains resident.
- Grouped tile order: Visiting related output tiles close together so they can reuse input data through L2 cache.
- Autotuned configuration: A measured choice of tile dimensions, cooperating warps and software-pipeline stages for the target tensor dimensions.
- FlashAttention: An attention algorithm and kernel family that streams tiles through on-chip resources and keeps online softmax statistics instead of storing the full attention matrix in HBM. It preserves the mathematical attention operation; floating-point rounding can differ.
- FlashAttention-1: The tiled online-softmax attention method that avoids full score and probability matrices in HBM, with additional activation storage linear in sequence length.
- FlashAttention-2: A FlashAttention version that improves work partitioning and reduces non-matrix work and shared-memory communication, with benefits depending on compatible shapes.
- FlashAttention-3: A FlashAttention version using Hopper features to overlap movement and computation. Its FP8 path requires suitable hardware and kernels.
xFormersattention kernels: Optimized attention implementations whose eligibility depends on dtype, mask, head dimension, sequence dimensions and backend.- Online softmax: Computing softmax over successive blocks while retaining and updating a running maximum and sum, rather than storing the full row’s intermediate probabilities.
Performance Models and Profiling
Teaching sources: Chapter 6 and Chapter 7.
- bytes moved: Traffic through the memory level being modeled for the measured region. The count must account for relevant reuse, intermediate buffers and reads or writes.
- Model FLOPs utilization (MFU): Useful model arithmetic per second divided by the matching hardware peak FLOP/s. It uses tokens per second times required FLOPs per token and excludes checkpointing recomputation from that work count.
- peak compute: The arithmetic throughput ceiling for the selected precision, hardware and kernel path, rather than an achieved rate.
- peak bandwidth: The bandwidth ceiling of the memory or link path being modeled, distinct from its measured transfer rate.
- Python profilers: Tools that collect Python execution evidence, through recorded function calls or samples of a running process.
- Event or tracing profiler: A tool recording timed operations and their relationships, such as CPU work, kernels, copies and communication on a timeline.
- NVTX: Application-inserted trace annotations that identify time ranges in a profiling timeline.
- Kernel profiler: A tool examining one GPU kernel’s resource use, transactions, instructions and stalls in detail.
- Trace viewer: A tool for inspecting exported execution timelines and their events interactively.
- Perfetto: A browser-based trace viewer used to inspect exported timelines.
- memory snapshot: A record of
PyTorch’s CUDA allocator state at one point, useful for allocation-pattern investigation. It does not cover every external GPU allocation or provide a kernel-timing trace.
Compilation and Runtime
Teaching source: Chapter 8.
- CUDA Graph: A recorded sequence of GPU operations that can be replayed with less host submission work, subject to capture and address requirements.
AOTAutograd: ThePyTorchcompilation component that prepares forward and backward graphs when gradients are needed.TorchDynamo: ThePyTorchcompiler component that captures compatible Python-level tensor operations into graph regions.TorchInductor: ThePyTorchcompiler backend that lowers captured graphs into executable code, including generated kernels and library calls. Compiler components.- eager execution: Executing
PyTorchoperations as they are encountered through the dispatcher, the layer that selects and calls an implementation. These operations are not combined into a compiler-captured region. - graph capture: Recording a sequence of operations for optimization or replay. Compiler graph capture, as in
torch.compile, records compatible tensor operations so they can be optimized. CUDA Graph capture records supported GPU operations so they can be replayed with less host work. The two records are different objects. - Graph optimization: Transforming a captured graph, potentially combining compatible operations, removing intermediates or choosing generated kernels.
- Lowering: Translating an optimized graph into executable backend code or calls.
- Guard: A recorded condition under which a compiled region can be reused. A failed guard may select another path or trigger compilation of another variant.
- graph break: A point where
TorchDynamo,PyTorch’s compiler front end, cannot continue recording one region. Default partial-graph compilation can leave separate compiled and eager pieces. Full-graph mode rejects breaks. - Scalar extraction: Moving a tensor value into Python, for example with
.item(). It commonly causes synchronization or a graph break, depending on the release and configuration. - Data-dependent Python branches: Python branch or loop decisions based on tensor values, which usually cannot be captured as one static graph.
- Unsupported operations: Operations or effects a compiler component cannot handle. Capture failure can cause a graph break. A backend failure after capture can instead raise an error.
- Dynamic shape churn: Frequent shape changes causing failed guards or repeated compilation, distinct from a graph break within a function.
- Host I/O in hot paths: Printing, logging or other host input/output within repeatedly executed GPU work, potentially forcing synchronization or graph breaks.
- Generated kernel: GPU code emitted by a compiler for a particular region, dtype, layout, and range of input shapes. Specialization does not guarantee optimal performance.
- Address stability: Keeping the pointer arguments recorded in a CUDA Graph capture unchanged during replay, while copying new values into those buffers.
- Allocation failure: An unsupported allocation during CUDA Graph capture that causes a capture error. Supported
PyTorchtensor allocations can use a graph-private pool that preserves addresses for replay. Preallocated input buffers and completed initialization keep the other parts of the steady path stable. - Root-mean-square (RMS) normalization: Dividing each feature by the square root of its row’s mean squared value, with a small epsilon added inside that root. The unweighted calculation in Section 8.8.1 omits learned scaling, unlike the model’s RMSNorm.
Training and Precision
Teaching source: Chapter 9.
- Step time: Elapsed time from one training step’s start to the next comparable step’s start after warmup, including data delivery, GPU work, optimizer work, and any communication that remains exposed.
- Global batch: All examples consumed before one optimizer update, across accumulation steps and data-parallel replicas.
- Exposed tail: Work remaining after useful overlapping work has finished, so that it adds to completion time.
- Saved activations: Forward-pass intermediate values retained until backward propagation uses them.
- Activation checkpointing: Retaining fewer forward activations and recomputing selected regions during backward propagation, trading extra computation for lower activation memory.
- Low-Rank Adaptation (LoRA): Fine-tuning small adapter matrices while keeping base parameters fixed, reducing trainable parameter, gradient and optimizer-state storage. It does not remove the need for activations.
- Quantized Low-Rank Adaptation (QLoRA): LoRA with a quantized frozen base model, requiring a compatible path to dequantize its stored values for computation.
- Data pipeline: The storage and CPU preparation path that reads examples, transforms or tokenizes them, assembles batches and supplies them for GPU transfer.
- Micro-batch: The examples processed in one forward and backward pass before optional gradient accumulation.
- Gradient accumulation: Adding gradients from several micro-batches before performing one optimizer update.
- Fused optimizer: An optimizer implementation combining update operations to reduce launches or intermediate memory traffic.
- sign bit: The bit selecting whether a floating-point value is positive or negative.
- exponent: The part of a floating-point representation determining its power-of-two scale.
- significand: The significant binary digits of a floating-point value, determining precision at a given exponent.
- FP32: A 32-bit floating-point format used for model values, numerically sensitive work and reference calculations.
- TF32: NVIDIA’s TensorFloat-32 Tensor Core mode for supported FP32 matrix operations. It keeps the FP32 exponent range with fewer significand bits.
- FP16: A 16-bit floating-point format with narrower range than FP32. Training may need loss scaling or other mixed-precision measures.
- BF16: A 16-bit floating-point format with FP32-like exponent range and fewer significand bits, reducing storage compared with FP32.
- FP8: Families of 8-bit floating-point formats used with suitable scaling, accumulation and kernel support on compatible accelerators.
- FP4 and block-scaled FP4: Four-bit floating-point formats, often paired with scales for small blocks of values. Scale storage, packing and compatible kernels affect their cost.
- INT8: An 8-bit integer representation used for quantized values, requiring a mapping to approximate real values and a compatible computation path.
- INT4: A 4-bit integer representation often used for compressed inference weights, with quality and speed depending on scales, grouping and kernels.
- Ternary weights: Weights restricted to three values, such as negative, zero and positive, in a model representation and kernel path designed for them.
- mixed-precision training: Using reduced precision for selected operations or storage while retaining higher precision where numerical range or accumulation requires it.
- FP32 master weights: Persistent FP32 parameters used for optimizer updates in one common FP16 mixed-precision arrangement, with reduced-precision copies for computation. BF16 and other arrangements can use different state layouts.
- Gradient underflow: A gradient becoming too small for its numeric format and rounding to zero. FP16 is especially exposed because its representable range is narrower than FP32 or BF16.
- Loss scaling: Multiplying the loss before backward propagation so small FP16 gradients are less likely to underflow, then dividing gradients by the same scale before updating parameters. Dynamic scaling can lower the scale after overflow.
- Higher-precision accumulation: Accumulating products, reductions or updates in a wider format than their inputs when supported by the hardware and kernel.
- amax: A tensor’s maximum absolute element value, used directly or through its history by some FP8 scaling methods.
Inference Workloads
Teaching sources: Chapter 10 and Chapter 11.
- QPS: Queries per second, where one query is one serving request. Interpret this rate together with the service-level objective and quality target. Serving metrics.
SentencePiece: A tokenizer implementation whose model file specifies text segmentation and the vocabulary used to produce token IDs.Hugging Face Tokenizers: A tokenizer implementation supporting operations such as batching, truncation, padding and special-token handling.safetensors: A tensor serialization format designed for safe metadata inspection and efficient checkpoint loading.- Generation utilities: Helpers controlling generation loops, including cache use, sampling, beam search and stopping conditions.
- Service-Level Objective (SLO): A measurable service target, such as a specified fraction of requests receiving their first token within a time limit.
- Service-Level Agreement (SLA): A customer contract promising specified service targets.
- Tokens per iteration: The total tokens actively processed in one forward pass, including admitted prompt tokens and tokens for decoding sequences.
- Context length: The number of token positions a request can attend to.
- Active cache blocks: Physical KV-cache blocks currently owned by admitted requests, consuming memory that can limit concurrency.
- Model compression: Reducing a model’s storage or computation through methods such as quantization, pruning or training a smaller distilled model, each with its own quality and runtime costs.
- student: A model trained to reproduce a teacher model’s output distributions during knowledge distillation.
- teacher: A model whose output distributions supply training targets for a student during knowledge distillation.
- Weight quantization: Representing model parameters in fewer bits to reduce their storage and repeated reads.
- Activation quantization: Representing or computing intermediate tensors in lower precision, with accuracy depending on the values encountered in the workload.
- KV-cache quantization: Storing attention keys and values in fewer bits to reduce the cache memory required by active requests.
- Dequantization: Converting compact stored values to a wider numeric representation for computation, adding work that can reduce the speed benefit.
- Scale metadata: Extra scaling values mapping low-precision stored codes to approximate real values.
- GPTQ: A post-training weight-quantization method that uses calibration inputs and approximate second-order information to limit changes in a layer’s output.
- AWQ: Activation-aware Weight Quantization, a method protecting weight channels important for expected activations during low-bit weight quantization.
bitsandbytes: A library providing supported low-precision optimizer and weight paths for training and inference.torchao:PyTorchtooling integrating supported quantized and sparse representations with model execution.- Calibration set: Workload-representative prompts, activations or layer inputs used to estimate numeric ranges for quantization.
- Clipping range: A selected numeric interval limiting outliers before quantization, potentially harming important values outside it.
- Per-channel or per-group quantization: Quantization using separate scales for individual channels or smaller groups of values, adding metadata and kernel complexity.
- Post-Training Quantization (PTQ): Converting an already-trained model to a lower-precision deployment representation. Calibration and parameter selection depend on whether the path is static, dynamic or weight-only.
- Quantization-Aware Training (QAT): Training with simulated quantization and dequantization in the forward pass so updates account for an approximation of deployment error.
- Straight-Through Estimator (STE): A backward-pass derivative approximation allowing gradient-based training through a rounding-like fake-quantization operation.
- Perplexity: The exponential of average per-token cross-entropy loss on held-out text. Lower values mean better prediction under that evaluation; task quality still needs separate checks.
- Pruning: Setting selected weights to zero or removing structures. Runtime savings require a representation and kernel that can skip the removed work.
- Unstructured sparsity: Zeros at individually selected weight positions, requiring suitable compressed storage and sparse kernels for scattered zeros to save computation.
- Structured sparsity: Removing complete rows, columns, heads, or blocks. Whole-channel removal can yield smaller dense tensors. Isolated blocks need compatible sparse kernels.
- Semi-structured 2:4 sparsity: A weight pattern with at most two nonzero values in each group of four consecutive weights, used with a supported compressed format and sparse computation path, such as Sparse Tensor Cores on NVIDIA Ampere and later GPUs.
- Knowledge distillation: Training a smaller student model to reproduce a larger teacher model’s output distributions.
Decode and KV Cache
Teaching source: Chapter 12.
- Naive decode: Reprocessing the whole prefix from scratch for each new token instead of retaining its attention keys and values.
- Cached decode: Processing the newest token while attending to keys and values retained from previous positions, trading recomputation for persistent memory and reads.
- Optimized decode: Cached decode with serving and kernel changes that reduce overhead or improve batching, cache reads, and memory management.
- GEMV-like decode: Small-batch projection work multiplying weight matrices by one or a few token vectors, with relatively little weight reuse.
- GEMM-like prefill: Projection work processing many prompt positions together, reusing weights across a matrix of activations.
- Key-value cache layout: The physical ordering and indexing of cached layers, sequences, tokens, heads and head components in GPU memory.
- PagedAttention: Mapping logical token blocks to reusable physical KV-cache blocks, allocating blocks on demand rather than reserving one maximum-length contiguous region per request. Block tables and partially filled blocks still add cost.
- preemption: Stopping a running request and releasing its KV blocks when the engine needs capacity. Resuming requires recovering its state, such as by recomputation or swapping.
- vLLM: An open-source language-model serving engine. Version 0.6.4 defaults to recovering a preempted single-sequence group’s KV state by recomputation; multi-sequence groups default to CPU swap, and an explicit mode setting can also select swap. Section 12.5 explains the conditions and costs.
- swapping: Moving evicted KV blocks to CPU memory and later restoring them, trading transfer traffic for recomputation.
- state-space model (SSM): A model using a structured recurrent state update and output mapping. Selective forms can make parts of those mappings depend on the current input.
- scan: Computing states for successive input prefixes under a recurrence. Suitable affine updates can be composed for a parallel scan while preserving their dependencies.
- Sliding-window attention: Attention limited to the most recent W positions in each layer. Earlier information can still propagate through stacked layers. The window belongs to the model’s design.
- Attention sinks: Tokens that receive unusually large attention weights even when their content is not important to the current prediction. StreamingLLM retains the KV entries of initial sink tokens plus a recent window; evicted KV entries are unavailable to later attention.
- Latent KV compression: Storing compressed low-rank state per token and reconstructing what attention needs, requiring a model trained or adapted for that architecture.
Serving Engines
Teaching sources: Chapter 13, Chapter 14 and Chapter 15.
Inference engine: A broad term for inference software that can include model runtimes, compilers, hardware-specific executors and serving-engine components. Serving engines.
control plane: In the serving arrangement described here, the CPU-side request-processing and scheduling work: formatting, tokenization, admission, batch selection, KV bookkeeping and output streaming. CPU control plane.
admission control: Deciding whether a request can enter active execution under memory, context-length, batch and service-target constraints.
CPU control-plane delay: CPU scheduling, tokenization, or block bookkeeping delays the next GPU work, leaving idle gaps between kernels.
Static batching: Running a fixed group of requests together until that batch completes.
Dynamic batching: Briefly collecting requests into a fuller launch batch, then running the launched work mostly as a fixed unit.
continuous batching: Updating the active request set at each decode iteration, allowing completed requests to leave and new ones to enter.
Multi-step scheduling: Planning several decode iterations ahead to reduce CPU scheduler round trips for the GPU worker.
Little’s law: In a stable system, L = lambda W: L is the average number of requests present, lambda is the average arrival rate, and W is the average time spent in the system. The relevant long-run averages must exist and be finite. All averages describe the same request population and matching entry/exit flow over the same long interval. Exit includes rejection or cancellation when those requests are included.
Chunked prefill: Splitting a long prompt’s processing into smaller scheduled portions so other requests can progress between them, with extra scheduling and launch overhead.
Disaggregation: Separating prefill and decode onto different worker pools, transferring the initial KV cache and compatible request state from prefill to decode. Transfer, coordination, and duplicated-weight costs affect whether the split helps. Separate workers.
Cache hit: Finding stored, valid KV state for a new request’s exact token prefix with matching model weights, adapter, tokenizer output, position handling, and cache format.
Radix tree: An index sharing common token-prefix paths, used to find the longest reusable prefix.
RadixAttention: SGLang’s prefix-reuse method that organizes shared token prefixes in a radix tree and reuses execution-compatible KV state on the matching path. Radix-style reuse.
Invalidation: Making stored KV state ineligible for reuse when model weights, adapter, tokenizer output, position handling, cache format, or prefix tokens change its meaning. Sampling settings matter only when they affect those inputs.
Block hashing: Computing a cache-block key from a complete block’s tokens and relevant model, adapter or cache identity information.
Parent-prefix hash: A hash of preceding complete blocks included in a block key, distinguishing identical token blocks reached through different prefixes.
Complete-block reuse: Reusing only filled cache blocks under a block-based policy. A partially filled final block may remain ineligible.
Per-block LRU: Least-recently-used eviction applied to individual cache blocks under memory pressure, which can evict shared prefixes that future requests would reuse. Prefix caching and radix-style reuse.
Cache-blind routing: Routing requests without considering where reusable prefix state is already stored.
Cache-aware routing: Sending requests toward workers holding reusable prefix blocks, while also managing load and routing state.
external KV layer: A cache tier outside one GPU worker’s HBM that preserves or reuses prefill state across requests, replicas, or time windows when the stored state remains compatible. It can use CPU memory, disk, or remote storage. External cache tiers.
speculative decoding: Proposing several tokens with a cheaper draft method and verifying them with the target model. Preserving the target output distribution requires the acceptance rule and the residual resampling rule.
Residual distribution: The normalized positive part of target probability minus draft probability, used to sample the correction after a rejected proposal under the standard speculative rule. Draft and verify.
cache rollback: Removing KV entries computed for a rejected speculative suffix before continuation from the retained prefix. Speculative cost.
lenience: A research parameter that permits more draft tokens to be accepted than the exact speculative rule allows. It may improve speed, but the output no longer exactly follows the target model’s distribution.
Parallel verification: Evaluating a drafted suffix with one larger target-model forward pass rather than a separate target pass per proposed token.
N-gram or context-copy draft: Proposing tokens from repeated context instead of a separate neural draft model, useful when the input contains suitable repetition.
Constrained decoding: Restricting token selection to outputs allowed by a grammar, schema, regular expression or choice set, maintaining constraint state and valid-token masks per request.
state: In a finite-state constraint machine, the record of progress through the permitted character sequence, which determines possible continuations. This is distinct from a model’s hidden state or KV-cache state. Constrained decoding.
transition: In a finite-state constraint machine, an edge that consumes a permitted character and moves to its destination state. The age-field walk.
Queue Time: Time from a request’s arrival at the scheduler until admission.
open-loop load: A request-arrival policy whose timing is independent of replies, allowing a slow service to accumulate a queue.
closed-loop load: A request-arrival policy where each client waits for its reply before submitting again, so slower replies lower the offered rate.
SGLang: A serving and programming stack for structured language-model programs, including branching and radix-style prefix reuse.
TensorRT-LLM: NVIDIA’s inference stack for supported hardware and model families. Its preparation and execution workflow depends on the installed release.
Triton Inference Server: NVIDIA’s serving layer for endpoints, model repositories, versions and metrics. Configured backends supply model execution; it is distinct from the Triton kernel language.
NVIDIA NIM: Packaged NVIDIA inference microservices with defined software, version and hardware requirements.
llama.cpp: Inference software with supported CPU and GPU backends, used for local and hardware-constrained deployment. Fleet throughput still needs a separately tuned serving engine and scheduler. Local inference software.GGUF: A model file format storing tensors and metadata for compatible inference software, including
llama.cpp. It can store quantized tensors but does not supply serving-engine scheduling. Model formats.Ollama: A local model-management layer commonly using llama.cpp-style execution.
Performance Experiments
Teaching source: Chapter 7.
- Shape drift: An accidental change in tensor dimensions between compared runs, potentially changing compilation or selected kernels.
- benchmark harness: Wrapper code that prepares inputs, runs warmup, measures the intended region, checks correctness and reports results. Controlled benchmark workflow.
Communication and Distributed Systems
Teaching sources: Chapter 16 and Chapter 17.
- process group: A set of ranks issuing compatible communication operations through one selected backend.
- Rank coupling: A dependency that makes distributed processes wait for each other at a shared operation.
- Critical-path communication: Data transfer that must finish before the next dependent computation can start, so its exposed time affects completion.
- Payload size: The number of data bytes transferred by the communication being measured.
- Startup latency: Fixed communication cost before useful payload bytes move.
- Topology effect: The effect of the selected GPU, switch and network route on communication time and available bandwidth.
- Bisection bandwidth: The minimum aggregate bandwidth across any division of network endpoints into two equal-sized groups. Values may count capacity in one direction or both.
- oversubscription: Upstream links having less aggregate capacity than the endpoints connected below them.
- Scale-up: Communication within a group of GPUs joined by fast direct links such as NVLink. The group can extend beyond one server.
- rank skew: A difference in when participating ranks become ready for the same operation, potentially adding waiting before useful transfer.
- Point-to-point communication: A transfer from one rank to another.
- Collective communication: A group operation in which participating ranks exchange or combine tensors to produce a defined result.
- Ordering requirement: The requirement that ranks issue compatible collectives in compatible order. A mismatch can prevent correct distributed execution.
- Broadcast: Copying one source rank’s tensor to every other participating rank.
- Reduce: Combining participating ranks’ tensors into one result on a destination rank.
- all-reduce: Combining participating ranks’ tensors and delivering the reduced result to every participant. The reduction operation determines whether the result is a sum or another supported combination.
- All-gather: Collecting each rank’s shard so every participant receives the complete collection.
- Reduce-scatter: Combining tensors across ranks, then delivering one shard of the reduced result to each participant.
- All-to-all: Each rank sending a distinct slice to each other rank and receiving the slices addressed to it.
- Routed tokens: Token representations sent to the ranks holding their selected experts. See expert and MoE for the routed feed-forward computation.
- Top-k multiplier: In expert routing, the number of experts selected for each token, affecting how many token representations are sent. See expert and MoE.
- Ring collective: A collective schedule passing chunks around an ordered ring of ranks.
- Tree collective: A collective schedule aggregating or distributing data through a branching structure.
- Hierarchical collective: A collective schedule using separate exchanges within fast domains and across slower domains to match their different costs.
- Topology awareness: Choosing communication schedules using the actual device and network layout rather than assuming every rank pair has the same cost.
- Backend: The implementation selected when a program creates a process group to execute communication, such as
NCCL(NVIDIA Collective Communications Library) orGlooon supported paths. - transport: The underlying communication path, such as GPU peer links, shared memory, sockets or an RDMA fabric.
NCCLlogs: Diagnostic records of the NVIDIA Collective Communications Library’s decisions, enabled by settings such asNCCL_DEBUG=INFO. They do not alone establish achieved performance.- Scale-out domain: Multiple nodes communicating through network cards and switches, with costs determined by the cluster path.
- GPU/NIC affinity: The physical locality of a GPU and the NIC used for its network traffic, affecting intervening links and socket crossings.
- Placement policy: The rule mapping ranks, workers, replicas or serving groups to physical GPUs and nodes.
- Per-token communication: Transfers required during token-generation steps, such as tensor-parallel exchanges, expert routing or KV movement.
- Per-layer communication: Exchanges around a layer’s computation, such as partial results or gathered parameters. Frequency depends on the method and retention policy.
- Per-step communication: Exchanges associated with a training update, such as gradient synchronization. Accumulation and overlap policies determine when traffic occurs and how much remains exposed.
- Per-request communication: Exchanges associated with a request, such as routing or a prefill-to-decode handoff.
- Background communication: Transfers outside immediate dependent execution, such as checkpointing, model loading or cache movement, when scheduled away from the hot path.
Distributed Training and Serving
Teaching sources: Chapter 18, Chapter 19, Chapter 20 and Chapter 21.
- Replica: A complete copy of model parameters. In the one-process-per-GPU DDP setup, each rank holds one copy and gradient synchronization keeps their updates aligned. In serving, one complete copy can be sharded across a worker group, with several such groups acting as replicas.
- Distributed Data Parallel (DDP): Replicated training in which each rank computes local gradients and synchronizes them before equivalent optimizer updates. The application partitions input data. DDP.
- Fully Sharded Data Parallel (FSDP): The
PyTorchAPI family for sharded training state. UnderFULL_SHARD, parameters, gradients and optimizer state are partitioned outside required full-parameter use windows. Sharding. - Tensor parallelism: Dividing operations within a model layer across ranks and exchanging required partial results. Tensor-parallel execution.
- Pipeline parallelism: Placing layer ranges on separate stages and passing activations between them. Pipeline execution.
- Model parallelism: Splitting one model execution across ranks, including layer-internal or stage partitions. Tensor-parallel execution.
- Stochastic gradient descent (SGD): An optimizer that subtracts learning rate times gradient from each parameter. Replicated update trace.
- momentum buffer: Persistent optimizer state retaining a decaying sum of past gradients for each parameter in the stated SGD-with-momentum rule. Full-shard update.
- Local mini-batch: The examples one rank processes in one forward and backward pass, also called its local micro-batch.
- gradient buckets: Parameter gradients grouped for fewer, larger communication operations rather than one collective per tensor.
- Ready time: The point when backward propagation has produced all gradients in a bucket, allowing its communication to proceed.
- Hidden communication: Communication completed during independent backward work without an exposed tail; contention can still lengthen the step.
- ZeRO-1: Stage 1 of the Zero Redundancy Optimizer (ZeRO): sharding optimizer state across data-parallel ranks while parameters and gradients remain replicated.
- ZeRO-2: Sharding optimizer state and gradients while parameters remain replicated. Communication timing depends on the implementation’s readiness and overlap policy.
- ZeRO-3: Sharding parameters, gradients, and optimizer state across data-parallel ranks, with parameters gathered when needed. Implementations may reshard or retain gathered parameters. Full parameters and communication buffers still add temporary memory.
- Stochastic gradient descent (SGD) with momentum: An optimizer retaining a decaying sum of past gradients and updating parameters against that accumulated direction.
- Offload: Keeping selected model state in host memory and transferring needed values for GPU work. Some implementations also execute optimizer updates on the CPU, trading GPU capacity for transfers and CPU time.
- sigmoid linear unit (SiLU): An elementwise nonlinear function, also called Swish-1, with SiLU(a) = a / (1 + exp(-a)).
- pipeline bubble: Idle intervals on pipeline stages while dependent forward or backward work reaches or leaves them, such as during fill and drain.
- GPipe: Pipeline-parallel training using micro-batches whose gradients accumulate for one batch update. The fill-and-drain schedule described here completes their initial forward work before backward work begins. Pipeline execution.
- column-parallel linear layer: A distributed linear layer split by output channels. Each rank computes its output slice from the full input. Tensor-parallel execution.
- row-parallel linear layer: A distributed linear layer split by input channels. Each rank computes a partial output; the required sums can use all-reduce or reduce-scatter according to the output layout. Tensor-parallel execution.
- KV cache utilization: Occupied KV-cache blocks or bytes under an engine’s stated definition, interpreted against its configured capacity and retention policy. Serving metrics.
- shared-memory bank: One independently served group of shared-memory locations. Different-address requests from one warp to the same bank can need extra transactions. GPU storage.
- bank conflict: Extra shared-memory transactions required when threads of one warp request different addresses in the same bank; the same-address broadcast case is separate. GPU storage.
- Stage balance: How evenly work is divided among pipeline stages. The slowest stage limits pipeline throughput.
- parallelism degree: The number of ranks in a group for one kind of parallel work. With one rank per GPU, multiplying these counts gives the GPU total for independent ways of dividing the work. Parallel layout sizing explains the group conditions.
- MoE: Mixture of Experts, a model structure with many expert submodules but only selected experts active for each token.
- expert: A feed-forward submodule in a mixture-of-experts model, with a router selecting which experts process each token.
- Expert parallelism: Distributing experts across GPUs and routing token representations to the devices holding their selected experts.
- Sequence parallelism: Splitting sequence positions across tensor-parallel ranks for operations otherwise replicated, such as normalization and dropout, with reduce-scatter and all-gather around those regions.
- Context parallelism: Splitting sequence positions across ranks for attention and other layer operations, exchanging key and value blocks or equivalent required state. Hiding those transfers requires enough independent computation per block relative to transfer time.
- capacity factor: A multiplier used to set an expert’s token-buffer capacity. The cited top-1 Switch Transformer uses tokens per batch divided by expert count, times this factor; other routing schemes can count capacity differently.
- Sticky placement: Keeping an active request on the worker group that owns its current KV blocks.
Code and API Reference
Literal API and control names are listed separately from formal glossary terms.
cudaMemcpy: A CUDA runtime operation for copying bytes between host and device or between devices. Transfer roles.@triton.jit: The decorator that marks a Python function for Triton compilation. Triton program structure.tl.program_id: The Triton operation identifying which logical program instance handles the current tile. Triton program structure.tl.arangeand offsets: Vectorized positions created within a Triton tile and used to calculate the elements or memory addresses that tile handles. Triton program structure.tl.loadandtl.store: Triton operations for loading values from GPU memory and storing results back to it. Triton program structure.tl.constexpr: A Triton compile-time parameter annotation whose values can specialize generated code rather than act as ordinary runtime tensor values. Triton program structure.AutoTokenizer: A Hugging FaceTransformersfactory that loads a checkpoint’s tokenizer configuration and returns an object that converts text to token IDs, masks, and batch encodings. Tokenizer objects.AutoModelForCausalLM: A Hugging FaceTransformersfactory for loading autoregressive language-model architectures and checkpoints. Model interfaces.model.eval(): APyTorchcall changing module behavior, such as dropout and batch-normalization statistics, for evaluation or inference. Gradient recording is controlled separately. Inference modes.torch.no_grad(): APyTorchcontext disabling gradient recording for enclosed operations, reducing autograd work and retained backward state. Inference modes.torch.inference_mode(): A stricter inference context that also disables view tracking and version-counter updates for tensors created inside it. Inference modes.max_num_batched_tokens: In vLLM 0.6.4.post1, the limit on total prompt and decode tokens admitted to one scheduling step. Scheduler controls.max_num_seqs: In vLLM 0.6.4.post1, the limit on sequences in one scheduling iteration. Scheduler controls.--gpu-memory-utilization: In vLLM 0.6.4.post1, the fraction of GPU memory the engine may use for weights, workspaces, and KV cache. Scheduler controls.--max-model-len: In vLLM 0.6.4.post1, the context-length limit, constraining possible KV-cache growth. Scheduler controls.--kv-cache-dtype: In vLLM 0.6.4.post1, the cache-format option, effective only with compatible engine and kernel support. Scheduler controls.--enable-prefix-caching: In vLLM 0.6.4.post1, the option enabling valid exact-prefix KV reuse. Scheduler controls.torch.distributed:PyTorch’s API for process groups, rank identities, and point-to-point or collective communication. Communication backends.no_sync()context: ThePyTorch2.14FullyShardedDataParallelcontext that defers gradient synchronization during accumulation. The first forward-backward pass after leaving it synchronizes accumulated gradients. Full-parameter retention depends on theShardingStrategysetting. FSDP state rules.
Notation and List Labels
Reference labels and notation, not formal glossary terms.
- On-chip: Location label for resources on the GPU die, including SM-local resources and device-wide L2 cache. Chapter 4 opening.
- Same-GPU: Visibility label for resources accessible across one GPU, including L2 cache and HBM. L2 is both on-chip and same-GPU, while HBM is same-GPU but off-chip. Chapter 4 opening.
- Same-server: Location label for GPUs and host memory joined within one server. Chapter 4 opening.
- Process- or rank-local: Label for the process or rank that holds and manages an object, regardless of where it is stored. Chapter 4 opening.
- Allocated versus reserved: Comparison label: allocated memory is storage used by live tensors. Reserved memory is the allocator’s whole pool, including those allocations and cached free blocks. The two figures are not added together. Section 4.3.
- V: In the parameter-count estimate of Section 10.4, the vocabulary size.
- B: In the KV-cache size estimate of Section 10.6, the active batch size or number of admitted sequences.
- T: In the KV-cache size estimate of Section 10.6, the sequence length retained in KV cache, including relevant prompt and generated tokens.
- L: In the KV-cache size estimate of Section 10.6, the number of transformer layers maintaining KV state.
- H with subscript kv: In the KV-cache size estimate of Section 10.6, the number of key-value heads, which can differ from the number of query heads.
- b: In the KV-cache size estimate of Section 10.6, bytes per cached element in the specified cache representation.
References
Consolidate the primary and supporting sources cited for technical claims, interfaces, measurements, and stated limits throughout the guide.
The list retains existing numbers as further sources are added. The numbers do not indicate first-citation order. Footnotes identify the supported claims and their limits. Software documentation describes the cited interface or version. It does not establish a general performance advantage.
- Grattafiori, A., et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783 (v1 2024-07-31, v3 2024-11-23).
- Touvron, H., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
- Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The Curious Case of Neural Text Degeneration. ICLR 2020. arXiv:1904.09751.
- PyTorch. (2024). Tensor Attributes (PyTorch 2.4 documentation). PyTorch tensor attributes.
- Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. 3rd International Conference on Learning Representations. Adam paper.
- Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2020). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. ZeRO paper (v3 2020-05-13).
- Ainslie, J., et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023. arXiv:2305.13245.
- Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
- PyTorch. (2024). CrossEntropyLoss (PyTorch 2.4 documentation). Input and target shapes.
- NVIDIA. (2024). CUDA C++ Programming Guide 12.4. CUDA C++ Programming Guide 12.4.
- Amdahl, G. M. (1967). Validity of the single processor approach to achieving large scale computing capabilities. In Proceedings of the April 18-20, 1967, Spring Joint Computer Conference (AFIPS ’67 Spring), 483-485. Amdahl’s law paper.
- NVIDIA. (n.d.). Parallel Thread Execution ISA 8.4 (CUDA 12.4 documentation). PTX ISA 8.4 documentation.
- Andersch, M., Palmer, G., Krashinsky, R., Stam, N., Mehta, V., Brito, G., & Ramaswamy, S. (2022). NVIDIA Hopper Architecture In-Depth (NVIDIA Technical Blog). NVIDIA Hopper architecture article.
- NVIDIA. (2023). Matrix Multiplication Background User’s Guide. Matrix multiplication performance guide.
- NVIDIA. (2026). GPUDirect RDMA Documentation. GPUDirect RDMA documentation.
- NVIDIA. (2016). Tesla P100 Datasheet. Tesla P100 datasheet (PDF).
- NVIDIA. (2018). Tesla V100 Datasheet. Tesla V100 datasheet (PDF).
- NVIDIA. (n.d.). NVIDIA A100 Tensor Core GPU specifications. NVIDIA A100 specifications.
- NVIDIA. (n.d.). NVIDIA H100 Tensor Core GPU product specifications. NVIDIA H100 specifications.
- NVIDIA. (n.d.). NVIDIA DGX B200 specifications. NVIDIA DGX B200 specifications.
- NVIDIA. (n.d.). cuBLAS Library (cuBLAS 13.4 documentation). cuBLAS documentation.
- Tillet, P., Kung, H. T., & Cox, D. (2019). Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations. In Proceedings of MAPL 2019 (pp. 10-19). Triton compiler paper.
- Triton project. (n.d.).
triton.Config, Python API reference for the Triton main branch.triton.ConfigAPI reference. - Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Re, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. FlashAttention paper.
- Dao, T. (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. FlashAttention-2 paper (v1).
- Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., & Dao, T. (2024). FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision. FlashAttention-3 paper (v2).
- Lefaudeux, B., Massa, F., et al. (2022). xFormers: A modular and hackable Transformer modelling library. xFormers repository.
- Chowdhery, A., et al. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311.
- NVIDIA. (n.d.). nvidia-smi documentation. nvidia-smi documentation.
- Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4), 65-76. Roofline paper.
- PyTorch. (2.14 documentation). torch.profiler. PyTorch profiler documentation.
- NVIDIA. (n.d.). Nsight Systems User Guide. Nsight Systems User Guide.
- NVIDIA. (n.d.). Nsight Compute Profiling Guide. Nsight Compute Profiling Guide.
- PyTorch. (2.14 documentation). CUDA semantics. PyTorch CUDA semantics.
- Ansel, J., et al. (2024). PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In Proceedings of ASPLOS 2024. PyTorch 2 paper.
- PyTorch. (2.14 documentation). torch.compile. torch.compile API documentation.
- PyTorch. (2.14 documentation). RMSNorm. RMSNorm API documentation.
- Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. LoRA paper.
- PyTorch. (2.14 documentation). Automatic Mixed Precision examples, Gradient accumulation. PyTorch gradient accumulation examples.
- NVIDIA. (n.d.). Floating Point and IEEE 754, Sections 2.1-2.2 (CUDA 13.4 documentation). Retrieved September 29, 2026, from Floating Point and IEEE 754 guide.
- NVIDIA. (Transformer Engine 2.19.0 documentation). Using FP8 and FP4 with Transformer Engine. Transformer Engine FP8 and FP4 guide.
- Micikevicius, P., et al. (2018). Mixed Precision Training. arXiv:1710.03740. Mixed Precision Training paper.
- Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv:2002.05202. GLU Variants Improve Transformer.
- Meta Llama. (llama-models at commit 0e0b8c5, Oct 10, 2025). models/sku_list.py. Llama model specifications source.
- Meta Llama. (llama-models at commit 0e0b8c5, Oct 10, 2025). models/llama3_3/MODEL_CARD.md. Llama 3.3 model card.
- Shazeer, N. (2019). Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150. Multi-query attention paper.
- Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323. GPTQ paper.
- Lin, J., et al. (2024). AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. In Proceedings of MLSys 2024. arXiv:2306.00978. AWQ paper.
- bitsandbytes-foundation. (n.d.). bitsandbytes. bitsandbytes repository.
- PyTorch. (torchao stable documentation). Welcome to the torchao Documentation. torchao documentation.
- Subramanian, S., Saroufim, M., & Zhang, J. (2022, February 8). Practical Quantization in PyTorch. PyTorch. Practical Quantization in PyTorch.
- PyTorch. (2.14 documentation). MinMaxObserver. MinMaxObserver API documentation.
- PyTorch. (2.14 documentation). torch.fake_quantize_per_tensor_affine. torch.fake_quantize_per_tensor_affine API documentation.
- Mishra, A., Albericio Latorre, J., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., & Micikevicius, P. (2021). Accelerating Sparse Deep Neural Networks. arXiv:2104.08378.
- Frantar, E., & Alistarh, D. (2023). SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot. ICML 2023. arXiv:2301.00774.
- Sun, M., Liu, Z., Bair, A., & Kolter, J. Z. (2024). A Simple and Effective Pruning Approach for Large Language Models. ICLR 2024. arXiv:2306.11695.
- Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
- NVIDIA. (n.d.). L2 Cache Control (CUDA Programming Guide 13.4.2). Retrieved September 28, 2026, from CUDA L2 cache control guide.
- Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. In SOSP 2023. arXiv:2309.06180. PagedAttention paper.
- vLLM contributors. (n.d.). Performance and Tuning, Preemption (v0.6.4.post1 documentation). vLLM performance and preemption guide.
- Gu, A., & Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752. Mamba paper.
- Jiang, A. Q., et al. (2023). Mistral 7B. arXiv:2310.06825.
- Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2024). Efficient Streaming Language Models with Attention Sinks. ICLR 2024. arXiv:2309.17453.
- DeepSeek-AI. (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434.
- vLLM contributors. (n.d.). Engine Arguments (v0.6.4.post1 documentation). vLLM engine arguments.
- Hugging Face. (n.d.). Utilities for generation. Retrieved September 28, 2026, from Transformers generation utilities.
- Little, J. D. C. (1961). A Proof for the Queuing Formula: L = λW. Operations Research, 9(3), 383-387. Little’s law paper.
- Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., & Ramjee, R. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. Sarathi-Serve paper.
- Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, I., Maleki, S., & Bianchini, R. (2024). Splitwise: Efficient Generative LLM Inference Using Phase Splitting. Splitwise paper.
- Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., & Sheng, Y. (2024). SGLang: Efficient Execution of Structured Language Model Programs. SGLang paper.
- Liu, Y., Cheng, Y., Yao, J., An, Y., Chen, X., Feng, S., Huang, Y., Shen, S., Zhang, R., Du, K., & Jiang, J. (2025). LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference. LMCache paper.
- Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. Speculative decoding paper.
- Willard, B. T., & Louf, R. (2023). Efficient Guided Generation for Large Language Models. Guided generation paper.
- NVIDIA. (n.d.). Overview (TensorRT LLM 1.3.0rc28 documentation). Retrieved September 28, 2026, from TensorRT LLM overview.
- NVIDIA. (n.d.). Migration Guide: TensorRT Backend Removed (TensorRT LLM 1.3.0rc28 documentation). Retrieved September 28, 2026, from TensorRT backend removal migration guide.
- PyTorch. (2.14 documentation). torch.allclose. torch.allclose API documentation.
- PyTorch contributors. (2026). Common Graph Breaks (PyTorch 2.14 documentation). PyTorch common graph breaks.
- PyTorch. (2.14 documentation).
torch.Tensor.copy_.torch.Tensor.copy_API documentation. - Bindel, D. (2015, October 6). Distributed memory: Networks and models. Cornell University, CS 5220 course notes, Network properties. Distributed memory: Networks and models.
- NVIDIA. (2026). NVIDIA Collective Communications Library Documentation. NCCL documentation.
- NVIDIA. (2024). NCCL Tests PERFORMANCE Documentation. NCCL Tests performance documentation.
- PyTorch. (2026). Distributed Communication Package: torch.distributed. PyTorch distributed communication documentation.
- NVIDIA. (NCCL 2.32.3 documentation). Environment Variables. Retrieved September 28, 2026, from NCCL environment variables.
- PyTorch. (2026). Distributed Data Parallel. Distributed Data Parallel design note.
- PyTorch. (2026). FullyShardedDataParallel. FullyShardedDataParallel documentation.
- PyTorch. (2026). SGD. Versioned v2.14 API documentation at SGD API documentation.
- Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., & He, Y. (2021). ZeRO-Offload: Democratizing Billion-Scale Model Training. USENIX ATC 2021. arXiv:2101.06840.
- Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. Megatron-LM paper.
- Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., & Chen, Z. (2019). GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism. GPipe paper (HTML, v5).
- Lepikhin, D., et al. (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. GShard paper.
- Korthikanti, V., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., & Catanzaro, B. (2022). Reducing Activation Recomputation in Large Transformer Models. arXiv:2205.05198.
- Liu, H., Zaharia, M., & Abbeel, P. (2023). Ring Attention with Blockwise Transformers for Near-Infinite Context. arXiv:2310.01889.
- NVIDIA. (n.d.). Parallelisms Guide (Megatron Bridge documentation). Retrieved September 28, 2026, from Megatron Bridge parallelisms guide.
- Narayanan, D., et al. (2021). Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. SC21. arXiv:2104.04473.
- Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 23(120), 1-39. arXiv:2101.03961.
- ggml-org contributors. (n.d.). GGUF model file specification. Retrieved October 11, 2026, from the GGUF model file specification.
- ggml-org contributors. (n.d.). llama.cpp project README. Retrieved October 11, 2026, from the llama.cpp project README.
- PyTorch contributors. (2.14 documentation).
torch.cuda.synchronize. Retrieved October 11, 2026, from the CUDA device synchronization API. - PyTorch contributors. (2.14 documentation).
torch._logging.set_logs. Retrieved October 11, 2026, from the PyTorch compiler logging API. - PyTorch contributors. (2.14 documentation).
torch.autograd.grad_mode.inference_mode. Retrieved October 11, 2026, from the PyTorch inference mode API. - vLLM contributors. (v0.6.4 source). Scheduler preemption policy. Retrieved October 11, 2026, from the tagged vLLM scheduler source.
- PyTorch contributors. (2.4 documentation). CUDA semantics. Retrieved October 11, 2026, from the PyTorch CUDA stream and allocator guidance.
- PyTorch contributors. (2.14 documentation).
torch.mm. Retrieved October 14, 2026, from the matrix multiplication API. - PyTorch contributors. (2.14 documentation).
torch.autograd.profiler.record_function. Retrieved October 14, 2026, from the profiler label API. - PyTorch contributors. (2.14 documentation). torchrun. Retrieved October 14, 2026, from the elastic launch guide.
- NVIDIA. (NCCL 2.32.3 documentation). Logging. Retrieved October 14, 2026, from the NCCL logging guide. The URL can change versions.
- PyTorch contributors. (2.14 documentation).
torch.testing.assert_close. Retrieved October 14, 2026, from the tensor comparison API. - vLLM contributors. (latest documentation). OpenAI-Compatible Server. Retrieved October 14, 2026, from the vLLM compatible server guide. The link follows the latest release.
- vLLM contributors. (latest source documentation). Chat-completion serving. Retrieved October 14, 2026, from the vLLM chat-completion implementation. The link follows the latest release.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. In Advances in Neural Information Processing Systems, 30. Original Transformer paper.
- Gardin, A. (2026, June 26). CUDA and Triton: Writing fast GPU kernels: from a naïve CUDA matrix-vector multiply to fused FlashAttention in Triton [Supplied slides,
Performance Engineering - L4.pdf]. - WHATWG. (n.d.). HTML Living Standard, Server-sent events. Retrieved October 23, 2026, from Parsing an event stream and Interpreting an event stream.
Huang, Y., et al. (2019). GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism. arXiv:1811.06965v5, Section 2.3 and Figure 2. GPipe paper (HTML, v5). The bubble estimate assumes balanced stages. Communication delays and stage imbalance can increase measured idle time.↩︎
Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast inference from Transformers via speculative decoding. Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 19274-19286. Speculative decoding paper. The analysis assumes independent acceptance and a simplified relative draft cost. The formulas do not by themselves predict latency under a loaded serving scheduler.↩︎
vLLM contributors. (n.d.). Engine Arguments (v0.6.4.post1 documentation). vLLM engine arguments. Documents the listed token, memory, context-length, and cache-format controls. Defaults and supported combinations can change by release.↩︎
NVIDIA. (NCCL 2.32.3 documentation). Environment Variables,
NCCL_DEBUGandNCCL_DEBUG_SUBSYS. Distinguishes the INFO logging level from the subsystem filter and lists its default and diagnostic categories. Logging does not establish achieved communication performance.↩︎