Introduction and Learning Path

Read the operation and required state before execution, measurement and optimization; use optional lab collections independently.

Large language model performance depends on the operations a workload runs and the data it keeps live. The hardware moves that data through memory, executes kernels and sometimes exchanges results with other GPUs. Understanding those mechanisms comes before measuring a suspected limit or choosing a change.

The guide assumes C, general programming, conventional computer and CPU architecture, and the mathematics used for vectors, matrices and elementary calculus. Python and PyTorch notation is explained beside its first use. GPU execution, transformer operations, caches and distributed processes are taught before the examples that depend on them.

Book map

Book map with five groups. Part I, chapters 1–3, covers model operations, workload state and GPU hardware. Part II, chapters 4–8, covers CUDA, Triton, performance models, profiling and compilation. Part III, chapters 9–12, covers training state, inference budgets, compression and KV-cache layout. Part IV, chapters 13–15, covers scheduling, decoding and serving measurement. Part V, chapters 16–21, covers collectives, launch, sharding, tensor and sequence parallelism, pipeline and expert layouts, and distributed serving. Each topic has a schematic icon; grouping follows reading order.
Figure 1: Five Parts group the twenty-one core chapters. Their order describes a reading path, not one runtime pipeline.

When included, four optional lab collections share the Optional Labs appendix: GPU experiments, training profiling, decode and serving probes, and distributed programs. These collect longer setup, executable programs, parameterized probes and pseudocode, each labeled with its requirements and limits. Each collection is independent and removable. The core chapters supply their own necessary explanations and short examples, so skipping every lab leaves a complete reading path.

Main references

These papers and technical guides introduce the main models, methods and execution rules used in the book. Chapter notes connect individual claims to their sources; the References give the complete bibliography.

  • Transformer attention: Vaswani and colleagues, Attention Is All You Need (2017), for attention and the Transformer architecture.
  • GPU execution: NVIDIA, CUDA C++ Programming Guide, version 12.4 (2024), for threads, memory and synchronization.
  • Framework execution: PyTorch, CUDA semantics, version 2.14, for tensor placement, streams and memory allocation.
  • Performance modelling: Williams, Waterman and Patterson, Roofline (2009), for relating arithmetic work and memory traffic to hardware limits.
  • Attention kernels: Dao and colleagues, FlashAttention (2022), for tiled attention that reduces reads and writes in the GPU’s main memory.
  • Serving memory: Kwon and colleagues, Efficient Memory Management for Large Language Model Serving with PagedAttention (2023), for managing the key-value cache in blocks.
  • Training memory: Rajbhandari and colleagues, ZeRO (2020), for dividing optimizer state, gradients and parameters across training processes.
  • Distributed training: Shoeybi and colleagues, Megatron-LM (2019), for tensor parallelism; Huang and colleagues, GPipe (2019), for pipeline parallelism.

The References appendix combines Reference Material, with formulas, software controls, compatibility checks and short practice questions; the Glossary, with definitions taught in the core chapters; and References, with papers and official documentation supporting the guide. Chapter footnotes identify their relevant claims and conditions.