Advanced LLM Performance Engineering

Contents
Front matter
- Introduction and Learning Path
Read the operation and required state before execution, measurement and optimization; use optional lab collections independently.
Part I: Models, workload state, and GPU hardware
Start with the operation and its stored arrays. Chapter 1 follows token IDs through model operations and explains tensor shape, dtype, layout and device. Chapter 2 follows training and generation before listing the state each phase keeps live. Chapter 3 then locates arithmetic and data in GPU hardware.
- Chapter 1: Model operations and tensors
Follow text/token IDs through the model; read tensor shape, dtype and storage descriptors. - Chapter 2: Workload phases and live memory
Trace a plain training/generation cycle before identifying which objects coexist. - Chapter 3: GPU hardware and memory
Locate arithmetic and bytes in the device and distinguish capacity from transfer rate.
Part II: GPU execution, measurement, and optimization
Chapter 4 follows host submission, CUDA kernels, copies and stream order. Chapter 5 maps Triton tiles to addresses and uses attention to show how retaining intermediate results changes memory traffic. Chapter 6 counts operations and bytes before Chapter 7 measures a known function and establishes a fair comparison. Chapter 8 then examines compilation and graph replay.
- Chapter 4: CUDA execution and data movement
Trace host submission, thread indices, copies and stream ordering. - Chapter 5: Triton and attention kernels
Trace tiles and masks, then attention and its intermediate storage; explain the mechanism before measured selection. - Chapter 6: Performance models
Count local work and bytes and bound a possible improvement without claiming a diagnosis. - Chapter 7: Measurement, profiling and fair comparisons
Measure a known local function, read its trace, and compare checked completed work using the minimum useful method. - Chapter 8: Compilation and runtime
Compare eager, compiled and replayed execution using a completed-work measurement.
Part III: Training and single-worker inference
Chapter 9 constructs one complete training update before studying input delivery, saved activations, numerical precision and state size. Chapter 10 follows one generation request and budgets its memory. Chapter 11 examines model compression for that inference workload. Chapter 12 follows repeated decode steps and KV-cache layout.
- Chapter 9: Training state and precision
Run and check a full local update before testing precision or training-state changes. - Chapter 10: Inference phases and memory budgets
Follow one request and calculate weight/KV memory and client timing. - Chapter 11: Model compression for inference
Relate saving, calibration, quality and an actually supported execution path. - Chapter 12: Decode and KV-cache layout
Trace cached versus uncached steps before byte equations, APIs and allocation choices.
Part IV: Serving systems
Chapter 13 coordinates requests through batching, admission, chunked prefill and prefix reuse. Chapter 14 covers speculative decoding and output constraints, including the result each method must preserve. Chapter 15 fixes a serving workload, measures client-visible results and compares the service stacks that can change them.
- Chapter 13: Request scheduling and prefix reuse
Trace admission, batching, compatible prefix reuse and prefill/decode handoff under a stated load. - Chapter 14: Speculative and constrained decoding
Trace exact sampling correction separately from constraint-state validity. - Chapter 15: Serving measurement and stack selection
Collect and interpret request outcomes/cost before selecting a service stack.
Part V: Distributed execution
Chapter 16 establishes collective inputs, outputs and transfer costs. Chapter 17 introduces ranks, launch, communication backends and completed distributed measurements. Chapters 18–20 then divide training state and model work through replicated, sharded, tensor, sequence, pipeline and expert layouts. Chapter 21 applies placement and communication to distributed serving.
- Chapter 16: Collective data movement and its cost
A collective defines the result each rank receives. Schedule, route and readiness affect its cost. - Chapter 17: Launching and measuring distributed work
A distributed program needs correct worker identities, device assignments and communication paths before its timing can be interpreted. - Chapter 18: Replicated and sharded training
Replicated and sharded training keep different model state on each rank and require different exchanges during an update. - Chapter 19: Tensor and sequence parallelism
Tensor, sequence and context parallelism divide different layer operations or saved state across ranks. - Chapter 20: Pipeline, expert and combined layouts
Pipeline stages move activations, expert ranks move token representations, and combined layouts require compatible rank groups. - Chapter 21: Distributed serving and KV locality
Moving an active request can add cache-transfer or recomputation time, so worker placement must account for its KV state.
Appendices
- References
- Appendix A. Reference Material
Provide compact formulas, software-control ownership, practice checks, and source mappings for implementing and checking the methods taught in the chapters. - Glossary
Define the guide’s formal terms and provide links to their teaching, with separate API and notation references. - References
Consolidate the primary and supporting sources cited for technical claims, interfaces, measurements, and stated limits throughout the guide.
- Appendix A. Reference Material
- Optional Labs