Advanced LLM Performance Engineering

Author

Vitaly Rubinovich

Publisher version
0.8.23
Released

A cutaway GPU on an engineering workbench shows a compute die, memory modules, incoming token tiles, layered persistent state, a fast interconnect to a second GPU, and a timing instrument.

Contents

Front matter

  • Introduction and Learning Path
    Read the operation and required state before execution, measurement and optimization; use optional lab collections independently.

Part I: Models, workload state, and GPU hardware

Start with the operation and its stored arrays. Chapter 1 follows token IDs through model operations and explains tensor shape, dtype, layout and device. Chapter 2 follows training and generation before listing the state each phase keeps live. Chapter 3 then locates arithmetic and data in GPU hardware.

Part II: GPU execution, measurement, and optimization

Chapter 4 follows host submission, CUDA kernels, copies and stream order. Chapter 5 maps Triton tiles to addresses and uses attention to show how retaining intermediate results changes memory traffic. Chapter 6 counts operations and bytes before Chapter 7 measures a known function and establishes a fair comparison. Chapter 8 then examines compilation and graph replay.

Part III: Training and single-worker inference

Chapter 9 constructs one complete training update before studying input delivery, saved activations, numerical precision and state size. Chapter 10 follows one generation request and budgets its memory. Chapter 11 examines model compression for that inference workload. Chapter 12 follows repeated decode steps and KV-cache layout.

Part IV: Serving systems

Chapter 13 coordinates requests through batching, admission, chunked prefill and prefix reuse. Chapter 14 covers speculative decoding and output constraints, including the result each method must preserve. Chapter 15 fixes a serving workload, measures client-visible results and compares the service stacks that can change them.

Part V: Distributed execution

Chapter 16 establishes collective inputs, outputs and transfer costs. Chapter 17 introduces ranks, launch, communication backends and completed distributed measurements. Chapters 18–20 then divide training state and model work through replicated, sharded, tensor, sequence, pipeline and expert layouts. Chapter 21 applies placement and communication to distributed serving.

Appendices