Part V: Distributed execution
When included, the optional distributed labs are in the Optional Labs appendix. The main chapters explain each update and communication result independently; the collection supplies longer executable launch and measurement programs.
Chapters in this part
- 16. Collective data movement and its cost: A collective defines the result each rank receives. Schedule, route and readiness affect its cost.
- 17. Launching and measuring distributed work: A distributed program needs correct worker identities, device assignments and communication paths before its timing can be interpreted.
- 18. Replicated and sharded training: Replicated and sharded training keep different model state on each rank and require different exchanges during an update.
- 19. Tensor and sequence parallelism: Tensor, sequence and context parallelism divide different layer operations or saved state across ranks.
- 20. Pipeline, expert and combined layouts: Pipeline stages move activations, expert ranks move token representations, and combined layouts require compatible rank groups.
- 21. Distributed serving and KV locality: Moving an active request can add cache-transfer or recomputation time, so worker placement must account for its KV state.
