Part V: Distributed execution

When included, the optional distributed labs are in the Optional Labs appendix. The main chapters explain each update and communication result independently; the collection supplies longer executable launch and measurement programs.

Six chapter scenes show collective results and readiness, worker launch and completed-work timing, replicated versus sharded state, row and token-row parallel computations, pipeline bubbles and expert returns, overlapping rank groups, and serving replicas with compatible KV handoff and prefix-aware placement.
Figure 1: Collectives exchange rank-owned data. Training and serving layouts distribute ownership and work, so placement, readiness and physical links affect the resulting time.

Chapters in this part