Part I: Reproducible training at cluster scale

An experiment is difficult to repeat or diagnose when its code, data, or runtime is unknown. A larger training run adds other constraints: model state may exceed one GPU, communication can delay updates, and a failed worker can erase useful work. The input data and saved training state must remain available when workers need them.

Chapter 1 establishes the records and responsibilities needed to reconstruct a run and estimate its capacity. Chapter 2 explains how devices share the training work and recover after failure. Chapter 3 connects their input path to durable storage and a tested recovery point. Together, these records and storage choices make the result of a distributed run identifiable and recoverable.

Chapter 1 establishes run identity, tracked artifacts, and team ownership; Chapter 2 coordinates and measures GPU workers; Chapter 3 supplies training data and verifies checkpoint recovery. The path ends with a recoverable training bundle.
Figure 1: The three chapters connect identifiable run inputs, coordinated GPU work, and tested checkpoint recovery into a recoverable training bundle.

Chapters in this part

  • Making machine learning runs reproducible: When a model run’s code, data, runtime, or approval record is unclear, another engineer cannot reliably reconstruct what was tested or decide whether that version is ready to deploy. Production adds shared resources, failures, and evidence requirements. Identified artifacts, an explicit responsibility split, and suitable infrastructure turn an ad-hoc run into a repeatable system.
  • Distributed training across devices and clusters: How can one training job use many GPUs without turning communication, scheduling, or failure into the real bottleneck? Chapter 1 defined the versioned artifacts and software layers around a run. Single-GPU limits lead to network and process identities, then to recoverable distributed execution and measurement.
  • Storing training data, checkpoints, and artifacts: A 512-GPU training run can stall even when every GPU is healthy. If the input path delivers batches late, the GPUs wait. If a save is interrupted halfway, a later restore can load a checkpoint in which some ranks’ shards are missing. Chapter 2 decided what a checkpoint must contain and how often to take one. Whether the run actually keeps its GPUs busy and recovers correctly depends on where its data and checkpoints live and how they are written.