Part I: Making training runs repeatable and recoverable
An experiment is difficult to repeat or diagnose when its code, data, or software environment is unknown. A larger training run may need more memory than one GPU (graphics processing unit), a processor for parallel numerical work, provides. Sharing training work adds communication that can delay updates, and a failed worker can erase useful work. The input data and saved training state must remain available when workers need them.
Chapter 1 establishes the records and responsibilities needed to reconstruct a run and estimate its capacity. Chapter 2 explains how devices share the training work and recover after failure. Chapter 3 connects their input path to durable storage and a tested recovery point. Together, these records and storage choices make the result of a distributed run identifiable and recoverable.
Chapters in this part
- Making machine learning runs reproducible: When a model run’s code, data, runtime, or approval record is unclear, another engineer cannot reliably reconstruct what was tested or decide whether that version is ready to deploy. Production adds shared resources, failures, and evidence requirements.
- Scaling training and recovering from interruptions: How can one training job use many GPUs without turning communication, scheduling, or failure into the real bottleneck? Chapter 1 defined the versioned artifacts and software layers around a run. Single-GPU limits lead to network and process identities, then to recoverable distributed execution and measurement.
- Supplying training data and recovering saved state: A 512-GPU training run can stall even when every GPU is healthy. If the input path delivers batches late, the GPUs wait. If a save is interrupted halfway, a later restore can load a checkpoint in which some ranks’ shards are missing. Whether the run actually keeps its GPUs busy and recovers correctly depends on where its data and checkpoints live and how they are written.