Part III: From Error to Learning

A loss measures prediction error, but its value alone does not specify a parameter change. Gradients describe local sensitivity to parameters. An update rule uses those sensitivities to change the model. The available representations determine which patterns those changes can express.

Chapter 10 compares update rules using supplied gradients, building on §5.4. Chapter 11 follows their calculation through graphs and completes the classifier updates in §11.5.

Chapter 12 shows how nonlinear hidden features separate XOR. Chapter 13 connects token IDs to trainable rows and masked document vectors. Chapter 14 supplies context-prediction objectives and examines the limits of static vectors. Those limits motivate the sequence models in Part IV.

Five numbered chapter panels form one horizontal row. they show optimizer updates, graph-derived gradients, nonlinear XOR features, embedding-table rows, and static word relationships learned from context. A return arrow runs from Chapter 11 to Chapter 10. A separate lower band shows validation and test evidence outside the training loop.
Figure III.1: The five panels map the chapters in Part III. Chapter 10 compares optimizer and regularization choices. Chapter 11 derives the gradients those updates use, and the return arrow links the two chapters. Chapter 12 shows how nonlinear features can separate XOR. Chapter 13 turns token IDs into trainable rows. Chapter 14 learns static word relationships from context. A separate lower band keeps validation and test evidence outside training.

Chapters in this part

  • Optimization and regularization: Update rules turn gradients into parameter changes. Controlled validation comparisons assess training settings.
  • Backpropagation: Parameter gradients need all contributing derivative paths before an optimizer can apply a consistent update.
  • Activation functions: Nonlinear hidden features can make patterns separable by a final affine classifier.
  • Embedding tables: Embedding lookup gives each token ID a vector row. Aggregation combines valid rows into a document representation.
  • Word vectors from context: Context statistics and learning objectives give token vectors relationships that can be inspected and evaluated.