Part IV: Modeling Sequences
A static token vector does not identify which surrounding words matter to a particular occurrence. A sequence model needs both an input representation and a way to use ordered context. Fixed histories, recurrent states, and attention provide different access to earlier information.
Chapter 15 begins with output and target layouts, then recalls the count model’s fixed-history limit before introducing learned recurrent state. Chapter 16 replaces long state paths with direct attention over permitted positions. Chapter 17 composes attention, feature transformations, normalization, positions, and optional expert routing into blocks.
Chapter 18 compares model-family visibility, target construction and fixed-parameter task prompting, then develops iterative masked generation as a different output schedule. Chapter 19 connects the training objectives to selected records, packed storage and a compute budget. Part V uses task evidence to choose later adaptation and follows the request state retained during generation.
Chapters in this part
- Recurrent networks: Context, state, and sequential computation: Recurrent state carries earlier information through a sequence, while output layouts determine which predictions are compared.
- Attention mathematics: A position needs a defined set of readable vectors and normalized comparison weights before attention can combine their content.
- Transformer blocks: Context, features, and position: Transformer blocks combine attention, feature transformations, residual paths and normalization, with position rules when order matters.
- Transformer families: Visibility, targets, and generation: Model families differ in input visibility. Objectives define training targets, while output tasks determine whether generation is needed.
- Pretraining data and compute allocation: Source selection and packing define training inputs and targets. A compute budget constrains model size and processed tokens.