Roadmap
Reading routes need the assumed starting knowledge and the teaching dependencies made explicit.
The book follows how data becomes a model input, how a calculation produces a prediction, and how training can change that calculation. The introduction distinguishes inference, training and evaluation before the mathematical chapters begin.
The six Parts organize learning dependencies. Two representation routes share the task definitions: document features can feed a classifier, while ordered token IDs can select vectors for a sequence model. Neither representation must pass through the other.
- Part I: From Text to Mathematical Objects, Chapters 1–4. Inputs, targets, token IDs, batches, subwords, and fixed feature coordinates.
- Part II: From Scores to Loss, Chapters 5–9. Affine predictions, local derivatives, probabilities, inference choices, loss, and held-out evaluation.
- Part III: From Error to Learning, Chapters 10–14. Parameter updates, gradient propagation, nonlinear models, and learned token vectors.
- Part IV: Modeling Sequences, Chapters 15–19. Limited history, recurrent state, attention, transformer families, and pretraining.
- Part V: Adapting and Running LLMs, Chapters 20–21. Adapting a pretrained model and managing generation state.
- Part VI: Implementation and Extensions, Chapters 22–24. Other input modalities, tensor storage, and computational labs.
The arrows relate prerequisites. Later chapters use fixed parameters during decoding, compare predictions with learning targets during training, or measure fixed-model behavior against task criteria during evaluation, as the local calculation requires.
The second map connects representation, prediction, learning, sequence context, serving, and implementation. These are related study routes, with the task determining which operations are required.
The introduction separates the required output, model structure, learning setting, and inference procedure. These choices remain distinct. A neural network can predict a category, a number, or a text sequence. Which form it produces depends on its inputs and output calculation.
The family comparison below is a lookup aid to the later mechanisms. A recurrent model uses the preceding hidden state when processing the next sequence position. Attention combines information from permitted positions. Vision and multimodal models first convert images, audio, or other inputs into suitable vectors.
| Family | Context used | Typical output | Chapters |
|---|---|---|---|
| Linear and logistic models | A fixed feature vector | A number or one class | 5-6, 9 |
| N-gram models | A fixed token history | A next-token probability | 3.4, 15.1 |
| RNNs and LSTMs | A recurrent hidden state | Sequence labels or generated tokens | 15 |
| Transformers | Attention over permitted positions | Classes, tokens, or a target sequence | 16-21 |
| Vision and multimodal models | Patches, frames, fields, or text vectors | A label, matched pair, or generated text | 22 |
Main sources of the book
Three general references support different parts of the book:
- Speech and Language Processing, by Daniel Jurafsky and James H. Martin, covers text representations, language models, sequence prediction and their evaluation.1
- The Elements of Statistical Learning, by Trevor Hastie, Robert Tibshirani and Jerome Friedman, develops prediction loss, generalization and model assessment.2
- Dive into Deep Learning, by Aston Zhang, Zachary C. Lipton, Mu Li and Alexander J. Smola, explains the sample and population losses, entropy and distribution comparisons used here.3
Research papers supply the particular methods and findings discussed in later chapters. They include papers from NeurIPS, ICML, ICLR and ACL, as well as manuscripts hosted on arXiv. The footnotes beside each explanation identify the relevant paper and the claim it supports.
Documentation and source code serve a different purpose: they specify how the cited versions of PyTorch, transformers, tokenizers, peft and other libraries implement those methods. Dataset references identify the inputs used in the labs. The full entries remain in References.
Starting knowledge and tensor notation
This book is for software engineers and technically trained readers who want to understand the calculations behind language-model training and inference. It assumes bachelor-level mathematics and statistics, together with programming experience sufficient to follow Python arrays, indexing, functions, and imports.
The book uses familiar mathematics while explaining what its quantities mean for a model. §5.4 applies derivatives and the chain rule to parameter updates. Chapter 11 extends that calculation to automatic differentiation through a graph.
A tensor is an array with explicit axes, a numerical type, and a storage device. An array with dimensions \([B,T,D]\) contains \(B\) examples, \(T\) positions per example, and \(D\) coordinates per position. Integer arrays hold token IDs, floating-point arrays hold learned vectors, and Boolean arrays can mark permitted positions. The device identifies where the array is stored and computed, such as central processing unit (CPU) memory or a graphics processing unit (GPU). CUDA is NVIDIA’s software and device platform through which the PyTorch examples can select a supported GPU, such as device cuda:0.
With \(F\) input features and \(D_{\mathrm{out}}\) outputs, the row-batch convention uses feature rows \([B,F]\) and mathematical weights \([F,D_{\mathrm{out}}]\) to produce \([B,D_{\mathrm{out}}]\) outputs. §5.3 relates this orientation to the transposed weight storage of nn.Linear. Chapter 23 develops storage, strides, copies, and device costs.
Reading paths and scope
Read Chapters 1–6 for targets, representations, and probability outputs. From there, Chapter 7 follows inference with fixed parameters. Chapter 8 develops losses, Chapter 9 evaluation, Chapter 10 update rules, and Chapter 11 gradient propagation. Chapter 10 uses supplied gradients to compare update rules. Chapter 11 then derives those gradients and applies the complete update cycle.
For text classification, continue from Chapter 11 through nonlinear classifiers, embedding lookup, and Word2Vec in Chapters 12–14 before the classification lab. For the count-model route, begin with n-gram counts and smoothing, read logarithmic loss and perplexity in §§8.1 and 8.5, then continue to the count-model lab. The count model provides a comparison with fixed history for the later sequence models. After Chapters 15–16, Chapters 17–19 develop transformer blocks, model families and pretraining. Chapters 20–21 then cover adaptation and serving before the audience-adaptation experiment. The multimodal lab examines image representations, captioning, and visual question answering.
The book develops the mathematical objects and calculations needed to inspect model training and inference: tensor dimensions, probabilities, gradients, attention, adapters, and cache storage. Kernel programming, Tensor Core scheduling, memory coalescing, production benchmarking, and complete distributed training configurations are outside its scope. Mixed precision and serving engines appear only where they change a mathematical assumption, memory estimate, or experimental choice.
References and execution guidance
Use the notation reference to check symbols, dimensions, and mask conventions. The library map connects each operation to its first introduction and practical exercise. The Glossary expands abbreviations and returns to a section that explains or applies them.
Each code example states its scope. A runnable calculation checks the listed inputs and outputs. An abbreviated fragment requires the surrounding model or data objects. A full experiment additionally needs its dataset split, model checkpoint, inference settings, and held-out results. A training experiment also needs its training configuration. Passing a tensor assertion establishes neither model quality nor a speed improvement. The environment and supporting notebooks explain the environment and evidence to retain.
The References collect the sources cited beside the explanations.
Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. Third edition draft, released August 19, 2026.↩︎
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction. Second edition, Springer.↩︎
Zhang, A., Lipton, Z. C., Li, M., & Smola, A. J. (2023). Dive into Deep Learning. Cambridge University Press, online version 1.0.3, §§3.6.1 and 22.11.↩︎