Mathematical Foundations of LLM Training and Inference

A language model’s behavior depends on calculations you can inspect: how text becomes numbers, how loss gradients guide parameter updates, and how a trained model produces an answer. This book develops the mathematical foundations of large language models through small examples and computational labs. It helps you connect those calculations to practical decisions about model quality, adaptation, memory, and inference.
Contents
Front matter
- Roadmap
Reading routes need the assumed starting knowledge and the teaching dependencies made explicit. - Notation
Recurring symbols and dimensions need local meanings and compatible orientation and mask conventions. - Library Map
Library operations need links to their mathematical roles, executable examples, and full experiment requirements. - Introduction: Models, tasks, and learning
Part I: From Text to Mathematical Objects
Numerical text representations supply model inputs. Reference answers support prediction-error calculations.
- Chapter 1: Prediction targets and training pairs
In supervised and self-supervised training, inputs are paired with reference answers and prediction errors are combined. - Chapter 2: Token IDs and batches
A shared vocabulary mapping makes token IDs consistent between tokenizer and model. Padding masks record added batch positions. - Chapter 3: Subwords and sequence counts
Reusable subword pieces balance vocabulary size against sequence length, while continuation counts supply a fixed-history prediction baseline. - Chapter 4: Count vectors and TF-IDF
Document terms need shared feature coordinates before the same classifier weights can score texts of different lengths.
Part II: From Scores to Loss
Scores support different calculations for predictions, token selection, loss, and evaluation.
- Chapter 5: Linear models
Weighted features define predictions and decision boundaries. Loss gradients describe the effect of small parameter changes. - Chapter 6: From scores to probabilities
Sigmoid and softmax turn scores into probabilities under different class assumptions. - Chapter 7: Decoding strategies
Next-token probabilities need a selection rule and repeated feedback to produce a generated sequence with fixed parameters. - Chapter 8: Cross-entropy and perplexity
Logarithmic losses compare probabilities with reference answers. An inclusion rule determines their mean and perplexity. - Chapter 9: Validation and testing
Performance on new inputs needs held-out examples, task-appropriate metrics, and development choices separated from final testing.
Part III: From Error to Learning
Gradients and update rules change parameters, while the model representation limits which patterns can be learned.
- Chapter 10: Optimization and regularization
Update rules turn gradients into parameter changes. Controlled validation comparisons assess training settings. - Chapter 11: Backpropagation
Parameter gradients need all contributing derivative paths before an optimizer can apply a consistent update. - Chapter 12: Activation functions
Nonlinear hidden features can make patterns separable by a final affine classifier. - Chapter 13: Embedding tables
Embedding lookup gives each token ID a vector row. Aggregation combines valid rows into a document representation. - Chapter 14: Word vectors from context
Context statistics and learning objectives give token vectors relationships that can be inspected and evaluated.
Part IV: Modeling Sequences
Sequence models use ordered context, with different rules for readable positions and prediction targets.
- Chapter 15: Recurrent networks: Context, state, and sequential computation
Recurrent state carries earlier information through a sequence, while output layouts determine which predictions are compared. - Chapter 16: Attention mathematics
A position needs a defined set of readable vectors and normalized comparison weights before attention can combine their content. - Chapter 17: Transformer blocks: Context, features, and position
Transformer blocks combine attention, feature transformations, residual paths and normalization, with position rules when order matters. - Chapter 18: Transformer families: Visibility, targets, and generation
Model families differ in input visibility. Objectives define training targets, while output tasks determine whether generation is needed. - Chapter 19: Pretraining data and compute allocation
Source selection and packing define training inputs and targets. A compute budget constrains model size and processed tokens.
Part V: Adapting and Running LLMs
A pretrained model needs task evidence and resource measurements to guide adaptation and generation choices.
- Chapter 20: Model adaptation: Context, preferences, and trainable weights
Adaptation can change context, parameters, or both. Its objective, trainable state, and storage format are separate choices. - Chapter 21: KV cache: Generation state, latency, and allocation
Cached generation reuses attention state. Scheduling and allocation affect request latency and memory use.
Part VI: Implementation and Extensions
Other input types, tensor storage, and experiments need separate checks of representation, execution, and task evidence.
- Chapter 22: Images, audio, and tables: Representations and tasks
Images, audio, and table fields need representations that preserve the structure required by their distinct output tasks. - Chapter 23: Tensor storage: Shapes, views, devices, and lifetime
Tensor memory and execution claims need storage, layout, numerical type, device, and state lifetime beyond logical shape. - Chapter 24: Computational labs: Checks, experiments, and evidence
Training and inference experiments declare their inputs, procedures, and comparison rules so results can support the intended claims.
Appendices
- Execution records and supporting notebooks
Reference records separate local checks, historical outputs and full experiments, retaining their data, versions, settings and missing execution. - Glossary
Recurring terms and abbreviations need short definitions linked to their teaching locations. - References
Claims need traceable publications and documentation linked to their explanations.