Mathematical Foundations of LLM Training and Inference

A language model’s behavior depends on calculations you can inspect: how text becomes numbers, how errors change weights, and how a trained model produces an answer. This book develops the mathematical foundations of large language models through small examples and computational labs. It helps you connect those calculations to practical decisions about model quality, adaptation, memory, and inference.

Contents

Front matter

• Models, tasks, and learning
• Roadmap
• Notation
• Library Map

Part I: From Text to Mathematical Objects

• Chapter 1: Prediction targets and training pairs
• Chapter 2: Token IDs and batches
• Chapter 3: Subwords and sequence counts
• Chapter 4: Count vectors and TF-IDF

Part II: From Scores to Loss

• Chapter 5: Linear models: Predictions, boundaries, and slopes
• Chapter 6: Sigmoid and softmax: Scores as probabilities
• Chapter 7: Decoding Strategies: Generating Text Token by Token
• Chapter 8: Cross-Entropy & Perplexity: Measuring Prediction Error and Uncertainty
• Chapter 9: Validation & Testing: Catching Overfitting and Measuring Generalization

Part III: From Error to Learning

• Chapter 10: Optimization and regularization
• Chapter 11: Backpropagation: Tracing Gradients through a Computation
• Chapter 12: Activation Functions: Nonlinear Features for Classification
• Chapter 13: Embedding Tables: Token Lookup, Gradients, and Pooling
• Chapter 14: Word2Vec: Learning Vector Relationships from Context

Part IV: Modeling Sequences

• Chapter 15: Recurrent networks: Context, state, and sequential computation
• Chapter 16: Attention mathematics
• Chapter 17: Transformer blocks: Context, features, and position
• Chapter 18: Transformer families: Visibility, targets, and generation
• Chapter 19: Pretraining data and compute allocation

Part V: Adapting and Running LLMs

• Chapter 20: Model adaptation: Context, preferences, and trainable weights
• Chapter 21: KV cache: Generation state, latency, and allocation

Part VI: Implementation and Extensions

• Chapter 22: Images, audio, and tables: Representations and tasks
• Chapter 23: Tensor storage: Shapes, views, devices, and lifetime
• Chapter 24: Computational labs: Checks, experiments, and evidence

Appendices

• Glossary
• References