Mathematical Foundations of LLM Training and Inference

Author

Vitaly Rubinovich

Publisher version
0.8.21
Released

An engineer adjusts a brass instrument with translucent vector grids and connected layers, beside a descending track, measuring gauge, books, and mathematical sketches.

A language model’s behavior depends on calculations you can inspect: how text becomes numbers, how loss gradients guide parameter updates, and how a trained model produces an answer. This book develops the mathematical foundations of large language models through small examples and computational labs. It helps you connect those calculations to practical decisions about model quality, adaptation, memory, and inference.

Contents

Front matter

  • Roadmap
    Reading routes need the assumed starting knowledge and the teaching dependencies made explicit.
  • Notation
    Recurring symbols and dimensions need local meanings and compatible orientation and mask conventions.
  • Library Map
    Library operations need links to their mathematical roles, executable examples, and full experiment requirements.
  • Introduction: Models, tasks, and learning

Part I: From Text to Mathematical Objects

Numerical text representations supply model inputs. Reference answers support prediction-error calculations.

Part II: From Scores to Loss

Scores support different calculations for predictions, token selection, loss, and evaluation.

Part III: From Error to Learning

Gradients and update rules change parameters, while the model representation limits which patterns can be learned.

Part IV: Modeling Sequences

Sequence models use ordered context, with different rules for readable positions and prediction targets.

Part V: Adapting and Running LLMs

A pretrained model needs task evidence and resource measurements to guide adaptation and generation choices.

Part VI: Implementation and Extensions

Other input types, tensor storage, and experiments need separate checks of representation, execution, and task evidence.

Appendices

  • Execution records and supporting notebooks
    Reference records separate local checks, historical outputs and full experiments, retaining their data, versions, settings and missing execution.
  • Glossary
    Recurring terms and abbreviations need short definitions linked to their teaching locations.
  • References
    Claims need traceable publications and documentation linked to their explanations.