References

Claims need traceable publications and documentation linked to their explanations.

These sources support the claims cited beside the explanations. Entries follow their first use in the book. The footnotes identify the relevant sections and any limits of the cited evidence.

  1. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. Third edition draft, released August 19, 2026.

  2. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction. Second edition, Springer.

  3. Zhang, A., Lipton, Z. C., Li, M., & Smola, A. J. (2023). Dive into Deep Learning. Cambridge University Press, online version 1.0.3.

  1. Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1715–1725.
  1. OpenAI. (2019). GPT-2 byte pair encoding implementation. GPT-2 source code.

  2. Hugging Face. (n.d.). WordPiece tokenization. LLM Course, Chapter 6. Retrieved September 24, 2026.

  1. Kudo, T. (2018). Subword regularization: Improving neural network translation models with multiple subword candidates. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 66–75.
  1. Google. (n.d.). SentencePiece: README. Retrieved September 24, 2026.

  2. Scikit-learn developers. (n.d.). Feature extraction: TF-IDF term weighting. Scikit-learn documentation. Retrieved September 24, 2026.

  3. Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The Curious Case of Neural Text Degeneration. International Conference on Learning Representations.

  4. Hugging Face. (n.d.). Generation. Transformers documentation. Retrieved September 24, 2026.

  5. PyTorch Contributors. (2026). CrossEntropyLoss. PyTorch 2.12 documentation.

  6. Hugging Face. (2026). Transformers 5.10.2: Causal language-model loss. Version 5.10.2.

  7. Hugging Face. (2026). Transformers 5.10.2: GPT-2 forward. Version 5.10.2.

  8. Scikit-learn developers. (2026). Cross-validation: evaluating estimator performance. Scikit-learn 1.9.1 documentation.

  9. Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874.

  10. Scikit-learn developers. (2026). Probability calibration. Scikit-learn 1.9.1 documentation.

  11. Scikit-learn developers. (2026). roc_auc_score. Scikit-learn 1.9.1 documentation.

  1. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–318.
  1. Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, 74–81.

  2. Chen, M., et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374.

  3. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). Measuring Massive Multitask Language Understanding. International Conference on Learning Representations.

  1. Lin, S., Hilton, J., & Evans, O. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3214–3252.
  1. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019). HellaSwag: Can a Machine Really Finish Your Sentence? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4791–4800.
  1. Phan, L., et al. (2025). Humanity’s Last Exam. arXiv:2501.14249, version 1.

  2. Chollet, F. (2019). On the measure of intelligence. arXiv:1911.01547v1.

  3. PyTorch Contributors. (2026). SGD. PyTorch 2.12 documentation.

  4. Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12(61), 2121–2159.

  5. PyTorch Contributors. (2026). Adagrad. PyTorch 2.12 documentation.

  6. Hinton, G. (n.d.). Neural Networks for Machine Learning: Lecture 6e. Lecture slides, slide 29, crediting Tijmen Tieleman’s unpublished method. Retrieved September 30, 2026.

  7. PyTorch Contributors. (2026). RMSprop. PyTorch 2.12 documentation.

  8. Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations.

  9. Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. International Conference on Learning Representations.

  10. PyTorch Contributors. (2026). AdamW. PyTorch 2.12 documentation.

  11. Parikh, N., & Boyd, S. (2014). Proximal algorithms. Foundations and Trends in Optimization, 1(3), 127–239.

  12. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56), 1929–1958.

  13. PyTorch Contributors. (2026). Dropout. PyTorch 2.12 documentation.

  14. Loshchilov, I., & Hutter, F. (2017). SGDR: Stochastic Gradient Descent with Warm Restarts. International Conference on Learning Representations.

  15. PyTorch Contributors. (2026). clip_grad_norm_. PyTorch 2.12 documentation.

  16. Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13, 281–305.

  17. Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 25.

  18. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.

  19. PyTorch Contributors. (2026). Autograd mechanics. PyTorch 2.12 documentation.

  20. Hendrycks, D., & Gimpel, K. (2016). Gaussian Error Linear Units (GELUs). arXiv:1606.08415.

  21. Elfwing, S., Uchibe, E., & Doya, K. (2018). Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning. Neural Networks, 107, 3–11.

  22. Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. Proceedings of Machine Learning Research, 9, 249–256.

  23. PyTorch Contributors. (2026). Embedding. PyTorch 2.12 documentation.

  24. Harris, Z. S. (1954). Distributional Structure. WORD, 10(2–3), 146–162.

  25. Eisenstein, J. (2018). Natural Language Processing notes. Draft of November 13, Chapter 14, pages 325–326.

  1. Lin, D. (1998). Automatic Retrieval and Clustering of Similar Words. 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2, 768–774.
  1. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26.

  2. Gensim Contributors. (2025). Word2Vec implementation. Gensim 4.4.0 source.

  3. Gensim Contributors. (2025). Keyed vector queries. Gensim 4.4.0 source, get_mean_vector and most_similar.

  4. Mikolov, T., Le, Q. V., & Sutskever, I. (2013). Exploiting similarities among languages for machine translation. arXiv:1309.4168.

  5. Eckart, C., & Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika, 1(3), 211–218.

  6. van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9, 2579–2605.

  7. Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015). Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. Advances in Neural Information Processing Systems, 28, 1171–1179.

  8. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.

  9. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1724–1734.

  10. PyTorch Contributors. (2026). LSTM. PyTorch 2.12 documentation.

  11. Vaswani, A., et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.

  12. PyTorch Contributors. (2026). Scaled dot-product attention. PyTorch 2.12 documentation.

  13. PyTorch Contributors. (2026). MultiheadAttention. PyTorch 2.12 documentation.

  14. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and memory-efficient exact attention with IO-awareness. Advances in Neural Information Processing Systems, 35.

  15. Ba, J. L., Kiros, J. R., & Hinton, G. E. (2016). Layer normalization. arXiv:1607.06450.

  16. Zhang, B., & Sennrich, R. (2019). Root mean square layer normalization. Advances in Neural Information Processing Systems, 32.

  17. Shazeer, N. (2020). GLU variants improve transformer. arXiv:2002.05202, equations 5–6.

  18. Press, O., Smith, N. A., & Lewis, M. (2022). Train short, test long: Attention with linear biases enables input length extrapolation. International Conference on Learning Representations.

  19. Su, J., et al. (2024). RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568, 127063.

  1. Shazeer, N., et al. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. International Conference on Learning Representations.
  1. Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. OpenAI.

  2. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT, 4171–4186.

  3. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI technical report.

  4. Nie, S., et al. (2025). Large language diffusion models. arXiv:2502.09992, version 1.

  5. Hoffmann, J., et al. (2022). Training compute-optimal large language models. Advances in Neural Information Processing Systems, 35, 30016–30030.

  6. Kaplan, J., et al. (2020). Scaling laws for neural language models. arXiv:2001.08361, version 1.

  7. Brown, T. B., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33.

  8. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-Efficient Transfer Learning for NLP. ICML 2019, Proceedings of Machine Learning Research, 97, 2790–2799.

  9. Lester, B., Al-Rfou, R., & Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3045–3059.

  10. Kirkpatrick, J., et al. (2017). Overcoming catastrophic forgetting in neural networks. arXiv:1612.00796, version 2.

  11. Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33.

  12. Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs. I. The method of paired comparisons. Biometrika, 39(3–4), 324–345.

  13. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.

  14. Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073, version 1.

  15. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229–256.

  16. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv:1707.06347.

  17. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

  18. Hu, E. J., et al. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations.

  19. Hugging Face. (2025). PEFT checkpoint format. PEFT 0.17.0 documentation.

  20. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv:1503.02531.

  21. PyTorch Contributors. (2026). Tensor attributes. PyTorch 2.12 documentation.

  22. PyTorch Contributors. (2026). Type Info. PyTorch 2.12 documentation.

  23. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems, 36.

  24. Hugging Face. (2025). Quantization. PEFT 0.17.0 documentation.

  25. Ainslie, J., et al. (2023). GQA: Training generalized multi-query transformer models from multi-head checkpoints. Proceedings of EMNLP, 4895–4901.

  26. vLLM Contributors. (n.d.). Benchmark CLI: Understanding the latency metrics. Retrieved September 24, 2026.

  27. vLLM Contributors. (2025). Automatic prefix caching. vLLM 0.10.2 design documentation.

  1. Kwon, W., et al. (2023). Efficient memory management for large language model serving with PagedAttention. Proceedings of the 29th Symposium on Operating Systems Principles, 611–626.
  1. PyTorch Contributors. (2026). Conv2d. PyTorch 2.12 documentation.

  2. Geirhos, R., et al. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2, 665–673.

  3. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25.

  4. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778.

  1. Dosovitskiy, A., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations.
  1. Radford, A., et al. (2021). Learning transferable visual models from natural language supervision. Proceedings of Machine Learning Research, 139, 8748–8763.

  2. Kim, W., Son, B., & Kim, I. (2021). ViLT: Vision-and-language transformer without convolution or region supervision. Proceedings of Machine Learning Research, 139, 5583–5594.

  3. Dong, C., Loy, C. C., He, K., & Tang, X. (2015). Image super-resolution using deep convolutional networks. arXiv:1501.00092, version 3.

  4. van den Oord, A., Vinyals, O., & Kavukcuoglu, K. (2017). Neural discrete representation learning. Advances in Neural Information Processing Systems, 30.

  5. PyTorch Contributors. (2026). STFT. PyTorch 2.12 documentation.

  6. Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019). wav2vec: Unsupervised pre-training for speech recognition. Proceedings of Interspeech 2019, 3465–3469. DOI: 10.21437/Interspeech.2019-1873.

  7. Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449–12460.

  1. Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, 369–376.
  1. Radford, A., et al. (2023). Robust speech recognition via large-scale weak supervision. Proceedings of Machine Learning Research, 202, 28492–28518.

  2. Das, N., et al. (2024). SpeechVerse: A large-scale generalizable audio language model. arXiv:2405.08295, version 2.

  3. Borsos, Z., et al. (2023). AudioLM: A language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31, 2523–2533.

  4. Huang, X., Khetan, A., Cvitkovic, M., & Karnin, Z. (2020). TabTransformer: Tabular data modeling using contextual embeddings. arXiv:2012.06678.

  5. Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5), 1189–1232.

  6. PyTorch Contributors. (2026). Broadcasting semantics. PyTorch 2.12 documentation.

  7. PyTorch Contributors. (2026). Tensor views. PyTorch 2.12 documentation.

  8. PyTorch Contributors. (2026). Tensor.detach. PyTorch 2.12 documentation.

  9. PyTorch Contributors. (2026). Tensor.numpy. PyTorch 2.12 documentation.

  10. PyTorch Contributors. (2026). DistributedDataParallel. PyTorch 2.12 documentation.

  11. PyTorch Contributors. (2026). Automatic mixed precision examples. PyTorch 2.12 documentation.

  12. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., & Potts, C. (2013). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1631–1642.

  13. SetFit. (n.d.). Stanford Sentiment Treebank - Binary. Dataset SetFit/sst2, revision 00ea8ccb7a54b4e3780a3e51aa3f80361ff849c0.

  14. fancyzhx. (n.d.). AG News dataset card. Repository fancyzhx/ag_news, revision eb185aade064a813bc0b7f42de02595523103ca4.

  15. Hugging Face. (n.d.). WordPiece vocabulary trainer. Tokenizers 0.22.2 source code.

  16. Hugging Face. (n.d.). BPE vocabulary trainer. Tokenizers 0.22.2 source code.

  17. NLTK Project. (2025). Language-model preprocessing. NLTK 3.9.2 source, padded_everygram_pipeline.

  18. NLTK Project. (2025). Language-model API. NLTK 3.9.2 source, logscore, entropy, perplexity, and generate.

  19. NLTK Project. (2025). Language-model vocabulary. NLTK 3.9.2 source, Vocabulary and unk_cutoff.

  20. NLTK Project. (2025). Treebank tokenization and detokenization. NLTK 3.9.2 source, TreebankWordDetokenizer.

  21. Hugging Face. (n.d.). BLIP. Transformers 5.10.2 documentation.

  22. Hugging Face. (n.d.). GIT. Transformers 5.10.2 documentation.

  23. PyTorch Contributors. (2026). Reproducibility. PyTorch 2.12 documentation.

  24. vLLM Contributors. (2025). Engine arguments. vLLM 0.10.2 documentation.

  25. alvations. (2019). language-never-random.txt. Text file, GitHub Gist. Pinned revision b01ff96a5f76848450e648f35da6497ca9454e4a.

  26. kingburrito666. (n.d.). (Better) - Donald Trump Tweets!. Dataset, Kaggle.