Notation

Recurring symbols and dimensions need local meanings and compatible orientation and mask conventions.

Several equations reuse symbols for examples, positions, dimensions, and parameters. This reference collects recurring meanings so a reader can resolve a symbol or shape and return to its calculation. The linked sections explain the mechanisms and provide examples. A chapter’s local definitions take precedence over shorthand in this page.

A corpus is a collection of texts. This reference writes its \(N\) documents as \(d_i\):

\[ C = \{d_1,\ldots,d_N\} \tag{1}\]

Section 3.1 counts token pairs within a training corpus. Section 4.2 arranges documents in feature rows.

The tokenization process in §2.1 maps text to an ordered sequence of pieces. This reference writes that map as \(\operatorname{tok}\), with \(T\) pieces \(s_i\):

\[ \operatorname{tok}(\text{text}) = (s_1,\ldots,s_T) \tag{2}\]

Vocabulary lookup assigns each piece an integer ID \(x_i\). Here \(V\) is the vocabulary set and \(|V|\) its size:

\[ x_i = \operatorname{id}(s_i),\quad x \in \{0,\ldots,\lvert V\rvert-1\}^{T} \tag{3}\]

A rectangular batch of \(B\) token sequences has \(T\) stored positions in every row. The integer-array notation below gives its shape. Actual entries must also be valid vocabulary IDs. §2.2 explains how padding creates this layout and records which positions are real:

\[ X \in \mathbb{Z}^{B\times T} \tag{4}\]

Symbols and dimensions

The same letter can denote different objects in different equations. Its local dimensions identify which meaning applies.

Table 1: Recurring quantities and their dimensions.
Symbol Meaning and dimensions Explanation
\(B,T,\lvert V\rvert,D\) Batch size, stored sequence length, vocabulary size, representation width Token batches
\(\mathcal D,N_{\mathrm{ex}}\) Dataset and its number of examples Training pairs
\(F,C,D_{\mathrm{in}},D_{\mathrm{out}},D_{\mathrm{ff}}\) Feature count, class count, layer input and output widths, and feed-forward intermediate width Linear shapes, transformer blocks
\(X\) Feature rows \([B,F]\) or integer token IDs \([B,T]\), as stated locally Batch scores
\(E\) Embedding table \([\lvert V\rvert,D]\); locally, expert count or a fitted limiting-loss constant Embedding lookup, experts, scaling fits
\(\theta,W,b\) All model parameters, a weight matrix, and a bias Linear models
\(z,p,y\) Logits, probabilities, and observed targets Probability outputs
\(L,\widehat R\) Per-example loss and its average over an observed sample Empirical risk, loss reduction
\(\sigma,\phi\) Sigmoid and a locally specified activation function Sigmoid, activations
\(\nabla_\theta L,\eta\) Parameter gradient, with the dimensions of \(\theta\), and learning rate Gradients, updates
\(Q,K,V\) Query, key, and value arrays, commonly \([B,H,T,d_h]\) Attention
\(H,d_h,d_k\) \(H\) is locally a hidden width or head count; \(d_h\) is per-head width and \(d_k\) the query/key width used for score scaling Hidden layers, multiple heads
\(m,v\) First and second optimizer moments, with parameter dimensions Adam
\(\tau\) Positive temperature in a probability transformation Temperature
\(r,\alpha_{\mathrm{LoRA}}\) LoRA rank and scale numerator, giving scale \(\alpha_{\mathrm{LoRA}}/r\) LoRA
\(N_{\mathrm{param}},N_{\mathrm{tok}}\) Parameter count and training-token count in scaling calculations Compute allocation

The vocabulary set is \(V\) and its size is \(|V|\). In attention, \(V\) is an array of value vectors. Local uses of \(C\) include a training corpus in Chapter 3, class count in classification, compute in §19.4, image channels in §22.1, and an NF4 codebook in §20.7. Likewise, \(L\) can denote scalar loss or a layer count in a cache estimate; \(H\) can denote hidden width or head count; and \(E\) can denote an embedding table, expert count, or fitted limiting-loss constant. The letter \(T\) counts sequence positions. Natural logarithms give losses in nats and pair with ordinary exponentiation for perplexity. Base-two logarithms give bits and require base-two exponentiation. §8.5 relates that base choice to the calculation.

The notation \(\mathbb{E}_{z\sim P}[g(z)]\) denotes the expected value of \(g(z)\) for \(z\) drawn from distribution \(P\). §14.1 uses it for noise words drawn from a sampling distribution, and §20.3 uses it for responses drawn from a policy.

Masks and matrix orientation

A token value cannot identify an added padding position when the same value also marks a real end of sequence. Separate control arrays specify which positions each operation may use. The table distinguishes the operations that use them, their values, and links to their explanations.

Table 2: Masks exclude different positions from attention, pooling, or loss calculations.
Object Values and dimensions Consumer and effect
Tokenizer attention_mask \(M^{\mathrm{valid}}\in\{0,1\}^{B\times T}\), with entry \(m_{b,t}=1\) for a retained token and \(0\) for inserted padding Supplies padding visibility to the model. Padding validity
Additive attention mask \(M^{\mathrm{attn}}\), with \(0\) for permitted positions and \(-\infty\) for blocked positions, broadcastable to scores Added before softmax, making blocked attention weights zero. Attention visibility
Loss labels with ignore_index Target IDs with a sentinel such as \(-100\), \([B,T]\); \(N_{\mathrm{tgt}}\) counts included targets Excludes positions from both summed loss and the supervised-position count. Loss exclusion
Pooling mask \(1\) for included vectors and \(0\) otherwise, \([B,T]\) Excludes padding vectors and sets the mean’s denominator. Pooling

Boolean conventions depend on the receiving API: True may mean permitted or blocked. A tokenizer’s validity array therefore cannot be passed to every mask argument without checking its contract. Loss exclusion and pooling also require their own included-position counts. Attention requires a defined policy when no position remains visible. The linked chapters explain these different operations.

For a row-vector batch, mathematical weights \(W\) have shape \([D_{\mathrm{in}},D_{\mathrm{out}}]\). PyTorch stores a linear weight \(A\) with shape \([D_{\mathrm{out}},D_{\mathrm{in}}]\). Thus \(W=A^{\top}\) and the same output can be written \(XW+b\) or \(XA^{\top}+b\). One input written as a column vector instead uses \(y=A x+b\), with the stored matrix’s output-by-input shape. Chapters 12, 15, 17, and 20 use that column-vector convention in some equations. Read each equation’s declared shapes before combining it with a row-batch formula. §5.3 gives the row-batch multiplication and Chapter 23 develops storage details.

The word decoding depends on its object. Tokenizer decoding converts token IDs back to text. Generation decoding selects the next ID from model scores. A transformer decoder computes representations under its visibility rule, after which a generation rule chooses a token. An audio decoder can map sound features toward a waveform. A model checkpoint is a saved model state, while a chapter checkpoint is a question for the reader.