8  Cross-entropy and perplexity

Logarithmic losses compare probabilities with reference answers. An inclusion rule determines their mean and perplexity.

Two classifiers can choose the same category while assigning different probabilities to the recorded answer. A correctness flag hides that difference. Logarithmic loss measures it. The penalty grows as the model assigns less probability to the observed target.

Five numbered panels are arranged in one row. Panel 8.1 turns the probability of one target into negative log loss. Panel 8.2 compares a target distribution with a model distribution using cross-entropy and KL divergence. Panel 8.3 sums and averages three losses. Panel 8.4 subtracts a one-hot target from a three-class prediction to produce the logit gradient. Panel 8.5 shows a two-sequence batch with shifted targets, attention and loss masks, mean loss, and perplexity.
Figure 8.1: Section 8.1 turns the probability assigned to the target into negative log loss. Section 8.2 compares full target and model distributions when the target is uncertain. Section 8.3 explains how sums, means, masks, and weights combine example losses. Section 8.4 shows that the gradient for each logit is the prediction minus the target. Section 8.5 aligns next token targets and masks in a batch, then calculates mean loss and perplexity. The map’s supplied losses (0.22, 1.20, 1.55), averaging 0.99, are separate from the running batch in Section 8.3.

8.1 Target probability and logarithmic loss

For a positive example, probabilities 0.6 and 0.9 both produce a positive decision at threshold 0.5. Zero-one loss gives zero for either correct decision and one for any incorrect decision. It cannot distinguish the two probability estimates. Summing it counts errors, while averaging it gives the error fraction. Away from the decision boundary, a small score change leaves this loss constant, so its ordinary derivative is zero. At the boundary it jumps. Logarithmic loss supplies sensitivity to the assigned probability before the predicted class changes.

The surprisal of an event with probability \(r>0\) is \(-\log r\). A more probable event has a smaller value. Using natural logarithms measures this quantity in nat units. One nat equals \(1/\log2\approx1.443\) bits. For the two positive predictions above, losses are \(-\log0.6\approx0.511\) and \(-\log0.9\approx0.105\).

Negative log likelihood, abbreviated NLL, applies this penalty to an observed answer under the model:

\[ \mathrm{NLL}=-\log P_\theta(y\mid x) \tag{8.1}\]

Here \(x\) is the input, \(y\) is its recorded target, and \(\theta\) is the parameter version that produced the probability. The event can be one class or a whole sequence, depending on the task. For a sequence, the probability chain rule supplies conditional token factors. Their negative logarithm becomes a sum of token losses. The tokens themselves need not be independent.

Maximum-likelihood estimation (MLE) chooses parameters that maximize the probability of the observed data or recorded target answers under the model. For a fixed collection of conditionally independent examples, their joint likelihood is the product of their target probabilities. Taking a logarithm turns that product into a sum, and changing its sign turns maximization into minimization of total NLL. Dividing by the fixed number of examples gives the same optimum.1

This explains why the unsmoothed relative continuation counts in §3.4 fit their observed histories and why training can minimize log loss. Adding a parameter penalty, a term that discourages selected parameter values, or changing the included targets changes the optimization problem. §10.5 develops parameter penalties.

In binary classification, \(p\) is the predicted probability of label 1, and \(1-p\) is the probability of label 0. The target \(y\in\{0,1\}\) selects the corresponding term in binary cross-entropy, abbreviated BCE and also called binary log loss:

\[ L_{\mathrm{BCE}}=-y\log p-(1-y)\log(1-p) \tag{8.2}\]

If \(y=1\), the loss is \(-\log p\). If \(y=0\), it is \(-\log(1-p)\). For several possible classes, cross-entropy weights each class’s negative log probability by its target mass:

\[ L_{\mathrm{CE}}=-\sum_i y_i\log p_i \tag{8.3}\]

The index \(i\) runs over classes, \(p_i\) is the model probability, and \(y_i\) is the target probability for class \(i\). Both vectors are nonnegative and sum to one. For a recorded class index \(c\), let \(e_c\) be its one-hot target vector. Only component \(c\) is one, so the sum collapses to that class’s term:

\[ H(e_c,p)=-\sum_i (e_c)_i\log p_i=-\log p_c \tag{8.4}\]

Here \(p_c\) is the probability assigned to the recorded class. With a distributed target \(y\), several terms can contribute instead.2

A positive target mass assigned zero model probability gives infinite loss in the limiting mathematical expression. Zero target mass contributes zero, using the convention \(0\log0=0\). Direct floating-point evaluation of a logarithm at zero can still produce invalid arithmetic. Stable implementations work from logits without explicitly forming that problematic logarithm.

Example: Target probabilities and penalties

For a positive label, \(p=0.8\) gives \(-\log0.8\approx0.223\) nats. Changing it to 0.1 gives approximately 2.303 nats. The second prediction assigns much less probability to the event that occurred.

For a multiclass target whose probability is 0.25, the loss is \(-\log0.25\approx1.386\) nats. Both NLL and one-hot cross-entropy select that same target-class term.

Conclusion: The penalty depends on the probability of the recorded answer. It distinguishes probability estimates even when thresholding or argmax would leave the selected category unchanged.

In PyTorch, BCEWithLogitsLoss receives raw binary logits and same-shaped floating-point targets. CrossEntropyLoss and torch.nn.functional.cross_entropy receive raw class logits. Hard targets are integer class indices. Probability targets instead have the logits’ shape and must be nonnegative distributions summing to one. PyTorch does not check those probability constraints.3

Later, §11.5 uses this interface in a runnable random-batch update, once the calculation of gradients through the model is established.

8.2 Target uncertainty and distribution mismatch

Even a correct probability model cannot predict a fair coin’s next outcome with certainty. Its average log loss reflects the outcome distribution’s uncertainty. A mismatched prediction adds another contribution.

In this section, \(p\) denotes the target distribution and \(q\) the predicted distribution. This differs from §8.1, where \(p\) denoted a prediction. The target probabilities \(p_i\) weight all averages below. Entropy is the expected surprisal when outcomes and probabilities both come from \(p\):

\[ H(p)=-\sum_i p_i\log p_i \tag{8.5}\]

On a fixed finite set, entropy is zero for a certain outcome and largest for a uniform distribution. It measures uncertainty in that distribution, not errors against individual labels.

Cross-entropy instead uses predicted probabilities in those logarithms: \(H(p,q)=-\sum_i p_i\log q_i\). Subtracting \(H(p)\) gives \(\sum_i p_i(\log p_i-\log q_i)\). Combining each logarithm difference yields Kullback-Leibler (KL) divergence:

\[ \operatorname{KL}(p\mathbin\Vert q)=\sum_i p_i\log\left(\frac{p_i}{q_i}\right) \tag{8.6}\]

Consequently, \(H(p,q)=H(p)+\operatorname{KL}(p\Vert q)\). KL is the extra expected log loss from predicting \(q\) when outcomes follow \(p\). It is nonnegative and zero when the distributions agree. Exchanging \(p\) and \(q\) changes both the weights and the ratios, so KL is asymmetric and is not an ordinary distance.4

For a finite value, \(q_i\) must be positive wherever \(p_i\) is positive. If \(p_i=0\), its contribution is defined as zero.

The nonnegative total follows from \(-\log u\ge 1-u\) for \(u>0\). The function \(u-1-\log u\) has its minimum, zero, at \(u=1\), which gives this inequality.

Apply this to \(u=q_i/p_i\) wherever \(p_i>0\), then multiply by \(p_i\) and sum. The KL sum is at least \(1-\sum_{i:p_i>0}q_i\), which is nonnegative because \(q\) has total mass one. Equality requires matching probabilities, including no extra mass outside the support of \(p\). Individual terms may still be negative. Only their complete sum is guaranteed to be nonnegative.

Example: Uncertainty plus mismatch

Take target distribution \(p=(0.5,0.5)\) and prediction \(q=(0.75,0.25)\). The entropy is \(-0.5\log0.5-0.5\log0.5\approx0.693147\) nats, or approximately 0.693.

Cross-entropy weights the model penalties by the target frequencies. Its terms are \(-0.5\log0.75\approx0.143841\) and \(-0.5\log0.25\approx0.693147\). Their sum is approximately 0.836988 nats.

The KL terms are \(0.5\log(0.5/0.75)\approx-0.202733\) and \(0.5\log(0.5/0.25)\approx0.346574\). Their sum is approximately 0.143841 nats, or 0.144. A single term can be negative even though the complete divergence is nonnegative.

Conclusion: The expected loss splits into \(0.836988\approx0.693147+0.143841\). Matching the prediction to \(p\) removes the mismatch contribution while leaving the fair distribution’s uncertainty.

Training observes examples rather than an exact target distribution. The chosen batch reduction specifies how their individual penalties contribute to an update.

8.3 Sums, means, and the running batch

Two batches can contain the same types of error but different numbers of examples. Summing their losses makes batch size affect the objective’s scale. A mean divides by the contributing count, allowing a different comparison.

A mean combines the per-example losses and divides by the number of contributing examples, using §1.3’s empirical-risk calculation. Every loss in a forward batch uses the same parameter values.

For \(n>0\) contributing examples, the mean gradient is the sum gradient divided by \(n\). With the plain gradient-descent update from §5.4 at the same learning rate, the parameter change from the summed loss is \(n\) times that from the mean loss. For \(n=1\) or a zero aggregate gradient, the two steps coincide. The adaptive update rules introduced later in §10.3 and §10.4 require a separate comparison. Masks and example weights introduce their own inclusion and divisor rules. A reported loss is incomplete without its reduction.

Example: Mean loss for the running batch

Keep the inputs and parameters from §5.3: \(X=[[1,0],[0,1],[1,1]]\), \(w=[0.8,-0.3]\), \(b=0.1\), and targets \(y=[1,0,1]\). The logits are \([0.9,-0.2,0.6]\). Section 6.2 calculated their probabilities, approximately \([0.710950,0.450166,0.645656]\).

Rows 1 and 3 have positive targets, so their penalties use \(-\log p\). Row 2 has a negative target and uses \(-\log(1-p)\). Keeping precision through these calculations gives losses approximately \([0.341154,0.598139,0.437488]\).

Their sum is approximately 1.376781 and their mean is approximately 0.458927, or 0.459 at three decimals. Chapter 1’s supplied rounded penalties give a slightly different mean, which also rounds to 0.459.

Conclusion: The target determines which class probability each row contributes. Averaging those three penalties gives the objective whose gradient is used in §11.5.

A recorded language-model token contributes the same categorical penalty:

\[ L=-\log P_\theta(x_t\mid x_{<t}) \tag{8.7}\]

The observed token is \(x_t\), its available prefix is \(x_{<t}\), and \(\theta\) identifies the fixed parameters used in this forward calculation. Section 8.5 determines which positions belong in a sequence mean. First, the local derivative explains how one included penalty depends on its logits.

8.4 Gradients with respect to logits

A target probability determines the loss, but a parameter update needs a sensitivity to the score that produced it. The derivative chain rule from §5.4 connects these quantities. In the binary case, the path is logit \(z\), probability \(p=\sigma(z)\), then loss \(L\).

With the binary target \(y\) fixed, differentiating BCE gives \(dL/dp=-y/p+(1-y)/(1-p)\). The second term is positive because differentiating \(1-p\) introduces another minus sign. Multiplication by sigmoid’s derivative \(dp/dz=p(1-p)\) gives \(-y(1-p)+(1-y)p\), which simplifies to

\[ \frac{dL}{dz}=p-y \tag{8.8}\]

Here \(p\) is the predicted positive-class probability, \(y\in\{0,1\}\), and \(z\) is the scalar logit. For \(p=0.8\), the derivative is \(-0.2\) when \(y=1\) and \(0.8\) when \(y=0\). A small direct descent step on \(z\) raises the positive example’s score and lowers the negative example’s score.

For multiple classes, hold a target distribution \(y\) fixed with \(\sum_i y_i=1\). Let \(S=\sum_i e^{z_i}\). Since \(\log p_i=z_i-\log S\), cross-entropy becomes \(L=\log S-\sum_i y_i z_i\). Changing coordinate \(z_j\) changes \(S\) at rate \(e^{z_j}\). The derivative chain rule therefore gives \(\partial L/\partial z_j=e^{z_j}/S-y_j=p_j-y_j\).

Example: A direct logit step

Take probabilities \((0.7,0.2,0.1)\) and target class 0, so \(y=(1,0,0)\). The logit gradient is \((-0.3,0.2,0.1)\). One compatible logit vector is \(z=(\log0.7,\log0.2,\log0.1)\).

A direct step at rate 0.1 adds \((0.03,-0.02,-0.01)\) to that vector. Recomputing softmax gives target probability approximately 0.709705. Its loss falls from approximately 0.356675 to 0.342905 nats.

Conclusion: Subtracting the local gradient raises the target logit relative to its competitors in this direct-score example. Actual model training updates shared parameters, whose effects on scores must also be differentiated.

A confident wrong prediction can have a large loss even though its logit gradient remains between minus one and one. Both depend on the recorded target as well as the predicted probability.

Score-to-probability-to-loss diagram with a small unlabeled curve and accompanying gradient text.
Figure 8.2: The diagram connects a score, a probability, and a binary cross-entropy penalty.

For \(y=1\) and \(p=0.01\), the loss is \(-\log(0.01)\approx4.605170\), but the logit derivative is \(p-y=-0.99\). A supplied input coordinate of 2 multiplies that sensitivity to give a weight derivative of \(-1.98\). If the target were 0, the same probability would instead give logit derivative \(0.01\). The target is necessary to interpret either penalty or gradient.

Parameter gradients can be larger than the local logit derivative because input values and intermediate derivatives also contribute.

For a batch mean, each included example’s local logit derivative gains the divisor \(1/n\). Parameter gradients then combine these sensitivities with derivatives of the scores with respect to the parameters. Section 11.5 performs that complete calculation. The local descent argument does not guarantee that an arbitrary finite parameter step reduces the objective.

8.5 Aligned targets, loss masks, and perplexity

A padded sequence contains stored positions that are not observed next-token answers. Including them in loss would train or evaluate a different task. A loss mask identifies which target comparisons contribute. It is separate from the attention mask in §2.2, which controls which inputs a prediction may use.

Let \(N_{\mathrm{tgt}}\) count the included targets across the chosen sequence or batch. Each token loss is the negative log probability of one recorded next token. In a batch, the index \(t\) ranges over row-position pairs, and each prefix stays within its own sequence. Their unweighted mean is

\[ \mathrm{CE}=-\frac{1}{N_{\mathrm{tgt}}}\sum_{t\in\mathcal S_{\mathrm{tgt}}}\log P_\theta(x_t\mid x_{<t}) \tag{8.9}\]

The sum includes only selected target positions, and \(N_{\mathrm{tgt}}\) must be positive. An empty included set has no defined mean. A training program must reject or skip that batch before division, rather than report a zero loss as successful prediction.

For one unpadded sequence of \(T>1\) tokens with every following-token target included, the general mean has \(N_{\mathrm{tgt}}=T-1\). With the formula’s target-position indexing, the prediction for \(x_t\) uses \(x_{<t}\). Padding or another target-exclusion policy changes the included count. The denominator follows that count rather than the stored sequence width.

Example: The padded batch from §2.2

The input IDs from §2.2 are [[1,2,3,4],[1,5,4,0]], with validity mask [[1,1,1,1],[1,1,1,0]]. The six-entry vocabulary assigns BOS to 1, EOS to 4, and PAD to 0. Copying the IDs and ignoring padding gives labels [[1,2,3,4],[1,5,4,-100]].

Manual slicing pairs logits at positions 0 through 2 with labels at positions 1 through 3 before the loss call. The compared target rows are therefore [2,3,4] and [5,4,-100]. Five positions contribute. Both real EOS targets remain included, while the added PAD target is excluded.

Suppose those five target probabilities, in row order, are \((0.7,0.5,0.8,0.6,0.9)\). The negative log probabilities sum to approximately 1.889152 and give mean 0.377830. Dividing by six would incorrectly count the ignored comparison.

Conclusion: Seven retained input positions yield five next-token targets after alignment. Input validity and target inclusion count different objects. Ignoring a target removes its loss and its direct logit-gradient contribution.

The following runnable PyTorch check constructs supplied probabilities with those target values. Taking their logarithms creates compatible logits. It does not run a language model. The class axis has width 6, so flattening produces [B*(T-1),6] logits and matching integer targets.

Code example: Equivalent next-token alignments with five included targets

import torch
import torch.nn.functional as F

input_ids = torch.tensor([[1, 2, 3, 4], [1, 5, 4, 0]])
valid = torch.tensor([[1, 1, 1, 1], [1, 1, 1, 0]])
labels = input_ids.clone()
labels[valid == 0] = -100

# Supplied probabilities isolate the loss calculation from a model.
probabilities = torch.full((2, 4, 6), 1.0 / 6)
for row, pos, target, p in [
    (0, 0, 2, 0.7), (0, 1, 3, 0.5), (0, 2, 4, 0.8),
    (1, 0, 5, 0.6), (1, 1, 4, 0.9),
]:
    probabilities[row, pos] = (1 - p) / 5
    probabilities[row, pos, target] = p
logits = probabilities.log()

def included_mean(scores, targets):
    count = (targets != -100).sum().item()
    if count == 0:
        raise ValueError("No included next-token targets")
    return F.cross_entropy(scores, targets, ignore_index=-100)

shifted_targets = labels[:, 1:].reshape(-1)
loss = included_mean(logits[:, :-1].reshape(-1, 6), shifted_targets)
internal_targets = F.pad(labels, (0, 1), value=-100)[:, 1:]
equivalent = included_mean(logits.reshape(-1, 6), internal_targets.reshape(-1))
assert (shifted_targets != -100).sum().item() == 5
assert torch.allclose(loss, equivalent)
expected = -torch.tensor([0.7, 0.5, 0.8, 0.6, 0.9]).log().mean()
assert torch.allclose(loss, expected)
print(loss.item())

ignore_index=-100 applies to integer class targets and removes the ignored position from this unweighted mean.5 It is not a vocabulary entry. The assertions check the five included targets and compare direct alignment with the equivalent right-padded label alignment. The explicit error branch rejects all-ignored labels because the mean would have a zero divisor.

The construction assumes right padding. With left padding, the first real token can follow a padded input position, so target validity alone is insufficient. A next-token comparison needs both a valid predicting position and a valid target under the chosen objective. A real BOS input can provide the first valid prediction position.

GPT-2’s model-level loss in transformers 5.10.2 accepts copied, unshifted labels. Its loss helper pads the labels on the right with the ignore value, then takes labels from position 1 onward. This pairs each logit with the following token and ignores the final logit. The manual slice above excludes that final logit instead. Both constructions compare the same targets. Supplying already-shifted labels to an interface that shifts internally would shift twice.6

A response-only training objective can also ignore prompt targets while leaving the prompt visible as context. Suppose the first row’s prompt consists of BOS and token 2. Ignoring target 2 then leaves four included targets across the batch. An objective that learns every recorded continuation would retain that prompt target. Prompt masking is therefore an objective choice, not a universal language-model rule.

The smoothed count example in §3.4 provides another loss check. For an evaluation target bird after the, its probability \(1/6\) gives \(-\log(1/6)=\log6\approx1.792\) nats. The unsmoothed zero probability would give infinite loss. Smoothing makes this comparison finite without supplying evidence that the continuation occurred in the training corpus.

Mean log loss is useful for optimization, but a value in nats does not directly show the typical probability assigned to a recorded token. Perplexity converts that mean to a reciprocal probability scale:

\[ \mathrm{PPL}=\exp(\mathrm{CE}) \tag{8.10}\]

Write the included target probabilities as \(r_1,\ldots,r_{N_{\mathrm{tgt}}}\). Then \(\exp(-N_{\mathrm{tgt}}^{-1}\sum_i\log r_i)=(\prod_i r_i)^{-1/N_{\mathrm{tgt}}}\). The geometric mean target probability is \((\prod_i r_i)^{1/N_{\mathrm{tgt}}}\), so perplexity is its reciprocal. This is not generally the reciprocal arithmetic mean probability.7

For correct-token probabilities 0.7 and 0.5, mean loss is approximately 0.524911. Their geometric mean is \(\sqrt{0.35}\approx0.591608\), giving perplexity approximately 1.690309. The arithmetic mean is 0.6 and would give a different reciprocal.

Lecture page with perplexity written as sequence probability raised to the negative reciprocal of token count.
Figure 8.3: Perplexity is the reciprocal geometric mean of the included target probabilities. The sequence-probability expression uses N for their count.

A uniform distribution over \(|V|\) vocabulary choices has perplexity \(|V|\), which motivates an effective-choice-count interpretation. A nonuniform model need not have that literal number of available tokens.

These results establish how perplexity summarizes assigned target probabilities. Comparisons need compatible tokenization, data, context, and inclusion policies. With base-2 logarithms, exponentiation uses base 2 instead. Neither low perplexity nor high likelihood establishes factual correctness. Held-out evaluation in Chapter 9 measures prediction performance on separate inputs.

Chapter checkpoint

Why can two correct classifications have different log losses? Which divisor applies to shifted targets [2,3,4] and [5,4,-100]? What happens if every target is ignored?

Answer: Log loss uses the probability assigned to the recorded answer, which a correctness flag discards. Five targets contribute to this unweighted mean. An all-ignored batch has no defined mean, so the program must reject or skip it before averaging.

For a positive label with \(p=0.8\), what is the logit derivative? Does a lower training perplexity prove that generated answers are true?

Answer: The derivative is \(0.8-1=-0.2\), before any batch-mean divisor. Lower training perplexity establishes a larger geometric mean probability for those fitted targets. It supplies no independent factuality test.


  1. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. Third edition draft, released August 19, 2026, §§4.5 and 4.8. Independence of examples differs from the conditional factorization of dependent sequence tokens.↩︎

  2. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. Third edition draft, released August 19, 2026, §4.8.↩︎

  3. PyTorch contributors. (2026). CrossEntropyLoss. PyTorch 2.12 documentation. For unweighted class-index loss, the mean excludes ignored targets. Probability targets use a different input contract.↩︎

  4. Zhang, A., Lipton, Z. C., Li, M., & Smola, A. J. (2023). Dive into Deep Learning. Cambridge University Press, online version 1.0.3, §§22.11.2, 22.11.4 and 22.11.5.1. These give entropy, KL divergence and the cross-entropy identity. This book uses natural logarithms throughout this calculation.↩︎

  5. PyTorch contributors. (2026). CrossEntropyLoss. PyTorch 2.12 documentation. For unweighted class-index loss, the mean excludes ignored targets. Probability targets use a different input contract.↩︎

  6. Hugging Face. (2026). Transformers 5.10.2: Causal language-model loss and GPT-2 forward. This account uses the default unweighted mean without a supplied batch-item divisor.↩︎

  7. Jurafsky, D., & Martin, J. H. (2026). Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. Third edition draft, released August 19, 2026, §§3.3 and 3.7. The logarithm and exponent bases must agree.↩︎