24  Computational labs: Checks, experiments, and evidence

Training and inference experiments declare their inputs, procedures, and comparison rules so results can support the intended claims.

A correct tensor shape does not establish correct gradients, useful predictions, or faster execution. Local calculations isolate those mechanics. A full experiment additionally needs its data, fitting procedure, and held-out results. Keeping those evidence types separate makes a failed comparison easier to diagnose.

The execution reference records package requirements, notebook downloads and historical settings. Each lab below keeps its own task, local calculation and full experiment separate, so a passing assertion is tied to the result it actually checks.

Nine compact panels show an environment and data record, logistic regression and optimizer comparison, tokenizer and embedding comparisons, a controlled attention input change, adapter checks and audience evaluation, recurrent character generation with feedback, count-model fitting and held-out scoring, separate captioning, visual question, and retrieval tasks, and a final evidence record joining a baseline, one change, results, errors, costs, and limits.
Figure 24.1: Seven lab sections connect local calculations to classification, attention, audience adaptation, recurrent generation, count-model fitting and vision-language comparisons. The final classifier result separates an executed parameter update from unmeasured held-out quality.

24.1 Logistic regression and controlled optimizer comparisons

A classifier experiment needs an update that matches its loss before optimizer settings can be compared. The local check isolates two feature rows. The logistic regression and optimizer lab then extends that calculation to SST-2 sentiment data and controlled training comparisons.1

For the affine scores \(z=Xw+b\) from §5.3, \(X\) has shape \([B,F]\), \(w\) has shape \([F]\), and scalar \(b\) is shared. The local calculation uses \(X=I_2\), labels \((1,0)\), weights \((0.2,-0.1)\) and bias zero. Thus \(B=2,F=2\), and the two logits are \((0.2,-0.1)\).

The mean binary log loss uses §8.1’s positive probability \(p_i=\sigma(z_i)\) and the corresponding negative probability \(1-p_i\). The divisor counts the two examples. A stable library calculation receives logits and floating-point labels in the same shape.

The runnable code checks the logits, loss and gradients against their numerical values, then applies one plain-SGD step and checks the recomputed loss.

Code example: Numerical logistic gradients, one SGD update and recomputed loss

import torch

# Rows are examples; columns are features.
X = torch.tensor([[1.0, 0.0], [0.0, 1.0]])
y = torch.tensor([1.0, 0.0])
w = torch.tensor([0.2, -0.1], requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)

# The affine score is the logistic-regression logit.
logits = X @ w + b
loss = torch.nn.functional.binary_cross_entropy_with_logits(logits, y)
loss.backward()
assert logits.shape == (2,)
assert w.grad.shape == w.shape
assert torch.allclose(logits, torch.tensor([0.2, -0.1]), atol=1e-7)
assert torch.allclose(loss.detach(), torch.tensor(0.621267796), atol=1e-6)
assert torch.allclose(w.grad, torch.tensor([-0.225083023, 0.237510398]), atol=1e-6)
assert torch.allclose(b.grad, torch.tensor(0.012427375), atol=1e-6)

with torch.no_grad():
    w -= 0.1 * w.grad
    b -= 0.1 * b.grad
assert torch.allclose(w, torch.tensor([0.222508302, -0.123751040]), atol=1e-6)
assert torch.allclose(b, torch.tensor(-0.0012427375), atol=1e-7)
new_loss = torch.nn.functional.binary_cross_entropy_with_logits(X @ w + b, y)
assert torch.allclose(new_loss.detach(), torch.tensor(0.610611796), atol=1e-6)
assert new_loss < loss

Its logits are \((0.2,-0.1)\). Their probabilities are approximately \((0.549834,0.475021)\), and the mean loss is approximately 0.621268. The manual mean weight gradient is \((p-y)/2\approx(-0.225083,0.237510)\) because \(X\) is the identity. The bias gradient sums those entries, giving approximately 0.012427.

At rate 0.1, the checked SGD step gives weights approximately \((0.222508,-0.123751)\) and bias \(-0.001243\). Recomputing from those parameters gives mean loss approximately 0.610612 nats, below the initial 0.621268. The assertions check these values within the declared floating-point tolerance.

Conclusion: The mean divisor and gradient signs lead to a checked first update and a lower loss on these same two rows. Convergence and sentiment quality require the separate fitting and held-out comparisons below.

The synthetic experiment fits 1,000 two-feature rows, rather than the two rows above. Its record needs a loss history and predictions recomputed from the final parameters.

Experiment: Synthetic two-feature fitting

The gradient notebook supplies an intermediate experiment before the text classifier. It generates 1,000 two-dimensional examples around centers [-3,-3] and [0,0], with cluster_std=1.25. Convert features to float32 and labels to a [1000,1] column. A [2,1] weight column and shared bias produce one probability per row. First compare the prediction surfaces for different weights and biases. Set both weight entries to 1 while retaining the current bias, then set the bias to 3. Compare equal weight entries of 0.33 with bias 1 against equal weight entries of 5 with bias 15. Retain each prediction surface beside the reference labels. Restore parameters that record gradients before fitting. Record the data-generation and parameter-initialization seeds.

Use up to 1,000 full-data updates at rate 0.1. Store the loss before each update, change weights and bias under torch.no_grad(), clear both gradient buffers, and record every twentieth loop index. With stopping threshold 1e-5, first require finite losses and at least two recorded values. Stop only when the nonnegative improvement satisfies \(0\leq L_{t-1}-L_t<10^{-5}\). Record an increase or nonfinite value as a separate diagnostic event. Calculate the stable loss from logits as in §8.1 rather than taking logs of rounded sigmoid probabilities.

Plot the loss history against the update index, then recalculate predictions from the final parameters before the classification report and accuracy calculation. Use one declared threshold convention throughout, including equality at 0.5. The quantity exp(-mean log-loss) is the geometric mean of the reference-class probabilities. Their arithmetic mean is at least as large, with equality when all those probabilities are equal. State which summary a reported value represents.

Running that procedure would add a loss history and final-parameter metrics. Sentiment classification additionally needs sparse text features and validation choices.

The sentiment experiment compares learning rates and batch sizes for a sparse logistic classifier. Its five rates and three batch sizes require 15 runs under a common budget, with validation loss and failures retained.

Experiment: Sentiment settings and sparse features

The larger comparison uses SetFit/sst2 at the revision in the dataset reference, with its training and validation splits. Apply one recorded cleaning rule to both splits, fit the vocabulary on training text only, and transform validation text with that saved vocabulary. The notebook’s rule lowercases text, replaces hyphens with spaces, retains ASCII letters, digits, whitespace, and . , ! ?, then collapses repeated whitespace. Because splitting occurs on whitespace, good! and good remain different tokens. Record split sizes and class proportions before fitting. A sparse bag-of-words representation can retain up to 10,000 frequent training tokens. Corpus frequency determines which entries enter the vocabulary, while each document stores its own counts.

The supplied notebook proposes clamping probabilities at \(\epsilon=10^{-15}\) and \(1-\epsilon\). In float32, the upper value rounds to one, so this clamp still permits \(\log0\). Calculate binary loss from logits with a stable expression, such as \(\log(1+e^z)-yz\) evaluated through a stable log-sum operation, and compare against §8.1. Keep the full count-feature matrix sparse. A mostly zero dense array still allocates its zero entries.

The one-output sigmoid formulation and the two-class softmax formulation should agree on their binary decisions when they use corresponding parameters and the same threshold convention. Compare both tensor calculations with stable library losses and derivatives. A decision rule such as p >= 0.5 affects reported classes, while the training objective still uses probabilities or logits. The comparison should expose where a naive logarithm or exponential fails and why subtracting the largest logit preserves the softmax probabilities.

Initialization and array orientation are separate experimental choices. Record whether weights begin at zero, from a small random distribution, or from a supplied tensor, together with their seed, type, and shape. A [F,1] weight column and an [F] weight vector return differently shaped logits, so outputs and targets must follow the chosen convention. Mini-batch fitting should shuffle training rows, keep the full feature matrix sparse, and convert only a selected batch when a dense tensor is required. Preserve independent parameter snapshots and per-epoch training and validation measures. Report prediction loss separately from an objective that includes a penalty.

A controlled search crosses learning rates [0.01, 0.03, 0.1, 0.3, 1.0] with batch sizes [50, 100, 200]. The resulting 15 runs need a common update budget, metric, and validation rule, with failures retained rather than removed. A matched table or heatmap can then show which settings converge, diverge, or remain unstable without implying that the best observed setting is universal.

The L1 comparison keeps rate and batch size fixed while varying penalty strength across [0, 1e-4, 1e-3, 1e-2, 1e-1]. Count coefficients under one declared tolerance and retain validation outcomes beside training measures. Ordinary subgradient SGD can cross zero without producing exact zeros, so coefficient sparsity must be measured rather than inferred from the presence of an L1 term. A separate proximal-L1 comparison uses the soft-thresholding rule from §10.5 and answers a different question about exact zeros.

Its result is a matched comparison of settings, with prediction loss distinguished from any parameter penalty. No SST-2 outcome is supplied by the two-row local check.

Optimizer-surface comparisons isolate update behavior without a text representation. Their starting points, update budgets and plotted scales must match.

Experiment: Bowl and six-hump camel surfaces

Optimizer behavior can be isolated on the bowl \(x^2+4y^2\) and the six-hump camel function \(f(x,y)=(4-2.1x^2+x^4/3)x^2+xy+(-4+4y^2)y^2\). The second surface has two global minima near \((0.0898,-0.7126)\) and \((-0.0898,0.7126)\), with value about \(-1.0316\), as well as other local minima. GD, Momentum, AdaGrad, and Adam should begin from the same declared point within a comparison and use matched budgets and display axes. Their parameter paths, objective histories, oscillation, and local-minimum behavior show why the observed results differ, without supporting a universal winner.

Match the coordinate-grid axes to the objective function’s indexing. The camel surface can fall below \(-1\), where log1p is undefined, so use raw contours or a display transformation defined over the whole plotted range. Pair each stored loss with the parameters that produced it. When recording post-update parameters, recompute their loss or label the stored loss as pre-update. Sample row indices from \(0\) through \(n-1\) with the upper bound \(n\) excluded.

The recorded paths and objective histories compare update behavior on the two supplied functions. They do not establish a ranking across other tasks.

Use validation evidence to compare settings. Keep any final test separate from those decisions. The same gradient and evidence checks can now support a comparison of text representations.

24.2 Tokenizer, embedding, and classifier comparisons

A better nearest-neighbor display need not improve news classification. The AG News classification lab compares representations through a classifier while retaining the official test set for final evaluation. For this lab, use fancyzhx/ag_news and pass the revision in the dataset reference to load_dataset. This snapshot has 120,000 training records and 7,600 test records. Each record has a text string and a label: 0 for World, 1 for Sports, 2 for Business, or 3 for Sci/Tech.2

The runnable local check starts with two equal-length, already supplied token-vector sequences. It averages each sequence, then applies a random MLP to four class logits.

Code example: Pooled token vectors and four-class MLP logits

import torch
import torch.nn as nn

# A real run obtains these token vectors from the trained Word2Vec model.
token_vectors = torch.tensor([
    [[1.0, 0.0, 0.5], [0.5, 1.0, 0.0]],
    [[0.0, 1.0, 0.5], [1.0, 0.5, 0.0]],
])
article_vectors = token_vectors.mean(dim=1)
classifier = nn.Sequential(nn.Linear(3, 8), nn.ReLU(), nn.Linear(8, 4))
labels = torch.tensor([0, 2])

logits = classifier(article_vectors)
loss = nn.CrossEntropyLoss()(logits, labels)
assert article_vectors.shape == (2, 3)
assert logits.shape == (2, 4)
print(loss.item())

The document vectors are [[0.75,0.5,0.25],[0.5,0.75,0.25]], with shape [2,3]. The logits have shape [2,4], and loss is scalar. Random head weights make its value variable. There is no padding here. Unequal sequences need masked pooling or another justified aggregation policy.

Conclusion: The check joins two supplied document vectors to a class loss. It does not fit embeddings or evaluate AG News. The full experiment compares fitted tokenizers and vectors using one classifier and a fixed evaluation boundary.

The full comparison uses training data for fitting and development and the official test only after choices are fixed. Its result needs final accuracy, a labeled confusion matrix and inspectable errors, not just a neighbor plot.

An attribution measures how a selected model score changes under a specified input comparison.

Experiment: AG News representations and classification

Keep AG News’s official test split untouched and form any validation subset from training data. Inspect the text and label fields and record split sizes before fitting. Preprocessing, tokenizers, and word vectors use training text only. The four reference categories remain World, Sports, Business, and Sci/Tech under the dataset’s recorded integer mapping.

BPE and WordPiece can use the same example vocabulary budget, such as 10,000 entries, while producing different segmentations and unknown-token behavior. Compare their token sequences and lengths on the same sentences, including unfamiliar words, before choosing one for the classifier. The fitted tokenizer and its preprocessing must be saved together so later comparisons use the same mapping.

The tokenizers library supplies complete vocabulary trainers for the API comparison. The following offline example uses version 0.22.2, a three-sentence corpus, Whitespace pre-tokenization, and a BPE vocabulary limit of 50. It adds a WordPiece trainer on the same text. These small settings demonstrate the API before the full AG News experiment.

Code example: Training BPE and WordPiece on the same small corpus

import json
from tokenizers import Tokenizer
from tokenizers.models import BPE, WordPiece
from tokenizers.trainers import BpeTrainer, WordPieceTrainer
from tokenizers.pre_tokenizers import Whitespace
corpus = ["I love natural language processing", "Tokenization is important", "BPE builds subword units"]
for name, model, trainer in [
    ("BPE", BPE(unk_token="[UNK]"), BpeTrainer(vocab_size=50, special_tokens=["[UNK]"], show_progress=False)),
    ("WordPiece", WordPiece(unk_token="[UNK]"), WordPieceTrainer(vocab_size=50, special_tokens=["[UNK]"], show_progress=False)),
]:
    tokenizer = Tokenizer(model)
    tokenizer.pre_tokenizer = Whitespace()
    tokenizer.train_from_iterator(corpus, trainer)
    print(json.dumps({"model": name, "vocab_size": tokenizer.get_vocab_size(), "unk_id": tokenizer.token_to_id("[UNK]"), "unhappiness": tokenizer.encode("unhappiness").tokens, "training_sentence": tokenizer.encode(corpus[0]).tokens}))

In version 0.22.2, WordPieceTrainer delegates vocabulary training to BpeTrainer, whose merge priority uses pair counts, then converts the vocabulary to a WordPiece model. Its training rule therefore differs from the illustrative normalized pair score in §3.2. WordPiece encoding still follows its model’s matching and unknown-token rules.34

Tokenizer combines a model with preprocessing, and train_from_iterator fits its vocabulary from the supplied strings. Each trainer reserves [UNK]. The printed records identify the fitted vocabulary size, unknown-token ID, and encoded pieces. In this run both vocabularies contain 50 entries and assign ID 0 to [UNK].

The test word unhappiness contains h, which is absent from this training corpus. BPE retains the known pieces around an unknown marker. WordPiece returns one [UNK] for the whole word under its policy. In twelve runs of this code on the same strings, BPE returned the same segmentation of the first training sentence in every run, while WordPiece returned five distinct segmentations. Saving each fitted tokenizer with tokenizer.save(path) retains its vocabulary, model rules, and preprocessing for later comparisons.

Conclusion: Both trainers produce reusable tokenizers from the same training text, but their unknown-word handling differs. A small vocabulary trace establishes how to run the comparison, while the full experiment measures whether the representation helps classification.

Skip-gram and CBOW can then be compared at widths 50, 100, and 200 using the same tokenized training text. Record the actual learned vocabulary because a tokenizer entry may be excluded when its embedding-training frequency is below min_count. Neighbor queries such as movie, sport, and market can expose local vector behavior, including a query absent from the learned vocabulary. They do not establish which representation improves classification. When classification is the selection criterion, compare the candidate embeddings on development data under the same classifier and computation budget.

Mean, sum, max, or another declared pooling rule converts a variable-length token sequence into a fixed-width document vector. The rule must specify missing vectors and empty inputs. A multilayer classifier can consume the pooled vector, but the pooling discards some or all word-order information. Width, activation, optimizer, rate, batch size, and training duration therefore belong in the experiment record rather than being inferred from the representation alone.

After development choices are fixed, evaluate once on the official test split. Retain accuracy, a labeled confusion matrix, and error records containing the text, reference category, and prediction. An attribution method can support diagnosis if its scope is stated, but the prediction and held-out evidence remain valid even when no attribution method is added.

A batch loss is commonly already a mean. To report an epoch mean, multiply each batch mean by its included example count, sum those products, and divide by the total count. Averaging batch means equally gives a shorter final batch too much weight. Training-batch losses also come from changing parameter versions. To measure one checkpoint’s performance, evaluate that fixed checkpoint over the relevant split with gradients disabled and evaluation mode enabled.

One possible method keeps the classifier fixed, records its probability for a selected class, then removes one token and recomputes the representation and the same class probability. A drop from 0.80 to 0.55 gives a contribution of 0.25 under that removal. Repeat separately for each position. Removing a token also changes a mean-pooling denominator, so the comparison measures the whole removal operation. Interactions mean these contributions need not add to the original score, and they do not establish a real-world causal effect.

Retain training histories and the representation settings beside downstream results. Source-vector arithmetic and projection plots from Chapter 14 remain diagnostic evidence, not substitutes for classification evaluation.

24.3 Attention under a controlled input change

Changing a word while redrawing every input vector would confound a claimed word-specific effect. A controlled attention comparison uses the same representation mapping and projection matrices, then changes only the intended input.

The attention and fine-tuning notebook uses the cat sat on the mat at width 16 for its example attention calculation, then replaces cat with dog to compare the outputs. These settings illustrate the experiment. The attention calculation must still work for other inputs and widths.

The calculation uses the scaled, masked attention from §16.1. The local arrays below fix projections and causal visibility, allowing the effect of a chosen input change to be tested without changing the representation rule.

The runnable local code supplies projected arrays directly and applies a causal mask. It does not construct representations of actual words.

Code example: Scaled attention values and row-normalization checks

import torch

# Two query positions, two key positions, and scalar values.
Q = torch.tensor([[1.0, 0.0], [0.0, 1.0]])
K = torch.tensor([[1.0, 0.0], [0.0, 1.0]])
V = torch.tensor([[2.0], [6.0]])
scale = Q.shape[-1] ** 0.5
scores = Q @ K.T / scale
causal_mask = torch.triu(torch.full_like(scores, float("-inf")), diagonal=1)
masked_scores = scores + causal_mask
weights = torch.softmax(masked_scores, dim=-1)
output = weights @ V
assert scores.shape == (2, 2)
assert output.shape == (2, 1)
assert weights[0, 1].item() == 0.0
assert torch.allclose(weights.sum(dim=-1), torch.ones(2))

The unmasked score matrix is diagonal with entries \(1/\sqrt2\approx0.707107\). Its masked first row has only the first key available, giving weights \((1,0)\) and output 2. The second row gives weights approximately \((0.330238,0.669762)\) and output 4.679046. These values complement the code’s shape, masked-zero, and row-sum checks.

Conclusion: The first query excludes the future value, while the second mixes both. Row normalization applies only when at least one allowed finite score exists. A fully blocked row needs rejection, skipping, or a declared handling policy before normalization.

Experiment: One-word change with fixed projections

The trace retains the token representations and the query, key, and value projection matrices. It records unmasked scores, the mask, masked scores, normalized weights, and output vectors, with query rows and key columns aligned to token positions. Selected values should agree with an independent manual or trusted-library calculation before the heatmap is interpreted.

A controlled input change replaces one content word while keeping the representation rule and every projection fixed. Comparing the changed token’s projected vectors, the affected score row and column, and the downstream outputs identifies the numerical effect. A linguistic explanation remains a hypothesis unless another controlled comparison supports it.

Multiple heads require independently projected queries, keys, and values. Comparing at least two such heads under the same input change can show different weight patterns. Repeating one projection under two labels would only duplicate the same calculation. The recorded arrays can answer whether the word change affected the weights and whether the heads differ, but a heatmap alone does not prove why the model produced a prediction.

This task isolates representation changes. Adapter training additionally needs evidence of which parameters changed and whether held-out answers improved.

24.4 Audience adaptation with two model families

An adapter parameter count does not establish that an answer suits its intended audience. The fine-tuning notebook requires response-level comparisons for both a causal model and an encoder-decoder model. Their target layouts and adapter targets differ.

The LoRA initialization check verifies that the two factors contain eight trainable entries, initially add zero, and have first gradients of 7 in \(B\) and 0 in \(A\). The quantization check verifies one affine reconstruction. Neither loads or trains the two model families in this experiment.

This experiment compares base and adapted answers for child, student and expert audiences. Each level needs checked references and separate held-out outputs. A successful run must also show which model path executed and its training/resource cost.

Experiment: Audience answers with two model families

A Dolly 15K subset provides questions whose answers can be adapted for child, student, and expert audiences. Preserve the dataset revision and the rule used to select rows. The source notebook’s first-5,000-row selection and its keyword filter are examples of bounded sampling, not universal requirements. Inspect the resulting fields and row counts, including an empty selection, before generating training records.

Each selected question needs three answers that differ by audience rather than length alone. A child answer uses simple, intuitive language. A student answer adds moderate technical detail, while an expert answer is precise and specific to the subject. The source answer must be checked for correctness and suitability before it is rewritten. A separately reviewed held-out set should cover all three levels; the source notebook’s budget of 20 total cases is an inspection set, not a dependable population estimate.

Whatever storage format is used, validate the question, audience level, all answer fields, serialization, reload, and formatting on a small batch before fitting. Held-out questions stay outside training and development choices. A causal model can place question, audience level, and answer in one sequence with a response-target mask. An encoder-decoder model instead places the question and level in the source and the answer in the target. Padding and target alignment must follow the selected model interface.

The comparison uses HuggingFaceTB/SmolLM2-360M-Instruct and google/flan-t5-small as two model families. Four-bit loading and LoRA require separate hardware, numerical-format, adapter-target, and trainable-parameter checks for each architecture. A causal q_proj/v_proj fragment is not an encoder-decoder configuration. Unsupported paths remain unexecuted evidence rather than implied results.

Compare each base and adapted model on the same held-out questions and recorded generation settings. Keep the question, requested audience, reference answer, base output, adapted output, audience fit, clarity, relevance, and source faithfulness together. Apply the hallucination and source-faithfulness criteria from §9.4, separating unsupported or contradictory claims from formatting errors and incomplete answers. Interpret observed changes beside training cost, stored base and adapter payloads, peak memory, rank, and the distinction between prompting, LoRA, and QLoRA.

The comparison’s result is a set of actual base/adapted answers with their audience judgments and costs. A factor count or successful model import supplies none of those outcomes.

24.5 Character names, shifted targets, and generation feedback

A generated character must become the next recurrent input while the corresponding state is carried forward. The Character LSTM lab trains a dinosaur-name model and compares generation choices. It shares a supporting notebook with the AG News classification lab in §24.2. The small check below only calculates candidates from an untrained network.

Code example: Untrained character LSTM states and top-k candidate probabilities

import torch
import torch.nn as nn

vocab_size, hidden_width = 6, 8
inputs = torch.nn.functional.one_hot(
    torch.tensor([[0, 1, 2]]),
    vocab_size,
).float()
lstm = nn.LSTM(vocab_size, hidden_width, batch_first=True)
output_head = nn.Linear(hidden_width, vocab_size)

states, recurrent_state = lstm(inputs)
logits = output_head(states[:, -1])
top_values, top_ids = torch.topk(logits, k=3, dim=-1)
probabilities = torch.softmax(top_values, dim=-1)

assert states.shape == (1, 3, hidden_width)
assert top_ids.shape == (1, 3)
print(top_ids, probabilities)

One-hot input has shape [1,3,6], states have shape [1,3,8], and the final output head produces six logits. topk retains three IDs and softmax normalizes their scores. The printed probabilities sum to one over those candidates. No draw, append, repeated model call, or parameter update occurs.

Conclusion: The snippet checks a recurrent interface and a filtered candidate distribution. A complete name generator additionally needs learned parameters, token feedback, stopping, and state handling.

The full name experiment fits characters and compares generated names from fixed prefixes. It needs a name-level split before overlapping windows, learned parameters, repeated token feedback and an explicit stopping rule.

Experiment: Learned character-name generation

Add < and > boundary markers to each name. For example, anna becomes <anna>, so the width-four input <ann has the one-position-shifted target anna. Split complete names or source groups before constructing overlapping windows. Related windows then remain in one split under §9.1’s leakage rule. Fit the character vocabulary on training names, reuse that mapping unchanged for validation, and state how unseen validation characters are handled. Replacing the mapping after IDs have been created would make those IDs inconsistent.

Each input window is paired with its one-character-offset targets. One-hot encoding appends a vocabulary axis to the input axes and must be converted to the dtype and device expected by the LSTM. A fixed batch should expose each layer’s actual dimensions before training. The source configuration uses hidden width 256, two LSTM layers, dropout 0.5, Adam rate 0.001, batch size 64, ten epochs, and gradient clipping at 5. These values define the recorded case rather than a general recommendation.

The LSTM applies dropout between its recurrent layers during training, while a separate dropout operation acts on the last layer’s sequence output before the vocabulary head. Both are disabled in evaluation mode. Training at every position reshapes states from [B,T,256] to [B*T,256], produces logits \([BT,|V|]\) for vocabulary size \(|V|\), and pairs them with targets flattened in the same order. The sequence length used in this reshape must match the chosen window width. Record the actual example and update counts rather than inferring them from an epoch setting.

For a two-layer model, both hidden and cell state have shape [2,B,256], using the current batch size, dtype, and device. Reset them for independent names or windows. Carry them only across deliberate contiguous segments of one stream, with a declared detach policy during training. Carrying state across shuffled unrelated windows would mix their contexts.

Generation encodes a prefix, obtains its state, selects a character, appends it, and feeds that character into the next call with the carried state. It stops at > or a declared length limit and resets before another independent request. Random sampling, top-k sampling, temperature changes, and a greedy baseline can then be compared with fixed prefixes and seeds. Retain the selected settings, generated names, loss histories, and observed repetition or dependency failures.

Assign the result of a device conversion, for example inputs = inputs.to(device), so inputs and model share a device. Derive batch and sequence sizes from the current batch when reshaping targets and creating states. If a smaller final batch is dropped, record which examples that policy excludes.

For validation, switch to net.eval() and disable gradient recording, then restore training mode for the next training batch. In generation, test the stop marker and length limit after every selected character, including the first. Define the limit as the number of newly generated characters.

Its loss history and generated names then show whether training learned useful continuations. The untrained candidate-only check above establishes only the interface and normalized retained candidates.

The name task has the same next-token alignment as a larger language model. A count predictor instead uses an explicit fixed history, without a learned recurrent state.

24.6 The NLTK count-model tutorial and its extensions

A count table can expose how one observed history produces probabilities before a neural model is fitted. The NLTK notebook demonstrates count-model fitting and generation. Its nltk.lm.MLE model uses the maximum-likelihood principle explained in §8.1, fitting continuation probabilities from their observed relative frequencies. It then generates tokens from that fitted model. Smoothing and held-out perplexity below are book extensions.

The runnable local check uses Python’s standard-library Counter, not nltk. It counts bigrams in two boundary-padded sentences.

Code example: Padded bigram counts and maximum-likelihood probability

from collections import Counter

sentences = [["<s>", "the", "cat", "</s>"], ["<s>", "the", "dog", "</s>"]]
bigrams = Counter(
    (left, right)
    for sentence in sentences
    for left, right in zip(sentence, sentence[1:])
)
contexts = Counter(left for left, _ in bigrams.elements())

probability = bigrams[("the", "cat")] / contexts["the"]
assert probability == 0.5
print(bigrams, probability)

The context the has two continuations, one cat and one dog, giving \(P(\mathrm{cat}\mid\mathrm{the})=1/2\). Its numerator and denominator refer to the same one-token history.

Conclusion: The ratio is defined for this observed history. An unseen continuation has zero MLE probability, and an unseen history has a zero denominator in the direct count ratio. Library fallback behavior must be checked separately.

The NLTK interface adds vocabulary handling and a stored model to this count calculation. The next example uses nltk 3.9.2 and dill 0.4.1 on the same two short sentences, now with trigram histories. The code fits its counts from these local sentences without downloading a corpus or loading pretrained weights.

pad_both_ends adds two <s> tokens and two </s> tokens for order three. These boundary markers are modeled sentence content, unlike dense-batch placeholders excluded by a loss mask. padded_everygram_pipeline supplies one iterator over sentence n-grams of lengths one through three and another over vocabulary tokens. Both consume the original sentence collection, so the example keeps that collection as a reusable list. Inspecting and exhausting an iterator requires creating a fresh one before fitting.5

Code example: NLTK trigram counts, bit-based scores, and a saved-model round trip

from io import BytesIO
import dill
from nltk.lm import MLE
from nltk.lm.preprocessing import pad_both_ends, padded_everygram_pipeline

sentences = [["the", "cat"], ["the", "dog"]]
order = 3
print([list(pad_both_ends(s, n=order)) for s in sentences])
training, vocabulary = padded_everygram_pipeline(order, sentences)
model = MLE(order)
model.fit(training, vocabulary)

history = ("<s>", "the")
count = model.counts[history]["cat"]
total = model.context_counts(history).N()
score = model.score("cat", history)
print(count, total, score)
targets = [history + ("cat",)]
print(model.logscore("cat", history),
      model.entropy(targets), model.perplexity(targets))
print(model.vocab.lookup("never_seen"),
      model.score("never_seen", history))

saved = BytesIO()
dill.dump(model, saved)
saved.seek(0)
restored = dill.load(saved)
assert restored.counts[history]["cat"] == count
assert restored.score("cat", history) == score
assert (count, total, score) == (1, 2, 0.5)

from nltk.tokenize.treebank import TreebankWordDetokenizer

def sentence_text(tokens):
    words = []
    for token in tokens:
        if token == "<s>":
            continue
        if token == "</s>":
            break
        words.append(token)
    return TreebankWordDetokenizer().detokenize(words)

generated = model.generate(20, text_seed=["<s>", "<s>"], random_seed=7)
after_reload = restored.generate(20, text_seed=["<s>", "<s>"], random_seed=42)
assert len(generated) == len(after_reload) == 20
assert after_reload == model.generate(20, text_seed=["<s>", "<s>"], random_seed=42)
print(generated, sentence_text(generated))
print(after_reload, sentence_text(after_reload))

The padded sentences are <s> <s> the cat </s> </s> and <s> <s> the dog </s> </s>. For the two-token history ('<s>', 'the'), cat occurs once among two continuations. The first numerical line is therefore 1 2 0.5. The next is -1.0 1.0 2.0: log score, mean negative log score over the single supplied target, and perplexity. NLTK uses base-two logarithms here, so these log quantities are in bits. Chapter 8’s natural-log loss for probability 0.5 is about 0.693 nats. Either convention gives perplexity 2 when the matching exponential base is used.6

The default vocabulary cutoff is one. Inspect model.vocab and its length before and after fitting to distinguish an initialized model from one with corpus counts. model.vocab.lookup accepts either a token or a token sequence: known tokens retain their identities, while an unfamiliar token maps to <UNK>.

The vocabulary’s unk_cutoff sets the minimum occurrence count for a distinct token identity. Configure it before fitting. For example, supply Vocabulary(unk_cutoff=2) to a new MLE model, then call fit with fresh training and vocabulary iterators. Tokens occurring once map to <UNK>, and their n-gram counts are fitted under that identity. Changing the mapping later requires fitting the counts again under the new policy. With this example’s default cutoff, an unseen word maps to <UNK> but that lookup does not create an observed continuation or smoothing probability. The unknown-word line is <UNK> 0.0.7

The dill round trip writes the fitted vocabulary and counts to an in-memory stream and restores them. Its assertions compare the original and restored score and count. Loading a serialized model executes a reconstruction procedure, so use only a trusted saved file.

The example also generates 20 tokens with seed 7, then generates from the restored model with seed 42. text_seed supplies two start markers as the initial history. The conversion to readable text skips <s>, stops at the first </s>, and uses TreebankWordDetokenizer to join the retained tokens. The 20-token budget limits model draws. The displayed sentence can be shorter because it stops at an end marker. A seed controls sampling for this fitted model, not the quality of the text.89

Conclusion: The fitted model preserves its count ratio through saving and loading, supports generation after reload, and distinguishes unknown-token mapping from smoothing. The token-to-text helper preserves the first sentence’s boundary. This tiny calculation checks those API operations, not the quality of a model fitted to a large corpus.

The original tutorial inspections compare raw corpus counts with unigram, bigram and trigram scores. Their corpus and preprocessing differ from the two local sentences above.

Each count prediction uses at most the preceding \(n-1\) tokens from §3.4. Text outside that history cannot affect its next-token distribution, so locally plausible continuations can lose an earlier subject or constraint.

The notebook catalogue preserves the original prose and tweet corpora, preprocessing paths and all generation budgets/seeds. Starting without a supplied history and starting with boundary tokens are different experiments.

A local check using Python 3.11.9 and NLTK 3.9.2 on the original artificial corpus makes this distinction visible. Fit an order-three MLE model with its default cutoff. With seed 1 and no text_seed, the first generated token is </s>, so the tutorial’s text helper returns an empty string. The same seed with text_seed=['<s>','<s>'] instead starts with a. Removing boundary markers after generation does not supply the missing initial context.

The tutorial lists Lidstone, Laplace, InterpolatedLanguageModel and WittenBellInterpolated as API alternatives, without implementing a smoothing comparison. The add-alpha calculation is a separate book extension. Treat the listed classes as starting points for a chosen extension, not as completed experiments.

For a separate evaluation extension, hold out text before fitting the vocabulary and counts. Declare tokenization, boundary inclusion, unknown handling, and the mean-loss denominator. Compute target log losses and perplexity as in §8.5. An observed event with zero MLE probability gives infinite loss. Smoothing reserves probability but does not lengthen the retained history.

Keeping vocabulary, counts, and evaluation policy fixed isolates the smoothing rule’s effect on held-out loss. Neither finite perplexity nor a plausible generated sample establishes general language understanding.

24.7 Captioning, visual questions, and retrieval evidence

A caption, an answer to a visual question, and a retrieved object label describe different outputs from an image. The multimodal notebook supports retaining the inputs and failures separately for each task.

The patch conversion check verifies input preparation. Captioning, question answering and retrieval additionally need the model paths and output evidence below.

Captioning needs generated descriptions. VQA needs answers paired with questions, while retrieval needs held-out category decisions. The procedure below keeps their reference evidence separate even when the same image appears in several tasks.

Experiment: Captions, visual questions and object retrieval

For a full model, record the image processor, encoder, projection or fusion, text component, and selection settings. Preserve the original image, processed dimensions, prompt, output IDs, decoded text, and model revision. Resizing and cropping can remove evidence needed by a question.

The captioning protocol permits Salesforce/blip-image-captioning-base, Salesforce/blip-image-captioning-large, microsoft/git-base, or another suitable image-captioning model. Choose one and record its resolved revision.

The transformers library provides task-specific model classes. BLIP captioning pairs BlipProcessor, which prepares image and text inputs, with BlipForConditionalGeneration. BLIP visual question answering pairs that processor with BlipForQuestionAnswering. Load the processor and model for the same selected checkpoint through their from_pretrained methods. The captioning class generates text from the image and any supplied prompt, while the question-answering class also receives the question.10

For GIT captioning, GitProcessor prepares the inputs and GitForCausalLM supplies the image-conditioned text generator.11 Check the processor/model pairing, input dimensions, device, numerical types and decoded output on one image before the full protocol. A successful class import does not establish that checkpoint files loaded or that inference succeeded.

The source notebook uses an inspection set of at least five images spanning people, animals, indoor and outdoor scenes, and multiple objects. Retain each image with its generated caption and inspect contrasting successes and failures. A visible omission or false attribute is an observed error. An explanation of its cause remains a hypothesis until a controlled comparison supports it. This small inspection set can expose failure types but cannot provide a stable quality estimate.

For VQA, the source permits Salesforce/blip-vqa-base, Salesforce/blip-vqa-capfilt-large, or another suitable question-answering model. Match the processor and model to that checkpoint, and pass both the image and question. Its inspection set uses at least three images and asks object-recognition, counting, and color or attribute questions. Retain every image, question, model output, and reference answer, and inspect failures by question type. The same image can support a correct object answer and an incorrect count, so those results need separate judgments. A benchmark claim requires a larger evaluation sample selected independently of the observed failures.

The retrieval experiment uses objects or landmarks in four categories, with six images per category split into four references and two held-out queries. Include different viewpoints and backgrounds. These counts define the evaluation sample used for this comparison.

For the retrieval comparison, use one fixed pretrained encoder and its matched preprocessing, following the separate-encoder comparison in §22.3. Record the encoder identifier and resolved model revision, matched processor identifier and preprocessing configuration, and inference library versions. Retain that record with the stored reference vectors. Normalize nonzero image vectors, compare query vectors with stored references by cosine similarity, and aggregate reference scores using a declared rule. Report held-out top-one accuracy, a labeled confusion matrix, and two failures where available. Keep query images and near-duplicates out of the reference set.

Retrieval returns a category label from the candidate collection. It does not generate a description of attributes or actions. Captioning and VQA supply those separate outputs. A successful retrieval result therefore cannot substitute for the required generated-output analyses.

For each VQA failure, propose a possible cause grounded in the retained image, question and answer. Treat that cause as a hypothesis unless a controlled comparison supports it. Retain both successes and failures from the required samples. If the required number of failures does not occur in the initial set, inspect additional examples for diagnosis and report how they were selected. Keep those extra cases separate from the original evaluation sample and its accuracy denominator.

Caption errors, question-specific answers and retrieval accuracy support different conclusions. Extra diagnostic cases stay outside the original accuracy denominator.

24.8 A checked classifier result

The classifier in §24.1 supplies one result that can be checked from input through update. Its two identity-feature rows have labels 1 and 0. Mean binary log loss begins at 0.621268 nats, and the calculated gradient leads to new weights and bias.

Table 24.1: The local two-row classifier result and its evidence boundary.
Record Before update After the checked SGD step
Weights and shared bias w=(0.2,-0.1), b=0 w approximately (0.222508,-0.123751), b approximately -0.001243
Mean loss on the same two rows 0.621268 nats 0.610612 nats
Held-out sentiment result No measurement from this local check Still requires the separate full experiment

The lower loss was measured on the same two rows used to form the gradient. It verifies this local update under the chosen arithmetic and rate. It supplies no validation estimate, sentiment accuracy or convergence result.

The synthetic fitting experiment can extend that check to a complete optimization history. The SST-2 experiment adds text preprocessing, sparse features, a declared search budget and held-out evidence. Until those runs are executed and their results retained, their conclusions remain open.

The same distinction applies to every lab: a measured change needs an input, a baseline, an actual intervention and a result under fixed comparison conditions. The execution record keeps the general checklist. The Glossary, Notation and References support revisiting the terms, calculations and sources.

Chapter checkpoint

A patch-shape assertion passes, a LoRA factor count is correct, and a CPU device check succeeds. Which of caption quality, successful adapter training, and GPU speed has been established?

Answer: None of those broader results follows. The checks establish representation shape, factor count, and CPU placement under their inputs. The experiments still need generated-output judgments, actual updates and held-out comparisons, or a valid GPU measurement.

Why must the name split precede overlapping windows, and why is NLTK smoothing labeled a book extension?

Answer: Related windows can otherwise expose the same name across fitting and evaluation. The original tutorial implements MLE and only lists smoothing alternatives. A smoothing experiment needs its own fitted model and matched evaluation policy.


  1. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., & Potts, C. (2013). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1631–1642, §§3 and 5.2. The lab loads the binary distribution published by SetFit (n.d.), Stanford Sentiment Treebank - Binary, dataset SetFit/sst2, revision 00ea8ccb7a54b4e3780a3e51aa3f80361ff849c0. Pass that value through load_dataset’s revision argument. This sentence-level distribution contains 6,920 training, 872 validation and 1,821 test examples. Here 0 denotes negative and 1 positive sentiment.↩︎

  2. fancyzhx. (n.d.). AG News dataset card. Repository fancyzhx/ag_news, revision eb185aade064a813bc0b7f42de02595523103ca4. Pass this revision to load_dataset through its revision argument.↩︎

  3. Hugging Face. (n.d.). WordPiece vocabulary trainer. Tokenizers 0.22.2 source code, WordPieceTrainer::train. It calls the BPE trainer before conversion to a WordPiece model.↩︎

  4. Hugging Face. (n.d.). BPE vocabulary trainer. Tokenizers 0.22.2 source code, Merge::cmp and BpeTrainer. Merge priority uses pair counts.↩︎

  5. NLTK Project. (2025). Language-model preprocessing. NLTK 3.9.2 source, padded_everygram_pipeline.↩︎

  6. NLTK Project. (2025). Language-model API. NLTK 3.9.2 source, logscore, entropy, perplexity, and generate.↩︎

  7. NLTK Project. (2025). Language-model vocabulary. NLTK 3.9.2 source, Vocabulary and unk_cutoff.↩︎

  8. NLTK Project. (2025). Language-model API. NLTK 3.9.2 source, logscore, entropy, perplexity, and generate.↩︎

  9. NLTK Project. (2025). Treebank tokenization and detokenization. NLTK 3.9.2 source, TreebankWordDetokenizer.↩︎

  10. Hugging Face. (n.d.). BLIP. Transformers 5.10.2 documentation. The captioning and visual question-answering classes have different input roles.↩︎

  11. Hugging Face. (n.d.). GIT. Transformers 5.10.2 documentation. The model conditions text generation on image inputs.↩︎