20 Model adaptation: Context, preferences, and trainable weights
Adaptation can change context, parameters, or both. Its objective, trainable state, and storage format are separate choices.
A pretrained model may lack the information needed for an answer or fail to follow a required output format. These failures call for different changes. A prompt or retrieved passage changes the current input. Full fine-tuning changes base parameters, while a smaller trainable addition can change the effective computation with less update state.
The choice depends on held-out task results, available training examples, and resource costs. A different parameter count or storage format alone establishes no improvement in behavior.
20.1 Context changes and parameter training
For a request such as Summarize this record in one sentence, a prompt demonstration can show the required length and structure. Comparing that prompt with a trained model on the same held-out requests tests whether changing context is enough or parameter updates help.
In-context learning supplies task examples inside the prompt while keeping model parameters fixed.1 The examples consume context positions and must be supplied again when their information is needed. §18.3 develops fixed-parameter task prompting.
Fine-tuning continues training from a pretrained checkpoint on selected data. It can continue a self-supervised objective on domain text without instruction-answer annotations.
For paired inputs and desired outputs, supervised fine-tuning, or SFT, fits those answers. Instruction tuning is an SFT case using instructions and corresponding responses, sometimes with additional context.
Full fine-tuning makes all base-model parameters eligible for updates. PEFT, parameter-efficient fine-tuning, instead fits selected parameters or smaller added components. Both use a loss and gradients. Their trainable tensors and stored update state differ.
Fine-tuning changes the selected parameters using a task loss. Full tuning selects the base parameters. An added adapter is a trainable transformation inserted into the computation. Adapter training selects its parameters while holding the base fixed.2 An update establishes a parameter change. A new loss calculation and held-out evaluation determine its effect.
The same format-following need gives a concrete comparison:
| Method | What changes | Cost and evidence |
|---|---|---|
| Written prompt or demonstrations | Context for each request, with fixed parameters | Extra input tokens. Test format success on held-out prompts. |
| Full fine-tuning | All eligible base-model parameters | Base gradients and optimizer state. Check task improvement and retained behavior. |
| Parameter-efficient tuning | Selected parameters or smaller added components | Less trainable state, with constrained changes. Measure the same task and retention criteria. |
The table compares interventions, not measured winners. A short demonstration is useful when context alone produces the required behavior. Training becomes a candidate when repeated failures justify its data preparation and update costs. Missing or changing factual information raises the separate retrieval choice in §20.2.
PEFT includes several parameter choices. Selected-parameter tuning can update only existing biases or chosen layers. Adapter training instead fits an added transformation. Learned prompt vectors are numerical input additions optimized by gradients, unlike ordinary written in-context examples.3 LoRA, developed in §20.5, factors a trainable matrix change into two smaller matrices. These alternatives differ in insertion points, trainable state, inference work, and available capacity.
For a causal instruction example, the model receives the formatted instruction, optional context, and recorded response. A response-only objective ignores prompt targets while retaining the prompt as visible context. Another objective can supervise all recorded continuations. The alignment and loss-mask rules determine which logits meet which targets, independently of which parameters are trainable.
Training examples can pair a problem with written solution steps and a final answer. Supervised fine-tuning on those sequences can encourage a model to produce the same response structure. Check the emitted steps against the problem and final answer. They do not reveal every operation inside the model.
Catastrophic forgetting is loss of previously useful behavior during further fitting.4 Narrow data or unsuitable updates can improve the target task while worsening others. Mixing tasks, selecting fewer updates, or changing the learning rate are possible controls to compare. None guarantees preservation, and freezing the base does not ensure that an active adapter preserves its outputs.
Validation should measure the requested behavior and any general capabilities that must remain useful. Training and validation prompts must stay separate. A learning rate below a pretraining rate is a possible starting choice, not a universal requirement. The execution record retains the checkpoint, data, parameter list and budget. §24.4 gives a fuller comparison protocol.
Context and parameter changes can also be combined. Before fitting changing facts into weights, a retrieval system can supply the relevant records as input.
20.2 Retrieval, selected evidence, and the generated answer
A model cannot use an updated record that is absent from both its learned behavior and its current input. RAG, retrieval-augmented generation, finds relevant external material and includes selected passages in the generation context. The generator can then condition on those passages while its parameters remain fixed.5
The retrieval component receives the question and searches an indexed collection. Keyword retrieval compares terms, while vector retrieval compares query and passage representations using a similarity measure such as cosine. The collection stores passage text and identifying information alongside the index. A retriever may itself have trained parameters. Fixed generator weights do not imply an untrained retriever.
Retrieval returns candidates with scores. A selection step checks relevance, record version, access, and fit within the context budget. An optional reranker scores question-passage pairs more closely. The application assembles the selected text with the question and task instruction. The generator then produces an answer, whose claims can be checked against the supplied records.
For response sequence \(y\), the conditional distribution is
\[ P_{\theta}(y\mid\mathrm{prompt},\mathrm{retrieved\,context}) \tag{20.1}\]
Here \(\theta\) denotes fixed generator parameters. The prompt contains the original request, and the retrieved context contains the selected evidence. This expression describes generation after retrieval. It does not specify an index, ranking algorithm, or guarantee that the selected passages answer the question.
A tuned generator can also read retrieved evidence: context selection and parameter training can be combined.
Example: A retrieved record and a version mismatch
The question is How many Batch A samples passed in the current run? The supplied collection contains three short records. These records and scores are chosen for this example.
| Record | Passage | Score |
|---|---|---|
| R1 | Last month. Batch A: 12 samples, 2 failed. | 0.92 |
| R2 | Current run. Batch A: 12 samples, 3 failed. | 0.88 |
| R3 | Current run. Batch B: 10 samples, 1 failed. | 0.70 |
Retrieval returns R1 before R2. Selecting only the highest score supplies last month’s record, from which the answer would be 10. That calculation is faithful to R1 but wrong for the requested current run.
A version-aware selection chooses R2 instead. The assembled context says Current run. Batch A: 12 samples, 3 failed. The requested output is a one-sentence answer with its record ID. A supported answer is Nine samples passed [R2]., since \(12-3=9\).
The reference for evaluation is the current record and its count, not the retrieval score. R3 discusses the wrong batch, despite its semantic similarity. Even with R2 present, a generated answer of 10 would still require correction.
Conclusion: Ranking supplies candidates, while selection determines which evidence reaches generation. The failed version choice shows why a high similarity score cannot certify an answer.
Retrieval can fail through missing records, unsuitable segmentation, poor ranking, stale versions, or truncation of necessary context. The generator can also ignore or misread useful evidence. In either case it may produce a hallucination: an unsupported or contradictory claim presented as established fact. Retrieved text should remain evidence to assess, rather than acquiring the authority of application instructions. Access controls and instruction handling belong to the application, not the similarity score.
Index storage, retrieval latency, extra prompt tokens, and citation checking add costs. Parameter training adds a different cost and may become outdated as records change. A useful comparison measures answer quality and resource use under the same questions. Preferences between otherwise plausible answers provide another learning signal.
20.3 Preference signals and reward fitting
Two answers to the same question can differ in usefulness even when both satisfy a basic format check. A preference record keeps the prompt, both responses, the preferred response, and the judgment criterion together. These comparisons can supervise a scorer or directly guide the response model.
Alignment denotes training and evaluation intended to guide behavior toward specified instructions, preferences, or constraints. The criteria and judging population matter. Preference agreement, factual accuracy, and source faithfulness remain different measurements. Target alignment in §8.5 pairs prediction positions with recorded targets.
A language policy is the model’s conditional distribution over output actions, here tokens forming a response. Write \(\pi_\theta(y\mid x)\) for its probability of response \(y\) to prompt \(x\). Using the sequence factorization from §1.2, its log probability sums the conditional token log probabilities, including the declared stopping convention.
A reward model is a learned scorer \(r_\phi(x,y)\) of a prompt-response pair. Its parameters \(\phi\) differ from the language policy’s \(\theta\). A Bradley-Terry comparison uses the difference between two scores: it predicts preference for response \(y_w\) over \(y_l\) with \(\sigma(r_\phi(x,y_w)-r_\phi(x,y_l))\). Negative log loss on the observed preference trains the scorer. This stage updates \(\phi\), not the language policy.67
For example, supplied scores 1.2 and 0.2 give a difference of 1. Their sigmoid is approximately 0.731059, with pair loss approximately 0.313262. Raising the preferred-minus-rejected gap lowers that loss. The calculation scores agreement with this comparison, not whether either response is true.
RLHF, reinforcement learning from human feedback, uses human judgments to train or supply a reward signal for policy learning. In the reward-model arrangement, the current language policy generates responses. A fitted scorer evaluates them, and policy gradients change the response distribution. A reference policy stays fixed to provide a comparison distribution.8
RLAIF, reinforcement learning from AI feedback, uses AI-generated judgments for some feedback records.9 Their origin changes the judging process, not the need to test its errors and criteria. Human and automated judgments can both be inconsistent or poorly matched to deployment needs.
A KL penalty subtracts a multiple of the divergence from that reference. For prompt distribution \(\mathcal D\), the idealized reward objective is
\[ \max_{\theta}\;\mathbb{E}_{x\sim\mathcal{D}}\!\left[\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)}r_\phi(x,y)-\beta\operatorname{KL}\!\left(\pi_\theta(\cdot\mid x)\mathbin\Vert\pi_{\mathrm{ref}}(\cdot\mid x)\right)\right] \tag{20.2}\]
The outer expectation averages prompts \(x\sim\mathcal D\). Within each prompt, \(y\sim\pi_\theta(\cdot\mid x)\) supplies the response average. The reward parameters \(\phi\) and reference policy remain fixed during this policy update. The coefficient \(\beta>0\) controls the penalty, with KL as defined in §8.2. Reference probabilities must support the responses used by the trained policy. A larger objective need not mean greater truthfulness if the reward model favors the wrong property.
A sampled response supplies recorded token choices and a reward. Policy learning differentiates the log probabilities assigned to those choices, rather than differentiating through the discrete sampling decision. Weighting that gradient by reward relative to a baseline favors choices with better-than-expected outcomes. The baseline predicts expected reward from the available context. An advantage estimate measures an outcome relative to that prediction.
For a finite set of possible responses with fixed rewards, expected reward is \(J(\theta)=\sum_y\pi_\theta(y)r(y)\). Differentiating the probabilities gives \(\nabla J=\sum_y r(y)\nabla\pi_\theta(y)\). Where probabilities are positive, \(\nabla\pi_\theta(y)=\pi_\theta(y)\nabla\log\pi_\theta(y)\). Substitution writes the same derivative as an average of reward-weighted log-probability gradients under the policy.
For two choices, let the first have probability \(p=\sigma(\theta)\) and reward 2, and the second probability \(1-p\) and reward 0. Then \(J=2p\) and \(dJ/d\theta=2p(1-p)\). At \(p=0.5\), the first choice’s log-probability derivative is \(1-p=0.5\). Its reward-weighted value is 1, but it occurs only half the time. The other choice contributes zero, so their expected contribution is 0.5, matching the direct derivative.
Subtracting a baseline \(b\) that does not depend on the chosen action leaves this expectation unchanged: its contribution is \(b\sum_y\nabla\pi_\theta(y)=b\nabla1=0\). The baseline is held fixed in this policy-gradient calculation, even when a separate predictor is fitted to estimate its value. These conditions explain how a sampled action supplies a gradient estimate without differentiating the act of sampling. This reward-weighted log-probability estimator is the basis of REINFORCE.10
20.4 Policy updates from rollouts and preference pairs
A fitted reward signal does not specify how far a policy should change in one update. PPO limits part of the sampled-update objective. DPO instead fits preference pairs directly against a fixed reference.
PPO, Proximal Policy Optimization, is one policy-update method used in this arrangement. A rollout here is a response sampled under a saved policy version. PPO compares the updated probability of each recorded action with its probability under that saved version.11
Write their ratio as \(\rho\) and the estimated advantage as \(a\). A common clipped objective uses \(\min(\rho a,\operatorname{clip}(\rho,1-\epsilon,1+\epsilon)a)\) for \(0<\epsilon<1\). Here clipping restricts the ratio used in one branch of this training objective. Positive advantages stop rewarding increases beyond the upper ratio, while negative advantages stop rewarding decreases beyond the lower ratio. This differs from directly constraining every policy probability. Repeated sampled responses and refreshed advantage estimates support later updates. Clipping and a KL penalty do not guarantee a hard bound on every distribution change.
For \(\epsilon=0.2\), advantage \(a=1\), and ratio \(\rho=1.4\), the objective contribution is \(\min(1.4,1.2)=1.2\). For a separate negative advantage \(a=-1\) and ratio \(\rho=0.6\), it is \(\min(-0.6,-0.8)=-0.8\). These calculations show the capped incentive on either side. They do not clip the model’s actual probability ratio to that interval.
DPO, Direct Preference Optimization, instead fits the language policy directly on preferred-rejected pairs against a fixed reference. Its one-pair loss is12
\[ -\log\sigma\!\left(\beta\left((\log\pi_{\theta}(y_w\mid x)-\log\pi_{\mathrm{ref}}(y_w\mid x))-(\log\pi_{\theta}(y_l\mid x)-\log\pi_{\mathrm{ref}}(y_l\mid x))\right)\right) \tag{20.3}\]
For the same prompt \(x\), \(y_w\) denotes the preferred response and \(y_l\) the rejected response. Each parenthesized difference is a log probability ratio between the trainable and reference policies. Their difference measures how much the policy favors the preferred response over the rejected response relative to the reference. The positive \(\beta\) scales that comparison before sigmoid and negative logarithm.
DPO differentiates these sequence log probabilities with respect to \(\theta\). It needs no separately fitted reward model or online rollout loop for this pair loss. A fixed reference alone does not certify preserved behavior on other prompts.
Example: Completing the DPO pair loss
Let the preferred response’s reference-relative log probability be 1.5 and the rejected response’s 0.5, with \(\beta=0.5\). The sigmoid input is \(0.5(1.5-0.5)=0.5\).
Sigmoid gives approximately 0.622459. The one-pair loss is \(-\log\sigma(0.5)\approx0.474077\), or 0.474 nats. An unchanged policy equal to its reference gives input zero and loss \(\log2\approx0.693147\) on the same pair.
Conclusion: The positive relative gap gives this preferred pair a lower loss than the zero-gap baseline. It establishes the direction rewarded by this objective, not universal human agreement or factual correctness.
Preference data can omit important cases or reward polished but incorrect answers. Held-out comparisons and independent task checks remain necessary. The objective specifies what to improve. A parameterization such as LoRA specifies which changes can realize that improvement.
20.5 Low-rank changes to a frozen layer
Updating every entry of a large matrix also requires its gradients and often optimizer-state arrays. Fitting a smaller parameter set can reduce that trainable storage while retaining the original matrix for forward computation.
LoRA, Low-Rank Adaptation, freezes a base matrix and trains two factors whose product supplies an additive change.13 A frozen base weight is retained for computation but excluded from parameter updates. For input column \(x\in\mathbb R^{D_{\mathrm{in}}}\), let \(W\) have shape \([D_{\mathrm{out}},D_{\mathrm{in}}]\).
The factors have shapes \(A\in\mathbb R^{r\times D_{\mathrm{in}}}\) and \(B\in\mathbb R^{D_{\mathrm{out}}\times r}\). Their positive inner dimension \(r\) is the LoRA rank. It bounds the rank of the update \(BA\) but does not require that update to attain rank \(r\). The chosen numerator \(\alpha_{\mathrm{LoRA}}\) sets the adapter scale \(\alpha_{\mathrm{LoRA}}/r\). Implementations commonly store this numerator in the code field lora_alpha. The effective weight is
\[ W_{\mathrm{eff}}=W+\frac{\alpha_{\mathrm{LoRA}}}{r}BA \tag{20.4}\]
The product \(BA\) has the same shape as \(W\). Its rank satisfies \(\operatorname{rank}(BA)\le r\), because every output lies in the span of \(B\)’s \(r\) columns. Dependent or zero columns can make the rank smaller. This constrains the update, not the rank of the frozen base.
Forward execution can calculate \(Wx+(\alpha_{\mathrm{LoRA}}/r)B(Ax)\) without constructing a dense update matrix. It still computes the frozen base path. The two factors contain
\[ \operatorname{params}_{\mathrm{LoRA}}=r(D_{\mathrm{in}}+D_{\mathrm{out}}) \tag{20.5}\]
trainable values, from \(rD_{\mathrm{in}}\) in \(A\) and \(D_{\mathrm{out}}r\) in \(B\). A full matrix update has \(D_{\mathrm{in}}D_{\mathrm{out}}\) values. The ratio is \(r(D_{\mathrm{in}}+D_{\mathrm{out}})/(D_{\mathrm{in}}D_{\mathrm{out}})\). Biases, other adapted layers, and activation storage are excluded from this count.
Example: Four-by-four shape, count, and output change
Use \(W\) of shape \([4,4]\), \(r=1\), and \(\alpha_{\mathrm{LoRA}}=r\), giving scale one. Factor \(A\) has shape \([1,4]\) and four trainable entries. Factor \(B\) has shape \([4,1]\) and four entries. Their product is \([4,4]\), so eight trainable values replace a 16-value dense update.
For a numerical instance, set \(W=I_4\), \(A=(1,-1,2,0.5)\), \(B=(0.1,0,0.2,-0.1)^{\top}\), and \(x=(1,2,3,4)^{\top}\). Then \(Ax=1-2+6+2=7\).
The adapter output is \(B(7)=(0.7,0,1.4,-0.7)^{\top}\). Adding the base output \(Wx=x\) gives \((1.7,2,4.4,3.3)^{\top}\).
Conclusion: Both paths return four coordinates and add at the output. The base remains a four-by-four computation. The eight trainable factor entries determine only its additive change.
In a separate large layer, \(D_{\mathrm{in}}=D_{\mathrm{out}}=4096\) and \(r=8\). Each factor has 32,768 values, giving 65,536 against 16,777,216 for a dense matrix. The ratio is \(1/256=0.390625\%\). A matrix with only 100 entries does not determine the factor count: a 10-by-10 rank-one choice uses 20 values, while 5-by-20 uses 25.
Initialization matters because both factors enter the same product. The original LoRA construction initializes \(A\) randomly and \(B\) at zero. This starts with zero adapter output, preserving the base function at initialization.14
Let \(g\) be the loss gradient with respect to this layer’s output and \(s=\alpha_{\mathrm{LoRA}}/r\). The derivative chain rule gives \(\nabla_B L=s\,g(Ax)^{\top}\) and \(\nabla_A L=s\,B^{\top}gx^{\top}\). If \(B=0\), the first gradient for \(A\) is zero, while \(B\) can receive a nonzero gradient through \(Ax\).
For \(A\) and \(x\) above, \(Ax=7\). With zero \(B\), unit scale, and supplied \(g=(1,1,1,1)^{\top}\), every entry of \(\nabla_B L\) is 7. If both factors were zero, both gradients would be zero. Under this product’s ordinary gradient updates, neither factor could start changing. Nonzero \(A\) permits learning to begin, though a particular input or upstream gradient can still yield zero.
The following listing checks the count of eight trainable factor entries, the zero initial addition, the gradient of 7 in each entry of \(B\), and the zero first gradient of \(A\). It contains no frozen-base output or training dataset.
Code example: LoRA factor count, zero initial addition and first factor gradients
import torch
# Adapter factors with inner dimension one for a four-input, four-output layer.
d_in, d_out, rank = 4, 4, 1
A = torch.tensor([[1.0, -1.0, 2.0, 0.5]], requires_grad=True)
B = torch.zeros(d_out, rank, requires_grad=True)
trainable = A.numel() + B.numel()
assert trainable == rank * (d_in + d_out)
# A nonzero first factor and zero second factor give zero adapter addition.
x = torch.tensor([1.0, 2.0, 3.0, 4.0])
delta = B @ (A @ x)
assert torch.allclose(delta, torch.zeros(d_out))
delta.sum().backward()
assert torch.allclose(B.grad, torch.full_like(B, 7.0))
assert torch.count_nonzero(A.grad) == 0The zero addition agrees with \(B=0\), while the summed-output derivative reaches \(B\) through \(Ax=7\). Section 20.8 instead derives the upstream gradient from target losses.
A trained LoRA adapter can be merged for compatible inference by storing the effective weight \(W_{\mathrm{eff}}\) defined above as one matrix.
The merged matrix has the same dimensions as \(W\) and absorbs the scale and product. This removes the separate branch, but switching adapters then requires another merge or another stored weight version. Storing weights through the compact codes developed in §20.7, chosen dtypes, and rounding can restrict or alter a merge. peft exposes compatible merges through merge_and_unload().15
20.6 Smaller models, distillation, and pruning
An adapter reduces trainable state but leaves the stored base in place. A deployment limit may instead require fewer stored parameters or less computation. Smaller models, distillation, pruning and quantization make different changes, whose task quality and resource costs need separate comparisons.
A small language model is a relatively low-parameter language model selected for a stated resource and task budget. There is no universal size cutoff. It can be trained from scratch or adapted from another model. A smaller parameter count alone guarantees neither lower latency for every workload nor adequate task quality.
Reducing the stored model can make local execution possible within a device’s memory. That can avoid sending inputs to a remote service and support use without a network connection. Lower runtime, energy use, and cost remain measurements to compare for the actual workload. A narrow training domain can leave too few examples of important inputs in the training data, and limited capacity can reduce accuracy on demanding tasks.
Parameter count alone cannot establish which model handles a task better. The evaluation procedure in §9.4 compares models on common task criteria and test inputs while recording training and execution conditions. Distillation offers one way to train a smaller model using a larger model’s outputs.
Distillation trains a student to match information supplied by a teacher. One form uses teacher output probabilities as soft targets. The teacher remains fixed while the student receives gradients. Hard labels may supply another loss term, and some variants also compare compatible intermediate representations.16
With a common two-class vocabulary, let teacher probabilities be \(p=(0.8,0.2)\) and student probabilities \(q=(0.6,0.4)\). The soft-target loss is \(-0.8\log0.6-0.2\log0.4\approx0.591919\). If the student matches \(p\), the value becomes the teacher entropy, approximately 0.500402. The reduction of about 0.091516 is the mismatch KL from §8.2.
Replacing the teacher distribution by hard label 0 would instead yield \(-\log0.6\approx0.510826\). That label discards the teacher’s 0.2 mass on the other class. Distillation can also soften both distributions using a declared positive temperature. A teacher’s errors and the chosen training examples can transfer to the student too.
Pruning removes selected weights or structures under a declared criterion, often followed by further fitting. For example, zeroing small weights creates a sparse matrix without shrinking its array shape. Deleting complete channels can change dimensions and requires compatible downstream changes. Actual speed depends on whether the implementation exploits the resulting structure.
Quantization reduces the precision of stored values. Section 20.7 develops its compact codes and reconstruction. MoE routing selects active expert paths without removing all stored experts, as §17.7 explains. These methods can be combined, but their storage and computation effects must be measured separately.
20.7 Weight formats, reconstruction, and scale metadata
A frozen base still occupies memory even when its gradient and optimizer state are absent. Quantization approximates retained values using compact codes that require fewer storage bits. This can reduce the weight payload, but reconstruction introduces approximation and metadata costs.
Storage precision is the numerical format retained in memory. Compute dtype is the numerical type used by an operation. They need not be the same.
Floating-point formats divide bits among a sign, an exponent, and a fraction. The exponent largely determines range. The fraction determines spacing between nearby representable values. float32 uses 1, 8, and 23 bits respectively. float16 uses 1, 5, and 10, while bfloat16 uses 1, 8, and 7.17
Thus the two 16-bit formats each use two bytes per element but have different range and precision. bfloat16 keeps a range comparable to float32, with coarser spacing. Rounding and overflow are separate numerical risks. An eight-bit integer code uses one byte, but needs a reconstruction convention to represent a real-valued weight.
For one weight \(x\), an affine quantizer chooses a positive scale \(s\) and integer zero point \(z\). It rounds \(x/s+z\) to a supported integer code \(q\), clipping values outside the available code range. Reconstruction uses the same \(s\), \(z\), and \(q\):
\[ \hat{x}=s(q-z) \tag{20.6}\]
The reconstructed \(\hat x\) approximates the original value. Rounding loses distinctions within a quantization interval, and clipping adds error beyond its range. For code 13, scale 0.1, and zero point 3, reconstruction gives \(0.1(13-3)=1.0\). This affine example prepares the arithmetic. The later model-loading configuration uses NF4’s nonuniform reconstruction levels instead of this evenly spaced affine rule.
A separate code check below reconstructs the same affine value. It tests that arithmetic convention, rather than NF4 or model memory.
Code example: One affine code reconstructed with its scale and zero point
# Affine quantization reconstructs a value from an integer code.
scale, q, zero_point = 0.1, 13, 3
x_hat = scale * (q - zero_point)
assert x_hat == 1.0Its reconstructed result is 1.0 under the declared scale and zero point.
Block quantization assigns scale information to small groups of weights. A large outlier then controls only its group’s range instead of the whole matrix. Smaller groups can fit local ranges more closely, but need more metadata. Let \(w_{b,i}\) be entry \(i\) in block \(b\), \(s_b>0\) its block scale, \(z_b\) its zero point, and \(q_{b,i}\) its stored integer code. The affine reconstruction is
\[ \hat{w}_{b,i}=s_b(q_{b,i}-z_b) \tag{20.7}\]
A symmetric absmax quantizer fixes the zero point at zero and uses codes from \(-Q\) through \(Q\). For a nonzero block, let \(a=\max_i|w_i|\) and choose reconstruction scale \(s=a/Q\). Encoding rounds \(w_i/s\), and reconstruction multiplies the code by \(s\). Equivalently, use the reciprocal multiplier \(c=Q/a\), giving \(q_i=\operatorname{round}(cw_i)\) and \(\hat w_i=q_i/c\).
Consider the following five-by-five weight matrix, with each row as one block. Let \(a_b=\max_i|w_{b,i}|\) be the largest absolute weight in block \(b\):
| Block | Value 1 | Value 2 | Value 3 | Value 4 | Value 5 |
|---|---|---|---|---|---|
| 1 | -0.7 | -0.3 | 0 | -0.4 | 0.3 |
| 2 | -1 | 0.2 | 0.7 | 1.7 | -0.9 |
| 3 | -0.1 | -1.5 | -0.1 | 0.8 | 0.5 |
| 4 | 1.2 | -1.7 | -0.9 | -0.3 | 0.7 |
| 5 | 0.4 | 0.1 | -1.4 | 2.2 | -1.1 |
The row maxima in absolute value are \((0.7,1.7,1.5,1.7,2.2)\). With \(Q=127\), the corresponding encoding multipliers \(c_b=127/a_b\) are approximately \((181.429,74.706,84.667,74.706,57.727)\). Their reciprocals are reconstruction scales \(s_b\). Applying one multiplier to all rows would change the codes, because each row uses a different range.
For a rounding-boundary case, choose signed codes from \(-7\) through 7, zero point 0, and rounding to the nearest integer with ties to the even integer. This hand calculation uses exact decimal values. For the block \((-1.4,-0.3,0.6,1.1)\), the largest absolute weight is 1.4. Setting \(s=1.4/7=0.2\) gives unrounded codes \((-7,-1.5,3,5.5)\). Rounding and clipping give \((-7,-2,3,6)\).
Reconstruction gives \((-1.4,-0.4,0.6,1.2)\). The reconstructed-minus-original errors are \((0,-0.1,0,0.1)\). For an all-zero block, choose scale 1 and all codes zero so the scale remains positive. An implementation must also account for the representation of its inputs: binary floating-point approximations can move an intended half-integer slightly to either side of a tie. NF4 uses a different, nonuniform set of reconstruction levels.
A ten-value check gives a first reconstructed value of 1.525984 instead of 1.5. The supporting calculation below records its different scale and all codes.
Supporting calculation: Ten values with signed eight-bit codes
For values \((1.5,2.3,3.7,4.1,5.6,6.8,7.9,8.4,9.2,10.2)\) and \(Q=127\), the multiplier is \(127/10.2\approx12.45098\). Nearest rounding gives codes \((19,29,46,51,70,85,98,105,115,127)\). For the first value, reconstruction gives \(19(10.2/127)\approx1.525984\), an error of about \(0.025984\). Decoding the rounded codes uses the stated scale or multiplier to approximate the original weights.
The reconstruction error is about 0.025984 for that first value. Its size depends on the block range and chosen code levels.
NormalFloat 4 (NF4) instead uses four-bit codes to select 16 nonuniform reconstruction levels. The levels are designed around a normalized normal distribution, with more levels near zero. These levels form a codebook, a table whose entries are selected by the codes. For codebook \(C\) and block scale \(s_b\), reconstruction has the form \(\hat w_{b,i}=s_b C[q_{b,i}]\). It uses lookup rather than equally spaced integer subtraction.18
The code identifies a level, not a four-bit floating-point multiplier. For example, code 13 does not mean \(13s_b\) or \(s_b(13-3)\) in NF4. Its result requires the actual codebook entry. Distribution mismatch and outliers can still increase quantization error.
Double quantization additionally compresses the collection of block-scale constants. Reconstruction first recovers those constants, then uses them to recover weight values. Let \(q_b^s\) be the code for block scale \(b\), \(s_s\) its second-level scale, and \(z_s\) its second-level zero point. The affine reconstruction is
\[ \hat{s}_b=s_s(q_b^{s}-z_s) \tag{20.8}\]
This illustrates nested metadata reconstruction, not the exact NF4 implementation’s scale format. A system can compress reconstruction scales directly or compress their reciprocal encoding multipliers \(c_b=1/s_b\). The stored metadata must identify the convention.
For reciprocal multipliers, set \(c_2=127/\max_b c_b\), store \(r_b=\operatorname{round}(c_2c_b)\), and recover \(\hat c_b=r_b/c_2\). Reconstruction then divides the weight code by \(\hat c_b\), so \(\hat c_b\) must remain positive. For \(c_b=(1,1000)\), \(c_2=0.127\) gives codes \((0,127)\). The recovered first multiplier is zero, making that division undefined. Retaining the small multiplier at higher precision preserves it. Clamping its code to one instead changes the scale and requires a separate error check.
Example: A scale approximation changes the weight
The earlier code 13 with zero point 3 and scale 0.1 reconstructed to 1.0. In a separate second-level example, scale code \(q_b^s=8\), scale \(s_s=0.01\), and zero point \(z_s=0\) reconstruct \(\hat s_b=0.08\).
Using that approximate scale for the same weight code gives \(0.08(13-3)=0.8\). It differs from 1.0 by \(-0.2\). The scale approximation therefore changes every weight reconstructed using that scale.
Conclusion: Two quantization levels introduce two reconstruction steps. Recovering a scale of 0.08 cannot be treated as recovering the earlier 0.1 exactly.
The 25-weight matrix uses 100 bytes in float32. Its eight-bit weight codes use 25 bytes before metadata. Choose five blocks of five weights for a separate storage exercise. Five float32 scales add 20 bytes, giving 45. Assume symmetric coding with zero points fixed at zero and no stored zero-point array.
If each block scale is replaced by an eight-bit code, those five codes use five bytes. One float32 second-level scale adds four bytes. Under this chosen grouping, the payload becomes \(25+5+4=34\) bytes, excluding alignment and other metadata. A different group size or stored offset changes the total. The original 100-byte matrix size alone cannot determine a universal compressed total.
Four-bit weight codes nominally use half a byte each. Packing 25 of them needs at least 13 whole bytes before scales, codebooks, padding, and other metadata. This raw payload comparison does not give total training memory. Higher-precision adapters, activations, gradients, and optimizer state remain separate allocations.
QLoRA, Quantized Low-Rank Adaptation, stores the frozen base in a quantized form while fitting higher-precision adapters. A forward sketch is
\[ y=\operatorname{dequant}(W_{4\mathrm{bit}})x+\frac{\alpha_{\mathrm{LoRA}}}{r}BAx \tag{20.9}\]
Here \(W_{4\mathrm{bit}}\) stores the frozen weight codes. The \(\operatorname{dequant}\) operation also uses their scales, codebook and other stored metadata to reconstruct values for computation. The input and output widths match the unquantized layer. Gradients can pass through the fixed base transformation to earlier trainable components without updating its stored weights.
QLoRA reconstructs frozen values into a supported compute format for matrix multiplication and adds the adapter path. An implementation may reconstruct blocks inside a kernel rather than allocate a full dequantized matrix. The paper’s paged optimizer manages training-state transfers, a separate mechanism from Chapter 21’s paged inference cache.19
Storage reduction and prediction quality need separate measurements. The loss and adapter gradients still depend on the values used in the forward calculation, including quantization error.
The following configuration fragment follows the peft 0.17.0 quantization guide. transformers supplies model loading, bitsandbytes supplies quantized linear operations, and PyTorch supplies tensor operations. prepare_model_for_kbit_training prepares the loaded quantized model before adapters are attached.20
The fragment requires an actual checkpoint replacing model-name, a compatible tokenizer, and a compatible bitsandbytes installation and supported hardware. Its q_proj and v_proj targets fit some causal-model architectures, not every model. task_type="CAUSAL_LM" likewise does not configure an encoder-decoder model. A real experiment must pin compatible package and checkpoint revisions.
Code example: QLoRA configuration and trainable-parameter inspection
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
quantization = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
base = AutoModelForCausalLM.from_pretrained(
"model-name",
quantization_config=quantization,
)
base = prepare_model_for_kbit_training(base)
lora_config = LoraConfig(
task_type="CAUSAL_LM",
bias="none",
r=8,
lora_alpha=16,
target_modules=["q_proj", "v_proj"],
)
model = get_peft_model(base, lora_config)
model.print_trainable_parameters()
trainable = [p for p in model.parameters() if p.requires_grad]
assert trainable, "No trainable adapter parameters"Here r=8 and lora_alpha=16 mean \(r=8\) and \(\alpha_{\mathrm{LoRA}}=16\), giving adapter scale 2. The NF4 and double-quantization fields select base storage, while torch.bfloat16 selects its compute format. The fragment prints trainable parameter counts and collects the trainable tensors. It contains no dataset, optimizer, backward call, or training result.
The optimizer cycle in §11.4 still applies to the collected adapter tensors. As §11.1 explains, requires_grad=True alone does not add them to an optimizer. Fewer trainable values chiefly reduce gradient and update-state storage. Frozen weights and needed activations still consume memory, and actual speed requires measurement.
20.8 Aligned targets through an adapter-training step
An adaptation batch must connect its selected target positions to the parameters allowed to change. The following small calculation fixes that connection without claiming a completed training run.
The supplied-gradient check in §20.5 showed why zero-initialized \(B\) can change. Here the gradient is calculated from actual target losses through a fixed vocabulary head. The optimizer receives only \(A\) and \(B\), while the base and head stay fixed. The larger shape example records the separate width-64 layout.
Example: From target loss to the first adapter update
Use width four and a five-entry vocabulary. Three positions share input \(x=(1,2,3,4)\) and have targets \((2,3,4)\). The base is \(W=I_4\), the rank-one factors start at \(A=(1,-1,2,0.5)\) and \(B=0\), and a fixed \([5,4]\) head maps each hidden row to vocabulary logits.
Because \(B=0\), the initial adapter output is zero. Each position has logits \((0.1,1.1,1.1,-0.7,0.5)\). Mean cross-entropy over the three targets is approximately 1.925576 nats. For position \(t\), the logit gradient is \((p_t-\operatorname{onehot}(y_t))/3\). Multiplying those rows by the fixed head gives the three output gradients entering the adapted layer.
Summing their contributions gives \(\nabla_B L=(-0.255806,0.105728,0.473174,0.491399)^{\top}\). The initial \(\nabla_A L\) is zero because its path contains \(B=0\). One SGD step at rate 0.01 changes only \(B\). A new forward calculation gives loss 1.920185 nats.
Conclusion: The targets determine the cross-entropy derivatives at the vocabulary logits. The fixed head carries those derivatives to the adapted layer, and the nonzero \(A\) then carries them to the zero-initialized \(B\). The new forward loss checks the effect of the actual factor update rather than a supplied intermediate gradient.
The listing below calculates those gradients, applies the update and checks the new loss.
Code example: Cross-entropy gradients through a vocabulary head and a rank-one adapter
import torch
X = torch.tensor([[1., 2., 3., 4.],
[1., 2., 3., 4.],
[1., 2., 3., 4.]])
targets = torch.tensor([2, 3, 4])
W = torch.eye(4)
A = torch.tensor([[1., -1., 2., .5]], requires_grad=True)
B = torch.zeros(4, 1, requires_grad=True)
head = torch.tensor([[.2, .1, -.1, .0],
[-.1, .1, .2, .1],
[.0, -.2, .1, .3],
[.1, .0, .0, -.2],
[.0, .2, -.1, .1]])
optimizer = torch.optim.SGD([A, B], lr=0.01)
hidden = X @ W.T + (X @ A.T) @ B.T
hidden.retain_grad()
logits = hidden @ head.T
loss = torch.nn.functional.cross_entropy(logits, targets)
loss.backward()
dlogits = (torch.softmax(logits.detach(), dim=-1)
- torch.nn.functional.one_hot(targets, 5)) / len(targets)
layer_gradient = dlogits @ head
assert torch.allclose(hidden.grad, layer_gradient, atol=1e-6)
assert torch.allclose(B.grad,
layer_gradient.T @ (X @ A.T).detach(), atol=1e-6)
assert torch.count_nonzero(A.grad) == 0
old_A = A.detach().clone()
before = loss.item()
optimizer.step()
new_hidden = X @ W.T + (X @ A.T) @ B.T
after = torch.nn.functional.cross_entropy(new_hidden @ head.T,
targets).item()
assert torch.equal(A, old_A)
assert after < before
print(B.grad, before, after)The assertions check the gradient formulas, that \(A\) remains unchanged, and that the loss falls. The printed values give the gradients and losses shown above.
If the frozen base uses the quantized storage from §20.7, its values are reconstructed for the forward calculation. That changes the computed values, not which recorded tokens supply targets. To save the trained adapter, retain its configuration with the matching base checkpoint, tokenizer, and quantization settings needed to reconstruct the effective model.
Once parameters are fixed for generation, Chapter 21 follows the token and cache state that continues to change.
Chapter checkpoint
Does a rank-eight LoRA adapter guarantee an update of rank eight? Why initialize only one factor at zero? Does a retrieved high-scoring passage certify the answer?
Answer: The rank is at most eight and can be smaller. Zero \(B\) preserves the initial base output, while nonzero \(A\) permits a first gradient for \(B\). Setting both factors to zero blocks both initial gradients through their product. A retrieval score measures the ranking rule, so version, relevance, and the generated claims still need checking.
Why can four-bit base storage coexist with a higher-precision forward calculation, and what does the 0.474 DPO loss establish?
Answer: Stored codes are reconstructed for computation, with separate adapter and training state. The DPO value evaluates the supplied preferred-rejected pair against a fixed reference. It does not establish general factual accuracy or a completed training improvement.
Brown, T. B., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33. The paper evaluates task examples supplied in the prompt without gradient updates to the model.↩︎
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-Efficient Transfer Learning for NLP. ICML 2019, Proceedings of Machine Learning Research, 97, 2790–2799, §2. The method trains inserted adapter modules while keeping the pretrained network fixed.↩︎
Lester, B., Al-Rfou, R., & Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3045–3059, §2. The prompt vectors are learned parameters. The underlying model remains fixed.↩︎
Kirkpatrick, J., et al. (2017). Overcoming catastrophic forgetting in neural networks. arXiv:1612.00796, version 2, introduction. Sequential fitting can change shared weights needed for earlier tasks.↩︎
Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33. The record-selection trace is an illustrative application, not reported paper output.↩︎
Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs. I. The method of paired comparisons. Biometrika, 39(3–4), 324–345. The reward-score form used here is also given in Rafailov et al. (2023), §3.↩︎
Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744, §3. The numerical reward-pair example is illustrative.↩︎
Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744, §3. The numerical reward-pair example is illustrative.↩︎
Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073, version 1, §§1.2 and 3. The harmlessness procedure uses AI preference labels. Helpfulness also uses human labels.↩︎
Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229–256. The finite-response calculation demonstrates the score-function gradient identity. It is not a paper experiment.↩︎
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv:1707.06347, §3.↩︎
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, §4.↩︎
Hu, E. J., et al. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations, §4.1.↩︎
Hu, E. J., et al. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations, §4.1.↩︎
Hugging Face. (2025). PEFT checkpoint format. PEFT 0.17.0 documentation. Merge support depends on the model, adapter, and quantization configuration.↩︎
Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv:1503.02531. The numerical probabilities here are illustrative.↩︎
PyTorch Contributors. (2026). Tensor attributes and Type Info. PyTorch 2.12 documentation. Format sizes and numerical limits are distinct from allocation overhead.↩︎
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems, 36, §3. Affine integer examples here are comparisons, not NF4 codebook calculations.↩︎
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems, 36, §3. Affine integer examples here are comparisons, not NF4 codebook calculations.↩︎
Hugging Face. (2025). Quantization. PEFT 0.17.0 documentation. Preparation precedes adapter attachment. The fragment is not a complete training experiment.↩︎