Appendix B — Glossary and software reference
Look up training terms, software responsibilities, and the adapter parameter calculation after their method-level introductions.
The method chapters introduce each calculation beside the problem it solves. When comparing implementations or returning to a formula, the same definitions and software responsibilities are useful to consult together. The entries below collect that reference material, followed by the adapter calculation used by the SFT and DPO implementations and a comparison with the source slide’s PPO notation.
B.1 Training terms
The same terms recur when describing the feedback, the different model roles, and the training calculation. This glossary gives their short meanings. The method explanations develop the details with examples.
- Action: A choice made by the policy. It is a sampled next token in the token-level examples and a tool request in the action-level example. See Sections 3.1 and 5.3.
- Activations: Intermediate values computed during a model’s forward pass. Training may retain or recompute them to calculate gradients. See Section 3.1.
- Adapter / adapters: A small trainable addition to a model whose base parameters stay fixed. Section 1.3 introduces adapters; the LoRA calculation counts the values added by one adapter method.
- Adapter parameter: A value in a small trainable module attached to a larger model whose original parameters remain unchanged. LoRA is the adapter method used in the supplied SFT/DPO notebook.
- Advantage: A number that says whether an observed result was better or worse than the comparison value used for that update.
- Agentic RL: Reinforcement learning in which a policy takes actions in an environment over several steps and learns from the resulting rewards. Section 5.3 separates actions, returned observations, and rewards.
- AI-generated preference label: A model judge’s selection of the preferred answer in a pair. It supplies a training label, not independent proof that the answer is correct. See Section 2.1.
- Application Programming Interface (API): An interface through which a program requests a defined operation from a library or service. Section 2.1 uses the local
trl.RewardTrainerlibrary API as an example. - Baseline: A comparison value used to judge whether a sampled result was better or worse than expected.
- Batch: A group of examples processed together during one update.
- Behavior policy: The distribution that actually sampled the retained actions, including any temperature or filtering used during generation. Section 3.1 distinguishes it from the changing policy and the fixed reference.
- Benchmark contamination: Prior exposure during training to evaluation material or close variants that can distort the measured result. See Section 6.2.
- Bootstrapping: Using a later state’s value estimate within a return target. A nonterminal cutoff can retain a valid next-state value; a true terminal state has zero continuation value. See Section 3.3.
- Chain of thought: Intermediate reasoning expressed as text before the final answer. It is an output format, not proof that each stated step is correct. See Section 4.3.
- Chain rule: The differentiation rule that combines local rates of change through a sequence of dependent calculations. Section 1.3 follows how a weight changes logits and how those logits change the loss.
- Checkpoint: Saved model parameters and related configuration that can be loaded for training or evaluation. An adapter checkpoint also requires the matching base model.
- Credit assignment: The calculation that estimates which earlier actions should receive learning weight for a later result. An advantage is useful training credit, not proof that one action caused the result. See Sections 3.3 and 5.3.
- DAPO: Decoupled Clip and Dynamic Sampling Policy Optimization. A group-based method that changes clipping, prompt-group sampling, token weighting, and rewards for overlong answers.1
- Derivative: A local rate of change of an output with respect to an input. A partial derivative varies one input while holding the others fixed. See Section 1.3.
- Device: Computing hardware such as a CPU or GPU. Distributed training may coordinate several processes across several devices.
- DPO: Direct Preference Optimization. The basic offline form trains from saved preferred and rejected answers by comparing their probabilities under the changing model and an unchanged reference copy. Iterative/online DPO refreshes the pairs.2
- Entropy: A measure of how spread out a probability distribution is. Policy entropy describes variation in possible token choices, not the number of correct solution strategies. See Section 4.3.
- Equal support: Two distributions assign positive probability to exactly the same outcomes. Section 4.1 uses this condition when interpreting the sampled KL estimator.
- Evaluation: Measuring a saved model on held-out tasks without changing its parameters. The opening chapter distinguishes it from inference and training.
- Evaluation procedure: The fixed prompt split, generation settings, evaluator, cost measurements, and acceptance rules used to compare checkpoints. It is separate from the training reward. See Section 6.2.
- Evaluation sampling seed: A setting controlling random generation choices when evaluating a fixed checkpoint. Changing it tests variation in sampled answers, not variation between trained models. See Section 6.2.
- Explained variance: A critic diagnostic equal to one minus the variance of its prediction errors divided by the variance of the return targets on the measured batch. It is undefined for constant targets and does not judge answer correctness. See Section 3.6.
- GAE: Generalized Advantage Estimation. Combines reward and value estimates across token steps, with a setting that trades reliance on estimates against variability in sampled results.3
- GAE \(\lambda\): The trace parameter that controls how strongly later temporal-difference residuals contribute to a GAE estimate. Section 3.3 develops the calculation and its bias/variance trade-off.
- Gradient: The collection of local rates of change of an objective with respect to its trainable parameters. Gradient descent moves in the negative-gradient direction to seek a lower loss; gradient ascent uses the positive direction to seek a higher objective. A large step can overshoot. See Sections 1.3 and 3.4.
- GRPO: Group Relative Policy Optimization. The outcome-reward form taught here uses sibling completions for the same prompt to normalize completion rewards and assign a shared advantage to each completion’s generated tokens.4
- Held-out example: An example reserved from training and used later to test whether the saved model works on unseen material.
- Inference: Generating an answer from a prompt while model parameters stay fixed. See the opening explanation of operating modes.
- Iterative DPO / online DPO: DPO training that refreshes preference pairs using newly generated answers and new judgments. Generation, judging, and fitting may happen in separate rounds or a more integrated loop. See Section 2.4.
- JavaScript Object Notation (JSON): A text format for structured values such as objects, arrays, strings, and numbers. Valid syntax does not establish that the values answer the task correctly. See Section 2.5.
- Key-value (KV) cache: Stored attention keys and values from earlier tokens, reused during generation instead of recomputing them. Retaining more generated sequences can increase this storage. See Sections 3.1 and 4.2.
- KL divergence: Kullback-Leibler divergence. A numerical measure of how different one probability distribution is from another. Its direction matters; it is not generally symmetric.
- Large language model (LLM): A model trained on large text collections to assign probabilities to tokens and generate text.
- Logit / policy logits: Raw model scores before normalization into probabilities. For one token position, the policy returns one logit for each vocabulary entry. See Section 1.2.
- LoRA: Low-Rank Adaptation. Trainable low-rank matrix updates added to selected frozen model weight matrices.5
- Nat: The unit used for information quantities based on the natural logarithm. A log probability is the natural logarithm of a probability, not a different probability. The probability is recovered by exponentiation.
- Off-policy: Learning a return-based policy update from actions sampled by a different behavior policy, which requires explicit assumptions or correction for that distribution difference. Fitting DPO to an offline preference dataset is not automatically off-policy RL because its fixed-pair preference loss is not a sampled-return estimator. Compare Section 2.4.
- On-policy: Learning from attempts sampled by the policy being updated. PPO is commonly described as on-policy because each batch is generated by the current rollout policy and used only for a short update cycle. Once the trainable policy changes, the stored batch comes from an older policy. Section 3.5 explains the current-to-old ratio and why PPO keeps these batches recent.
- Outcome supervision: Feedback on the final result of an answer or attempt. It does not separately label the correctness of every intermediate step. See Section 4.3.
- Overlong reward shaping: An adjustment to an answer’s training reward based on its generated length near or beyond a chosen limit. It changes the score used for learning, not whether the answer is correct. See Section 4.4.
- Pass@k / pass@1: The probability that at least one of \(k\) sampled attempts is correct under the stated sampling setup; pass@1 is the one-attempt case. It does not identify which attempt is correct without a checker. Section 6.2 distinguishes the probability from its finite-sample estimate.
- PEFT: Parameter-Efficient Fine-Tuning. Methods that update a small part of a model instead of every base-model parameter.
- Policy / LLM policy \(\pi_\theta\): The LLM’s complete rule for mapping the current prompt and text prefix to probabilities over all possible next tokens. A selected-token probability is one value returned by that rule, not the policy itself.
- Policy output \(\pi_\theta(\cdot\mid s_t)\): The full next-token probability distribution returned by the policy for one given text state. Selecting one entry gives the probability used to score a particular token. See Section 1.2.
- Post-training: Further training of an already trained model to make desired behavior more likely. The opening explanation compares demonstrations, preferences, and rewards as feedback.
- PPO: Proximal Policy Optimization. Training from scored new attempts, with clipping that removes the extra incentive for some large changes in sampled-token probabilities. Clipping does not impose a hard limit on the final model change.6
- PPO epoch: One complete update pass over a retained rollout batch. Additional PPO epochs reuse the sampled data and can change the current policy. See Section 3.5.
- Preference optimization: Training that uses judgments about preferred outputs to change a policy. Section 1.1 distinguishes this feedback from demonstrations; Chapter 2 develops DPO as one method.
- Pre-trained language model: A language model whose parameters have already been learned from a text corpus before the post-training task begins. See Section 1.1.
- Pre-training: The initial training that learns a language model from a large corpus before later task- or feedback-specific training. See Section 1.1.
- Probability: A number from zero to one assigned to an outcome. An LLM policy assigns a probability to each possible next token after normalizing its logits. Section 1.2 shows that conversion.
- Process: One running program instance. Several training processes may cooperate across computing devices.
- Process supervision: Feedback on intermediate steps in an answer or attempt. Its usefulness depends on what the step labels or checks actually establish. See Section 4.3.
- Program of thought: Intermediate reasoning expressed as executable code, whose result can support the final answer. Executing code does not by itself prove that the chosen computation solves the intended task. See Section 4.3.
- Reference policy \(\pi_{\mathrm{ref}}\): An unchanged policy used to supply comparison probabilities during an update. It is distinct from the policy that sampled an online rollout. See Sections 2.2 and 3.1.
- REINFORCE: A sampled policy-gradient estimator that weights gradients of sampled actions’ log probabilities by observed returns. An optional baseline must not depend on the sampled action and is held fixed during the actor calculation. See Section 3.4.
- Reinforcement learning (RL): Learning a policy from rewards assigned to its sampled answers or actions. The opening explanation distinguishes rewards from demonstrated answers and preference labels.
- Rejection sampling: In the reasoning-data recipe, generating candidate answers and retaining those that pass the chosen check for later training. The check’s omissions can survive in the retained data. See Section 4.3.
- Return: The accumulated reward from a state or action onward, using the configured discount and terminal or truncation rule. Section 3.3 turns token rewards into return targets.
- Reward: A numerical score supplied to the learning calculation for an action or result. It reflects the scoring rule, which may miss part of the intended task. See Sections 3.1 and 5.1.
- Reward model: A learned scalar scorer trained to predict preferences over completions in their prompt context.
- Reward-model exploitation: Increasing a learned scorer’s output by using patterns it rewards, without a corresponding improvement in the intended task. See Section 2.1.
- RLHF: Reinforcement Learning from Human Feedback. In the classical recipe taught here, policy optimization uses a reward model trained from human preferences.7
- RLOO: REINFORCE Leave-One-Out. For each sampled completion, the other completions for the same prompt supply the comparison reward. Section 4.2 states the conditional-independence assumption.
- RLVR: Reinforcement Learning with Verifiable Rewards. RL driven by tests, exact answers, or other checkable outcomes.8
- Rollout: One answer or action sequence sampled from a policy and retained with the information needed to score and train from it.
- Sandbox: A restricted execution environment that limits the files, network access, or other resources available to a tool or generated program. Its actual restrictions must be enforced independently of the reward. See Section 5.2.
- SFT: Supervised Fine-Tuning. Continuation training on demonstrated prompt-response pairs.9
- Shaped token reward: A token-level learning score formed by combining task feedback with another term, such as a sampled reference-policy penalty. Section 3.2 distinguishes it from the final task score.
- Sharding: Dividing stored model parameters, gradients, or optimizer state across devices instead of retaining a complete copy on each one. See Section 3.1.
- Sigmoid \(\sigma\): The function \(\sigma(x)=1/(1+e^{-x})\), which maps a real score difference to a number between zero and one. Section 2.1 uses it for a predicted pairwise preference.
- Softmax: The operation that normalizes a set of logits into probabilities by dividing each exponentiated logit by their sum. Section 1.2 follows one next-token calculation.
- State: The information available to the policy when it chooses an action. The token examples use the prompt plus generated prefix; a tool task can also include returned observations. See Sections 3.1 and 5.3.
- Teacher-forced scoring: Scoring each recorded answer token while conditioning on the prompt and the recorded preceding tokens, rather than replacing them with newly sampled tokens. See Section 1.2.
- Temporal-difference residual: Immediate shaped reward plus the discounted next-state value estimate, minus the current-state value estimate. The next-state value is zero at a true terminal end. See Section 3.3.
- Tensor / tensors: A multidimensional array used by
PyTorchto hold token IDs, model scores, rewards, and other numerical values during a calculation.10 - Token: A text unit in the tokenizer’s vocabulary. It may be a word, part of a word, punctuation, or another encoded text piece. See Section 1.2.
- Token ID: The integer identifier assigned to one vocabulary token by the tokenizer. See Section 1.2.
- Tokenizer: Software that splits text into reusable pieces and maps those pieces to token IDs. Its configuration must match the checkpoint.11
- Training: Changing model parameters using an objective derived from demonstrations, preferences, or rewards. See the opening explanation of operating modes.
- Training objective: The numerical quantity training tries to improve. It may be written as a loss to minimize or an objective to maximize. See Sections 1.3 and 3.4.
- Training seed: A setting controlling random choices during a training run, such as data order or sampled rollouts. Changing it can produce a different checkpoint; it differs from changing evaluation sampling for one fixed checkpoint. See Section 6.2.
- Trajectory: The sequence of actions, observations, and rewards from an interaction with an environment. Section 5.3 relates this history to the model’s next action.
TRL: Hugging Face Transformer Reinforcement Learning. Trainer and objective components for SFT, DPO, reward models, and online RL.12- Trust region: A restriction intended to keep a policy update within a local neighborhood where its estimate remains useful. PPO clipping changes sampled surrogate terms but does not impose a hard trust region. Section 3.5 develops that limit.
- Value model / critic \(V_\phi\): A model that estimates the return expected from the current state. Its prediction supplies a baseline for the PPO advantage estimate; it is distinct from the model that scores a completed answer. See Section 3.1.
B.2 Software jobs in a training run
A training run must load examples, convert text into numbers the model can process, calculate how to change the model, and save the result for testing. Different libraries perform these jobs. Their interfaces must agree about the data they exchange. The training data and evaluation rules must still represent the real task.
Post-training divides into seven recurring jobs. Records become token IDs, token IDs become model scores, scores become an objective, and an optimizer changes selected weights before an independent evaluator checks the result. The following table assigns each job to concrete software and states what the project must still verify.
| Software | Job in the run | Produces or changes | What the project checks |
|---|---|---|---|
datasets / pyarrow |
Prepare records | Training examples and reserved test examples | Meaning, labels, and clean separation of the two sets |
transformers / tokenizers |
Represent and score text | Token IDs, raw token scores, and generated text | Text format and saved model fit the task |
peft |
Select small trainable model parts | Adapter parameters and saved adapter files | Which model parts can change and whether the adapter is large enough |
TRL trainers |
Run the chosen objective | Training values, batches, and saved run progress | Feedback and measurements match the method |
PyTorch |
Calculate and apply updates | Tensors, gradients, and optimizer progress | Numerical behavior and objective are valid |
accelerate |
Arrange work across devices | Models, data, and update code placed on hardware | Process launch and hardware setup |
| Evaluator / verifier / environment | Check behavior and return results | Scores, tests, observations, failure records | Hidden coverage, safety, and task success |
A training run uses the following operations. Batches repeat the update steps; candidate checkpoints are saved and evaluated at the selected intervals and before acceptance.
- Load evidence:
datasetsor project code selects a training batch while preserving a disjoint held-out split.13 - Prepare answers: The checkpoint tokenizer and chat template convert records into token IDs and masks. SFT and DPO use saved answers. Online methods first generate and record new answers from the prompts while keeping the model fixed during that sampling step.
- Score with each model:
transformersandPyTorchproduce the active model’s token scores, called policy logits. Depending on the method, an unchanged reference copy provides comparison probabilities, a scorer supplies rewards, and a value model estimates the return expected from the current text state. - Construct training values:
TRLor explicitPyTorchcode turns those outputs into token losses, pair margins, rewards, returns, or advantages. - Differentiate:
PyTorchautomatic differentiation calculates gradients for trainable base or adapter parameters that participate in the differentiable loss graph.14 - Apply and coordinate: The optimizer changes those parameters, while
acceleratecan coordinate supported processes and devices.15 - Save and test: The saving code records the model state. An independent evaluator or verifier measures held-out behavior before the checkpoint is accepted.
B.3 Low-rank adapters and trainable parameter counts
Updating every model parameter can require more optimizer memory and checkpoint storage than a project can afford. Section 2.3 uses adapters to restrict which parameters change while retaining the same DPO loss. The calculation below shows what an adapter adds to one matrix and how to count its trainable values.
DPO determines the loss, while Low-Rank Adaptation (LoRA) determines which model values can change. LoRA leaves a layer’s weight matrix \(W\), often called a projection, unchanged and adds the trainable correction \(\Delta W\), scaled by the configured adapter strength. The next equation quantifies that correction before the parameter-count example compares its size with the dense matrix.16
\[ W^{′} = W + \Delta W; \Delta W = \frac{\alpha_{\mathrm{LoRA}}}{r}BA \tag{B.1}\]
Variables
- \(W\): Frozen weight matrix with \(d_{\mathrm{out}}\) rows and \(d_{\mathrm{in}}\) columns.
- \(W'\): Effective weight matrix used after the adapter update is added.
- \(\Delta W\): Additive low-rank update created from the two adapter matrices.
- \(A\): Trainable matrix with shape \(r\times d_{\mathrm{in}}\).
- \(B\): Trainable matrix with shape \(d_{\mathrm{out}}\times r\).
- \(r\): Small adapter rank.
- \(\alpha_{\mathrm{LoRA}}/r\): Configured update scale. In the shown setup, \(32/16=2\).
Mechanism: The product \(BA\) has the same shape as \(W\). Scaling it produces \(\Delta W\), which is added to the frozen weight matrix. Interpretation: Change a small set of adapter parameters without creating labels, rewards, or a policy objective.
Example: LoRA reduces the number of trainable values in one layer
A square 4096 by 4096 weight matrix receives a rank-16 LoRA adapter with its scale set to 32.
Inputs and measurement
The layer dimensions come from the model configuration. The settings \(r\) and \(\alpha_{\mathrm{LoRA}}\) come from LoraConfig. Count \(A\) and \(B\) instead of the frozen \(W\) values, and compare the two counts.
Calculation
The configured adapter scale is \(32/16=2\). The two trainable matrices together contain
\[16(4096+4096)=131{,}072\]
parameters, while the frozen dense weight matrix contains
\[4096\times4096=16{,}777{,}216.\]
The trainable fraction is therefore \(131{,}072/16{,}777{,}216=1/128\).
Conclusion: The two adapter matrices contain 131,072 trainable values, one 128th as many as the 16,777,216 values in the dense weight matrix. Actual model savings depend on how many matrices receive adapters.
The base model still occupies memory, and forward and backward computation still pass through its layers. The parameter-count reduction is therefore not the same as an equal reduction in total training memory or elapsed time. LoRA develops this low-rank update and reports experiments with selected projection matrices.
B.4 PPO notation in the source slide
RL-LLM-Session-2.pdf, page 61, uses a different label for the clipped objective taught in Section 3.5. Its \(L^{\mathrm{CLIP}}\) is an objective to maximize, corresponding to this guide’s \(J_{\mathrm{PPO}}\), rather than a loss to minimize. Its ratio divides current by old-policy probability for the same sampled token and prefix. The hat marks an estimated advantage held fixed during the update.
The slide says clipping “stops rewarding the optimizer once the ratio leaves a band.” That statement needs the advantage sign. With a positive advantage, only an increase beyond the upper bound loses further objective benefit. With a negative advantage, only a decrease below the lower bound does so. Movement in the harmful direction still affects the objective, and clipping never forcibly returns the model probability to the band. The worked positive and negative cases in Section 3.5 make this distinction explicit.17
Yu, Q., et al. (2025). DAPO: An open-source LLM reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38. DOI: 10.52202/085713-3775. Sections 3.1–3.4. These are separate controls, not a guarantee of correct reasoning.↩︎
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023). Direct Preference Optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. DPO objective. Standard offline objective with a fixed reference.↩︎
Schulman, J., Moritz, P., Levine, S., Jordan, M. I., & Abbeel, P. (2016). High-dimensional continuous control using generalized advantage estimation. International Conference on Learning Representations. Advantage-estimation equations. Bias and variance depend on discount, trace parameter and value estimates.↩︎
Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv:2402.03300). Section 4. Removing the learned critic does not remove grouped-rollout storage.↩︎
Hu, E. J., et al. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations. Section 4.1, equation 3. Savings in trainable parameter count are not equal savings in total memory or runtime. The 4096-by-4096 rank-16 count in the adapter example is worked arithmetic using illustrative dimensions, not a benchmark result.↩︎
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. Section 3, equation 7 and Figure 1. The bound applies to a surrogate term, not directly to the resulting policy. The numerical examples illustrate the formula, not measured training behavior.↩︎
Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744. DOI: 10.52202/068431-2011. Methods. Supervised fine-tuning stage. This describes the classical reward-model recipe, not every method using human feedback. Examples and task distribution determine the supported behavior.↩︎
Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training (arXiv:2411.15124, v5). RLVR stage. A verifier checks only the conditions encoded by its tests.↩︎
Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744. DOI: 10.52202/068431-2011. Methods. Supervised fine-tuning stage. This describes the classical reward-model recipe, not every method using human feedback. Examples and task distribution determine the supported behavior.↩︎
Paszke, A., et al. (2019). PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32. Sections 2 and 5.1. Section 4.3. This describes the library’s numerical representation. The user must still supply the intended differentiable objective.↩︎
Wolf, T., et al. (2020). Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 38–45). Association for Computational Linguistics. DOI: 10.18653/v1/2020.emnlp-demos.6. Section 3, Figure 2. Tokenizer and model head must match the checkpoint.↩︎
Hugging Face. (2025). TRL: Transformer Reinforcement Learning (v0.22.2 documentation). Version 0.22.2. Official API documentation. Installed versions can differ.↩︎
Lhoest, Q., et al. (2021). Datasets: A community library for natural language processing. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 175–184). Association for Computational Linguistics. DOI: 10.18653/v1/2021.emnlp-demo.21. Section 3, S1–S4. Splitting functions do not by themselves prove absence of leakage.↩︎
Paszke, A., et al. (2019). PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32. Sections 2 and 5.1. Section 4.3. This describes the library’s numerical representation. The user must still supply the intended differentiable objective.↩︎
Hugging Face. (2025). Accelerate (Accelerate v1.10.1 documentation). Version 1.10.1,
Acceleratorandprepare. Versioned API guidance, not a throughput guarantee.↩︎Hu, E. J., et al. (2022). LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations. Section 4.1, equation 3. Savings in trainable parameter count are not equal savings in total memory or runtime. The 4096-by-4096 rank-16 count in the adapter example is worked arithmetic using illustrative dimensions, not a benchmark result.↩︎
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. Section 3, equation 7 and Figure 1. The bound applies to a surrogate term, not directly to the resulting policy. The numerical examples illustrate the formula, not measured training behavior.↩︎