5 Evaluation-based system improvement
A higher evaluation score does not by itself justify changing a product. The change must meet requirements for its users, remain within operating limits, and avoid unacceptable failures. Evaluation-driven development (EDD) is the book’s engineering workflow for treating every proposed improvement as an experiment with fixed test inputs, defined scoring rules, an estimate of uncertainty, and a release decision. The NIST AI Risk Management Framework1 also places measurement inside lifecycle risk management. The name and exact EDD loop are the book’s synthesis, not an external standard. The front matter maps each step to the NIST framework and related ISO/IEC standards.
The map connects product requirements to comparisons with the current system, run records, failure diagnosis, and release checks. An isolated score needs the cases tested and the costs of failure before it can support a decision. The final case combines these records and checks for a product-description comparison.
5.1 Product requirements and evaluation criteria
Chapter 2 defined a task specification for one request. A product needs requirements before that: what it must achieve for its users and organization, and what it must never do. A product requirement is a verifiable statement of required or forbidden system behavior under stated conditions, traced to a product outcome. “Escalate every ticket that needs a person” is a requirement. “Use the largest available model” is a design choice.
Requirements come from the business value the product must deliver, its users and their tolerance for delay or error, domain experts, legal and policy obligations, and the process the product replaces. The NIST AI Risk Management Framework asks organizations to document business value, risk tolerance, and system requirements gathered from the relevant people (MAP 1.4 to 1.6). It also asks them to examine the costs of expected errors (MAP 3.2).2
A requirement becomes testable through a chain: product outcome, requirement, measurement, threshold, and the evaluation cases that exercise it. The offline measurement usually serves as a proxy for the product outcome. Recall on labeled escalations is the fraction of tickets requiring escalation that the system correctly escalates. It measures whether those customers are routed to a person. Measuring the delay requires a separate timing check. Section 14.4 describes how production data tests whether a proxy predicts the outcome.
Table 5.1 uses p95 latency, the time at the 95th percentile of measured request completion times. The fraction that finishes within it depends on tied timing values and how the percentile is calculated. A p95 limit permits slower requests and cannot establish an every-request limit. Section 5.5 uses it in a release decision.
| Product outcome | Requirement | Measurement | Threshold source |
|---|---|---|---|
| Urgent customers get a person without delay | Miss no more required escalations than staff do | Recall on the escalation slice | Current manual triage |
| Triage costs less than manual handling | Stay within the cost per ticket | Model cost plus human-review cost per ticket | Product budget |
| Customer data stays private | Never show one customer another customer’s data | Cross-customer disclosures in cases designed to expose another customer’s data (any disclosure blocks release) | Legal and policy obligation |
| Customers get answers while they wait | Keep p95 latency within the limit | p95 latency on representative traffic | Measured user tolerance |
A threshold value needs a source that exists before any candidate is scored: the performance of the current process, the product budget, a legal obligation, or a measured user tolerance. A release gate is the set of score limits and mandatory checks a candidate must pass before release.
Example 5.1: Thresholds from the current process and the budget
This example uses a different requirement from escalating every ticket that needs a person: the system must miss no more required escalations than staff do. An audit of 200 manually triaged tickets finds 40 required escalations. Staff missed 2 of them, a recall of 38 / 40 = 0.95, and manual triage costs $3.00 per ticket.
- The system may miss no more escalations than staff do, so recall among escalation cases must be at least 0.95.
- The product goal halves the triage cost to $1.50 per ticket. A model call costs $0.02, and each ticket sent to human review costs $3.00.
- With review rate \(r\), the cost per ticket is \(0.02 + 3.00r \le 1.50\), so \(r \le 1.48 / 3\), about \(0.4933\).
Result: The release gate requires escalation recall of at least 0.95 and a human review rate of at most 49 percent, rounding the cost-derived ceiling down conservatively.
Interpretation: The staff audit supplies an observed 0.95 reference, not a measurement of the candidate system. Forty escalation cases also leave uncertainty about staff performance on other tickets. The candidate needs its own evaluation results to show whether it meets the chosen floor. Section 4.8 explains how limited test sets affect the uncertainty estimate.
Requirements for open-ended outputs often change after people read real outputs. Shankar et al. call this criteria drift: people need criteria to grade outputs, and grading outputs helps them define the criteria.3 The requirement list is versioned with the test collection, and a changed requirement starts a new evaluation version (Section 4.2).
Requirement traceability links each requirement to its measurement, threshold, evaluation cases, and results. With stable requirement identifiers, a release decision record can list each requirement with its result: passed, failed, or not tested. Section 14.3 shows how evaluation results and blocking failures support PASS or HOLD. A veto requirement is mandatory: its failure blocks release, producing HOLD. The evaluation rules and cases from Sections 4.1 and 4.2 make these requirements testable. The next section uses them to compare a baseline with a changed system.
5.2 Baseline comparisons
In the default EDD comparison, the versioned baseline and a candidate with one changed factor run on the same golden set. Both use the same scoring procedure and otherwise fixed settings in comparable environments. The measured difference then estimates the effect of that change rather than the combined effect of unrelated changes.
EDD adapts test-driven development, in which tests are written before the code they check, to systems with probabilistic outputs and imperfect proxies. EDD follows these steps:
- Define the behavior and observable final state the product must achieve, with its measurements, thresholds, and release decision rule fixed before candidate scoring.
- Create and version representative cases and risk-focused slices before tuning.
- Run the current system on the fixed set and store outputs, traces, latency, cost, and version metadata.
- Change one model, prompt, retrieval, workflow, or policy factor. Use this as the default comparison method. When two factors may interact, run a planned multi-factor experiment rather than assuming the one-factor result will generalize.
- Rerun the same evaluation, estimate uncertainty, and inspect failures.
- Release, revise, or revert according to the decision rule. Add validated new failure cases to the next version of the suite, not to the completed comparison.
Defining the eval before inspecting a preferred output helps it measure the underlying behavior across cases.
Reproducing a score requires the test inputs, system version, scoring rules, and relevant run conditions. Traces help diagnose failures. The decision threshold is needed to judge whether the score meets the release criterion.
The fixed set keeps ordinary, boundary, adversarial, and production-derived cases visible in the comparison. Validated production failures can become regression cases after privacy review and label validation. Duplicate or near-duplicate cases must be controlled across development and held-out partitions, and each case record must retain its expected outcome, rubric version, source record, and label review status.
5.3 Reproducible run records
Reproducing a comparison requires a record of the system and conditions that produced every case result. A run configuration records the case-set revision, model, prompt, context policy, decoder settings, and runtime environment. When used, the record also includes the retrieval index, tool versions, and judge versions. Cost reports need the price version too. It records any random seed set by the application or exposed by the service.
Each result row stores that configuration beside one case’s raw output, parsed output, validation result, trace, latency, token use, tool calls, citations, and evidence of the system’s final outcome. Unused fields are marked as not applicable, and unavailable values as unavailable. Together these records let a reviewer inspect the baseline and support rerunning it when the recorded versions and conditions remain available.
Pineau et al.4 recommend preserving code, data, checklists, and experimental conditions. The field list here is the book’s design.
See Example 5.2 for the implementation.
Example 5.2: An evaluation harness keeps configuration and per-case evidence together.
In the shortened harness below, case supplies the ID, slice, and input. config is a dataclass instance containing the fixed run settings, and asdict converts it to a dictionary. system accepts the input and configuration and returns an object with the output, validation, trace, and usage fields accessed in the code. perf_counter supplies the elapsed-time readings. The harness combines those values into one row. It does not define the system under test, and it records only calls that return those fields successfully. The calling runner invokes it for each case and trial, adds a trial identifier, and marks unavailable or inapplicable fields. It must record exceptions and timeouts as failed trials rather than omit them from the results.
from dataclasses import asdict
from time import perf_counter
def run_case(case, system, config):
started = perf_counter()
response = system(case.input, config=config)
elapsed_ms = 1000 * (perf_counter() - started)
return {
"case_id": case.id,
"slice": case.slice,
"config": asdict(config),
"raw_output": response.raw_output,
"parsed_output": response.parsed_output,
"validation": response.validation,
"trace": {
"tool_calls": response.trace.tool_calls,
"citations": response.trace.citations,
"final_state_evidence": response.trace.final_state_evidence,
},
"input_tokens": response.usage.input_tokens,
"output_tokens": response.usage.output_tokens,
"latency_ms": elapsed_ms,
}Example 5.3: One harness call produces one reviewable row
Case refund-017 belongs to the ambiguous_policy slice and contains a question about whether an opened item is returnable. Configuration prompt-v3/model-a/temp-0 is passed unchanged to the system. The returned row keeps that configuration beside the raw answer, parsed decision escalate, validation result pass, input and output token counts, and measured latency. A later metric calculation can use the parsed decision, while an error review can still inspect the original answer and trace.
The harness only preserves observations. Metrics are calculated from those observations and compared with the thresholds fixed before candidate scoring. When retrieval or model sampling can vary, the same case is run several times and each trial remains a separate record.
5.4 Error analysis
An uncertain or disappointing result needs a diagnosis before another change is tested.
Error analysis groups failures by cause, such as missing evidence, prompt misunderstanding, misapplied category definitions, output truncation, judge disagreement, policy conflicts, or tool errors. Each bucket should point to the responsible system component and a proposed next experiment.
| Failure bucket | Evidence to inspect | Next controlled change |
|---|---|---|
| Task ambiguity | Rater disagreement and prompt text | Success-criteria clarification or task split |
| Context omission | Serialized request and selected evidence IDs | Repair retrieval or context policy |
| Model reasoning or generation | Same context across saved model versions | Model or prompt comparison with evidence fixed |
| Validator or grader error | Raw output, parsed fields, hidden tests, judge rationale | Correct the evaluator before tuning the system |
| Slice regression | Per-slice delta and representative failures | Add validated cases to the next suite version, rerun both systems, and report each slice’s measured change |
Error analysis follows the uncertainty estimates. A failure pattern missing from the suite can supply a new case for its next version after source and label checks. Existing failed cases remain available for regression testing.
Suppose descriptions omit care instructions. The saved request contains the product name and material but omits the source record’s care field. The trace points to missing context. A separate experiment would test whether the model follows the instructions. The next experiment adds the care field while keeping the model, prompt template, cases, and evaluator fixed. The rendered request changes because it now includes that field. A reduction in omissions supports that hypothesis. The rerun also checks factual support and other slices because the added text can change more than completeness. If omissions remain, the reviewer examines the new trace and plans a test of another cause rather than treating the first diagnosis as settled.
The eval suite changes when new model behavior or production failures reveal a missing case. Stability means reproducible decision logic, not a frozen dataset forever.
5.5 Release thresholds and decisions
Chapter 4 supplied the metric and uncertainty estimate. The comparison and diagnosis now supply the evidence for a decision. A quality gain can still violate latency, cost, or safety limits.
A release gate can require minimum quality gains and prevent critical-slice regressions, while capping p95 latency, task cost, and policy violations. The p95 latency is the 0.95 quantile of measured request completion times, with the empirical quantile convention recorded for finite samples.
Example 5.4: A release gate combines requirements
Consider a candidate with a 95 percent paired bootstrap interval of \([0.01, 0.07]\) for its mean quality difference from the baseline, critical-slice success of 93%, p95 latency of 760 ms, and cost of $0.018 per task. Positive differences favor the candidate. The release rule requires a positive lower interval endpoint, at least 95% critical-slice success, p95 latency no greater than 800 ms, and cost no greater than $0.020.
- The interval, latency, and cost pass their stated requirements.
- The critical slice fails: 93% is below 95%.
Result: Hold the candidate. An average quality improvement does not cancel a failed critical slice.
These values are illustrative. A production gate must state its mandatory checks before examining the candidate results.
A release decision needs both an estimated change and defined limits for every critical slice, cost, latency, and policy condition. The next example combines the measurements from Chapter 4 with these comparison and release rules.
5.6 A product-description evaluation
Product descriptions have no single reference answer, so their evaluation combines a task specification, scoring methods, calibration, uncertainty, and failure analysis.
The evaluation defines each criterion and the rule for passing all required criteria. A baseline and candidate configuration then produce descriptions for every product while the application records token use and latency. The application calculates generation cost from usage and recorded prices for every product. Human raters label outputs against the rubric. A comparison can change the prompt while keeping the generation model fixed, or change the model while keeping the prompt fixed. The evaluation records that choice and its hypothesis, calibrates a model judge, and compares its criterion-level decisions with human labels.
Example 5.5: A product-description comparison leads to HOLD
Suppose the source record describes a 450 ml stainless-steel travel mug whose lid is dishwasher-safe and whose body must be hand-washed. It makes no leak-proof claim. A description passes only if all three rubric criteria pass: factual support, all required care instructions, and clear wording. The release rule separately requires no unsupported claims among products in the safety-sensitive slice, defined for this example as drinking containers. Both rules are fixed before comparison.
With the generation model fixed, the baseline prompt produces 450 ml stainless-steel travel mug. Dishwasher-safe. The candidate prompt includes a care-details instruction, and the model produces 450 ml leak-proof stainless-steel travel mug. Dishwasher-safe lid; hand-wash body. The candidate fixes the care error but adds an unsupported claim, so human reviewers still mark factual support as failed.
For this small, assumed comparison, human raters label both versions of all 40 held-out products after judge calibration on separate outputs. The final human labels give the candidate 34 passing descriptions and the baseline 30, while two of the five drinking containers have unsupported claims in the candidate descriptions. On factual support, the calibrated judge agrees with the final human labels for 36 of the 40 candidate descriptions. Reviewers inspect the four disagreements. Decision: HOLD because the safety-sensitive slice still contains unsupported claims, even though the aggregate pass count improves. These assumed counts illustrate the decision rule. They are not measured product results.
A reproducible evaluation preserves four concrete records: the rubric and pass rule, the executable prompts and model calls, row-level manual and automated scores, and the experiment log. A prose conclusion without the configuration and per-example evidence cannot show whether an apparent improvement came from the model, prompt, evaluator, or changed data.
See Example 5.6 for the implementation.
Example 5.6: A row-level evaluation record shows where each generation and judgment came from.
The following shortened listing assembles one result row from application-supplied variables. It does not generate or grade a description. usage supplies token counts and estimate_cost applies the supplied prices. The run configuration stores the case-set and price versions and sampling settings. Each row also needs the measured latency and whether it belongs to judge calibration or the held-out comparison. This shortened listing omits those row fields.
record = {
"product_id": product_id,
"input_text": product_text,
"generation_model": generation_model,
"prompt_version": prompt_version,
"generated_description": description,
"input_tokens": usage.input_tokens,
"output_tokens": usage.output_tokens,
"estimated_cost_usd": estimate_cost(usage, prices),
"human_labels": human_labels,
"judge_model": judge_model,
"judge_rubric_version": rubric_version,
"judge_labels": judge_labels,
"judge_explanations": judge_explanations,
}The generation model and judge model have different roles and must be recorded separately. Per-criterion comparison is more informative than one final agreement score because a judge may be reliable on clarity but poor on factual support. The improvement cycle is valid only when the rubric and test rows remain fixed or their changes are documented.
Chapter conclusion
Evaluation distinguishes a system improvement from a persuasive anecdote. Chapters 6 and 7 apply the same discipline to retrieval: first identify whether the right evidence was found, then whether generation used it correctly.
Tabassi, E. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1. The framework supports lifecycle risk governance and measurement.↩︎
Tabassi, E. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-1. MAP 1.4 to 1.6 cover business value, risk tolerance, and requirements gathered from relevant people, and MAP 3.2 covers the costs of expected errors. The framework is voluntary and does not prescribe metrics or thresholds.↩︎
Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (pp. 1–14). Association for Computing Machinery. https://doi.org/10.1145/3654777.3676450. The observation comes from a qualitative study of an evaluation-assistant tool. ↩︎
Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d’Alché-Buc, F., Fox, E., & Larochelle, H. (2021). Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program. Journal of Machine Learning Research, 22(164), 1–20. https://jmlr.org/papers/v22/20-303.html.↩︎
