Part II - Evaluation-based system improvement
An LLM response can follow the requested format and still contain incorrect information or fail to answer the question. Evaluating response quality requires representative test questions and clear criteria for judging the answers.
A useful comparison starts with a definition of success and fixed test cases, including important edge cases. The proposed system is compared with a chosen reference system on the complete set and on meaningful groups of questions. When answers have several valid forms, human reviewers or a separately prompted model use written grading criteria and rules for assigning scores. The release decision must also account for uncertainty in these measurements.
Chapter 4 first establishes evaluation criteria and test cases, then explains how metrics, benchmarks, model judges, human ratings, and uncertainty estimates measure answer quality. Chapter 5 uses that evidence to compare changes, diagnose failures, and decide whether a product meets its release requirements.
