Part II: Serving answers and assessing their quality

A serving system must keep model requests within its time and memory budgets. For the policy-answering service, the application also needs current passages the user may read. Operators then need evidence of whether the model used those passages correctly.

Chapter 4 establishes the request path and release controls. Chapter 5 builds the retrieval path that supplies authorized evidence. Chapter 6 follows requests across components and detects changes in inputs and outputs. Chapter 7 uses recorded experiments and quality judgments to compare candidate changes and check the evaluator itself. Shared request and release identifiers connect these records, so a wrong answer can be investigated across components.

Chapters in this part

  • Serving LLMs with predictable performance: Users of the grounded-answer service expect the first words of an answer within about a second and a steady stream after that. Part I produced a recoverable model checkpoint. A serving engine loads that saved model and handles independent requests of different lengths, latency targets and per-request attention memory, while the fleet introduces new versions through rolling releases.
  • Retrieving evidence for grounded answers: Suppose the remote-work policy changes after the model served in Chapter 4 was trained. Its weights retain what training learned, so it may still produce a fluent answer based on the old policy. The grounded-answer service finds current passages the user is allowed to read and supplies them with the question. Retrieval quality determines whether the right passages arrive, and answer evaluation checks whether the model uses them faithfully.
  • Tracing service behavior and detecting change: A grounded-answer service can return HTTP status 200 and still tell an employee the wrong remote-work rule. Serving measurements and retrieved passages alone do not explain which operations produced that answer or whether the questions users ask have changed.
  • Comparing experiments and evaluating model quality: A candidate model, prompt, or retrieval change may improve an average score while making important cases worse. A release decision needs comparable runs and records that show which inputs produced each result.