13 Logging system execution
An answer or final tool result does not show how the application produced it. Logs preserve the inputs, decisions, outputs and failures needed to investigate that execution. Chapter 12’s cases show the request paths whose stages need logs.
13.1 What to log at each stage
A useful log lets a reviewer connect an outcome to the operations that produced it. Each request needs a shared identifier, with its steps and attempts identified separately. Nested or parallel work also needs parent or dependency links because timestamps alone do not show those relationships. Application code records event time, operation, component version, outcome and failure reason where applicable. Elapsed time belongs beside the operation it measures. These stage-specific fields apply established logging practices to the book’s examples. They are application choices. Some need logging code beyond the short examples.
| Stage or operation | What to log | What the log helps establish |
|---|---|---|
| Source preparation | Source ID/version or hash; parser and chunker versions; document/page/chunk IDs; embedding revision and normalization; index build ID. | Reconstruct the source preparation and index used for a request. |
| Prompt and context assembly | Prompt version and output schema; permitted final messages or protected references; token budget; retained and omitted input IDs. | Identify the information and instructions actually submitted. |
| Retrieval and ranking | Query and filters; index/model versions; ordered candidate IDs, score types and scores; reranking and final selection. | Separate missing evidence from evidence discarded after retrieval. |
| Model call | Requested and served model/version where available; decoding settings; permitted request/response; usage, duration, finish status and error. | Compare the submitted request, returned answer and call conditions. |
| Workflow step | Step/node and attempt; permitted state before/after or field changes; validation decision; next transition; retry or failure. | Find which step changed state or selected a branch. |
| Tool execution | Tool/version; validated arguments; access decision and approval when required; attempted action; result or error; affected state. | Distinguish a model proposal from the action actually attempted. |
| Agent loop | Ordered decisions, tool calls and observations; attempt budget; stop reason; checked final outcome. | Explain how the loop reached its result or stopped. |
| Memory use and updates | Retrieved IDs and versions; source references; eligibility checks; promotion/correction decisions; expiry and deletion outcome. | Identify which past information was used or changed. |
| Evaluation | Case and dataset versions; expected behavior and slice; rubric/judge versions; measured scores and explanations; failures and unanswered cases. | Recompute a comparison under the same evaluation rules. |
| Final outcome | Returned answer or authoritative state; validation and policy checks; aggregate latency/usage; cost basis where measured. | Check task completion and connect it to the contributing execution. |
These rows describe operations, rather than one mandatory sequence. Memory can be read before request assembly, and an agent can repeat retrieval and tool calls. The application logs only stages it uses and keeps each attempt distinct, including rejected, failed and timed-out attempts. A log containing only successful returns cannot explain the failed requests omitted from it.
13.2 Established standards and practices
An event log describes an occurrence, such as a rejected tool argument. A trace links the operations belonging to a request, including nested or repeated steps. An evaluation log adds the test case, expected result, rubric and scores. Together they connect runtime behavior to the comparisons developed in Chapters 4, 5 and 7.
Services need shared ways to identify related work. Applications also need rules for what to log and how to protect it. These standards and practices address those needs:
- W3C Trace Context standardizes the HTTP headers
traceparentandtracestateused to pass trace context between services. This lets participating services associate their work with the same trace.1 - OpenTelemetry defines a stable data model for logs, with timestamps, log levels, event details, extra fields and optional trace-context fields.2 A span describes one operation, with its start and end times and associated information. Spans link through parent and trace identifiers to show related work.3
- OWASP’s Logging Cheat Sheet gives application logging guidance: capture enough information about when an event occurred, where it occurred, who initiated it and what happened. It includes validation, authorization and execution failures, and guidance on protecting the logs.4
These sources establish shared formats and logging practices. The table in Section 13.1 adds the domain information needed here, such as chunk IDs, candidate ranks and validation outcomes. Those fields must have documented meanings in the application. A shared trace identifier alone does not explain which passages entered a model request or whether a tool action was permitted.
13.3 Logging tools
Application code must log the stage information needed for diagnosis. A library cannot infer application details it has not received. A configured library integration may log model calls automatically. Application checks and state changes need additional logging code. The examples below cover event logging, tracing within and across services, and tracking evaluation runs with their settings and scores.
| Tool | What it provides | What the application still supplies |
|---|---|---|
logging with a configured formatter |
Python event logging to outputs such as the console or files. Extra fields can accompany log entries. | The event schema, request/step IDs, permitted payloads and output format. JSON output needs serialization or a JSON formatter. |
| OpenTelemetry5 | APIs and libraries for collecting logs and linked spans, plus software that sends them to a service for storage and analysis. | Application-specific fields, passing trace context between services, settings for sending the logs and spans, and a service for storage and analysis. |
| Langfuse6 | LLM-focused traces with linked observations for generation, retrieval and tools. | Additional logging code for application checks and fields an integration does not capture; permitted inputs and outputs. |
| MLflow7 | Experiment/run tracking with parameters, metrics and saved files (artifacts). | Evaluation definitions and results, run metadata, and the permitted log files to attach. |
For a small script, structured local logs can be sufficient. Section 12.7 writes one JSON object per request on a separate line in traces.jsonl. Each object contains session_id, question, latency_ms and the current request’s trace. A session can contain several requests, so a separate request identifier is needed to link each one’s logs. The example preserves the request steps, but a complete logging setup also needs its version information and a way to retain partial logs when a helper fails.
For work across services, application code passes trace context and records spans to connect model, retrieval and tool operations. A tracing service can then display the timing and relationships between operations. For repeated evaluation runs, application code keeps parameters and scores linked to the permitted log files used to calculate them. The same request identifier can appear in the application log, its trace and its evaluation result.
These optional tools serve different logging needs. Their interfaces and integrations depend on the version. The earlier examples do not depend on these tools.
13.4 Using logs to investigate a failure
For the RAG example in Section 6.7, follow the candidate IDs into the selected passages and then into model_request. If a supporting passage was retrieved but omitted from that request, inspect selection and request assembly. If it entered the request but the answer contradicts it, inspect generation and answer validation. Section 7.1 develops these distinctions. A final answer and a search-success message alone cannot separate them.
For the workflow in Section 8.3, inspect the inputs, validation decisions and transition at each step. For a tool call, distinguish a proposed action from an attempted action and its actual result. For an agent, preserve the ordered attempts and stop reason. A failure in a helper needs a partial log even when the main function cannot return its normal result.
Decide which data may be stored in the logs, who may read the logs and how long they remain available. Exclude credentials and other prohibited data. Where full text cannot be logged, preserve permitted identifiers, versions and protected references. A hash can check whether data matches, but cannot recover its contents. OWASP also recommends testing logging failures and restricting access to the logs.8
Logs support investigation; they do not themselves establish that an answer is correct or an action is authorized. Chapter 14 uses the logged execution and its evaluation results as evidence for release decisions.
W3C. (2021). Trace Context, W3C Recommendation, 23 November 2021. https://www.w3.org/TR/2021/REC-trace-context-1-20211123/.↩︎
OpenTelemetry. (n.d.). Logs data model (Specification 1.61.0). Retrieved October 16, 2026, from https://opentelemetry.io/docs/specs/otel/logs/data-model/.↩︎
OpenTelemetry. (n.d.). Traces. Retrieved October 16, 2026, from https://opentelemetry.io/docs/concepts/signals/traces/.↩︎
OWASP Foundation. (n.d.). Logging cheat sheet. OWASP Cheat Sheet Series. Retrieved October 16, 2026, from https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html.↩︎
OpenTelemetry. (n.d.). Components. Retrieved October 16, 2026, from https://opentelemetry.io/docs/concepts/components/.↩︎
Langfuse. (n.d.). Observability data model. Retrieved October 16, 2026, from https://langfuse.com/docs/observability/data-model.↩︎
MLflow. (n.d.). REST API. Retrieved October 16, 2026, from https://mlflow.org/docs/latest/api_reference/rest-api.html.↩︎
OWASP Foundation. (n.d.). Logging cheat sheet. OWASP Cheat Sheet Series. Retrieved October 16, 2026, from https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html.↩︎