Follow a request
Carry a trace identifier across the application, retrieval layer, model request and tool calls. The useful unit of investigation is often the whole workflow, not an isolated model response.
Measure the tradeoffs
Latency, token use and failure rates help explain system behaviour. Pair those operational measurements with evaluation of answer grounding and tool correctness. An inexpensive response is not useful if it cannot complete the task.
Keep sensitive data out
Logs can become another copy of private user data. Decide what to capture, redact sensitive fields and set appropriate access and retention rules. Collect enough context to investigate without recording everything by default.
01Understand
02Connect
03Evaluate
04Operate
All engineering notes