Agent Observability Spec Skill
You can't fix what you didn't record. For LLM systems the unit of observability is the trace — everything the model saw and did — because behaviour, not uptime, is what fails. This skill specifies what to capture, what to compute from it, and when to page someone.
What This Skill Produces
- A trace schema: per-request spans and the fields each must carry
- Metric definitions across health, quality, cost, and behaviour — each with a threshold and owner
- A sampling and retention policy that keeps cost sane and debugging possible
- A privacy note: what logged content contains, who can see it, and how long it lives
Required Inputs
Ask for (if not already provided):
- The system's shape — single LLM call, RAG pipeline, or multi-step tool-using agent
- Traffic volume and cost sensitivity — full tracing at 10M req/day is a budget decision
- What "misbehaving" means here — the two or three failure modes that matter most (wrong facts? wrong actions? cost? refusals?)
- Existing observability stack (Datadog, Langfuse, OTel, homegrown) — spec into it, not around it
Trace Schema
Every request produces one trace; every model call, retrieval, guardrail check, and tool execution is a span. Minimum fields:
| Span | Must capture |
|---|---|
| Request root | request id, user/session (pseudonymous), feature + prompt version, model id, total tokens, total cost, latency, terminal status |
| Model call | full input context (or content-addressed ref), output, finish reason, tokens in/out, cached-token share, temperature |
| Retrieval | query, top-k ids + scores, which chunks entered the context |
| Tool call | tool name, arguments, result (or ref), duration, error |
| Guardrail | check name, verdict, and what it did (blocked / rewrote / flagged) |
| User signal | edits, regenerates, thumbs, abandonment — joined to the trace id |
The test of the schema: an engineer can replay any incident from its trace alone (see agent-incident-postmortem).
Metrics and Alerts
Define four families; every metric gets a threshold, a window, and an owner.
- Health — error rate, p50/p95 latency, timeout rate, provider 429/5xx rate. Page on these.
- Cost — cost per request (p50, p99), tokens per request, cache hit rate, daily spend vs. budget (pair with
llm-cost-latency-budget). Alert on p99 and daily-budget burn — cost incidents are caused by the tail, not the mean. - Quality proxies — format/schema violation rate, refusal rate, groundedness-check failure rate, judge score on a sampled slice, regenerate/edit rate. Alert on drift vs. a rolling baseline: absolute thresholds go stale, deltas don't.
- Behaviour (agents) — steps per task, tool-error rate, loop detection (same tool + same args N times), unauthorised-action attempts caught by guardrails. Page on the last one.
Sampling & Retention
- Metadata for 100% of requests (ids, versions, tokens, cost, status) — this is cheap and non-negotiable.
- Full content traces: 100% for errors, guardrail hits, and negative user signals; [1-10]% random sample for the rest, adjusted to volume.
- Retention: full content [30-90] days, metadata [12+] months for trend baselines; incident traces pinned indefinitely.
- Privacy: logged context contains user data — state where it lives, who has access, how deletion requests reach it, and that traces are scrubbed or access-gated before wide sharing.
Output Format
Observability Spec: [feature/agent]
System shape: [calls/pipeline/agent] · Volume: [req/day] · Stack: [tooling]
Trace schema: [the span table, tailored]
Metrics:
| Metric | Family | Threshold / baseline | Window | Alert → owner |
|---|
Sampling & retention: [the policy]
Privacy: [content classification, access, deletion path]
Dashboards: [the 2-3 views: live health, quality drift, cost]
First incident drill: pick yesterday's worst trace and confirm it can be replayed end-to-end from the stored data.
Quality Checks
- Any incident is replayable from its trace alone — the schema was tested against that bar
- Every metric has a number, a window, and a named owner — no orphan dashboards
- Quality alerts are drift-based against a rolling baseline, not absolute guesses
- Sampling keeps 100% of error/guardrail/negative-signal traces
- The privacy note exists and names retention and access — logged prompts are user data
Anti-Patterns
- Do not log only inputs and outputs — without retrieval and tool spans, root cause analysis is guesswork
- Do not alert on mean cost or mean latency — the tail is where both incidents live
- Do not run judge-based quality scoring on 100% of traffic — sample; spend the budget on better baselines
- Do not treat observability as launch-week scaffolding — drift metrics only work with months of baseline
- Do not ship an agent that can take actions without logging the guardrail verdicts alongside the actions
