Agent Observability Replay Trace

Replay a trace against local code

A fast iteration loop on a single production trace: take a trace whose output a developer didn't like, optionally change the code, re-run it against their LOCAL code, and show a concise diff of old vs new output — repeating until they're happy. Assumes nothing about the project's layout.

Invoked from the developer's coding agent: /agent-observability-replay-trace <trace-id> [<changes to test>]. With no modification, do the replay + diff only (a reproduce/regression check), then offer to enter the loop.

This file is the workflow spine — terse on purpose. The depth lives in references/details.md (trace backend + pup flags, the runner contract, export mode, polling, the trace-link scoping fix) and references/local-setup.md (making a deployed-only app locally runnable). Read details.md before you touch pup or generate the runner.

Writing code — keep comments minimal to none. Everything you generate or edit (the annotation, the runner's ENTRYPOINTS entries, a local harness, iteration edits) should match the surrounding code and carry no unnecessary comments — don't narrate what the code plainly does; add a comment only for a genuinely non-obvious why.

Intent tagging: On every datadog-llmo MCP tool call, prefix telemetry.intent with skill:agent-observability-replay-trace[<inv_id>] — (a short per-run id, generated once and reused for every call) followed by a description of why the tool is being called. On the first MCP tool call only, use skill:agent-observability-replay-trace:start[<inv_id>] — instead (note the :start suffix). Example first call: skill:agent-observability-replay-trace:start[3a9f1c2b] — fetch the original trace's baseline output. pup-CLI calls carry no telemetry.intent, so this applies only on the MCP path.

Interaction model — selector gates, never a hard stop

This is a live loop. At every decision point present the choices as an AskUserQuestion selector (the plan-mode-style menu), not a plain question that ends your turn. Two gates: (a) after you propose code changes, before replaying; (b) after each diff. The selector's free-text option lets the user type detail (what to refine) inline — act on it directly, don't ask a follow-up. Keep re-presenting after every replay until they pick "stop here".

Scope — check first

• Traced with ddtrace / LLM Obs (an ml_app + a discoverable entrypoint). Python is first-class; other languages work but you write the runner to the contract in their SDK/build tooling.
• JSON-serializable entrypoint input, and a callable seam for the root span (see step 3.5 — not a binary "is it runnable?"; deployed-only apps often still expose a plain callable).
• A trace-access backend — the datadog-llmo MCP (used when present) or the pup CLI (fallback, and the easier install if you have neither) (step 0).
• Credentials: DD_API_KEY + DD_SITE + provider key(s). Not DD_APP_KEY — plain trace, not an Experiment (that's agent-observability-replay-experiment).
• Side effects, irreversible: replaying re-runs real code (model spend + real writes), and LLM Obs traces cannot be deleted — a mis-scoped replay (wrong ml_app) permanently pollutes the production app's dashboards/eval sets. That's why the <ml_app>-local isolation (steps 4/6/7) is load-bearing, not tidy. Warn before the first replay.

Workflow

0. Ensure a trace-access backend

Pick, in order: (1) the MCP if mcp__datadog-llmo-mcp__* tools are present — the default (slightly richer for reads: structured tree + content_info); (2) else pup if installed and pup auth targets the app's org; (3) else the user has neither → guide the pup install (it's easier to set up than the MCP, so recommend pup here):

Copy code
brew tap datadog-labs/pack && brew install datadog-labs/pack/pup
pup auth login

(MCP alternative: claude mcp add --scope user --transport http "datadog-llmo-mcp" "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs"; see https://docs.datadoghq.com/bits_ai/mcp_server/setup/.) Don't proceed without a backend. The backend↔operation mapping and pup's exact flags/gotchas are in details.md — read that section before using pup. Two pup musts: (1) results come back at data.spans[] or top-level spans[] (varies by version/--no-agent) — parse whichever is present, or you get zero hits on an ingested trace (a silent false negative, step 7); (2) check token expiry (pup auth status), not just that auth exists — expiry mid-loop looks like "trace not found."

1. Parse the command

<trace-id> + optional free-text modification (everything after the id); none → diff-only mode. Determine the ml_app from the project (LLMObs.enable(ml_app=…) / DD_LLMOBS_ML_APP) or the trace; confirm if ambiguous.

2. Fetch the trace + locate the baseline

Fetch via the backend; note total_duration_ms (drives step 7), the trace_url, and metadata.replay_input/replay_entrypoint if present. Locate the baseline field — it's not always the root output: the value the developer dislikes may be a tool-call input or an intermediate output several levels deep, and the app may post-process it before the span records it. Pick the field the code change can actually move, or the delta drowns in noise.

2.5. Check for fan-out

If the root span fans out into repeated sibling subtrees (a batch/map over N parallel sub-runs), the change under test is usually visible in a single branch — replaying the whole root costs ~N× spend and time for no extra signal. Offer to replay one representative branch; log what you skipped. Pick deliberately: the cheapest branch that reached the terminal / side-effecting tool (most branches are no-ops that prove nothing), and reconstruct its input from the child span's input, not the root's. Full root only if the change is inherently cross-branch.

3. Resolve the entrypoint + input

• Entrypoint: metadata.replay_entrypoint if present; else infer from the root span (name/kind) + code and confirm with the user.
• Input: metadata.replay_input if present; else derive a suggested input (prefer the code signature — the rendered prompt is lossy) and have the user confirm/edit.

3.5. Ensure a local run path (find the innermost callable seam)

Ask "what is the innermost callable seam for this root span, and can I call it directly with JSON?" — not "is the app runnable?". Already directly callable → skip, continue. Buried under a handler/service (deployed-only, no local __main__, live-infra coupling) → follow references/local-setup.md (detect → propose → approve → build). The common middle case — a deployed service whose core logic is already a plain callable (ports-and-adapters) — just extract/call that seam; full local-setup is overkill.

4. Ensure the two persistent artifacts (one-time setup)

• a) In-entrypoint annotation on the app's real entrypoint, so all future traces (production too) self-describe. Stamp it at span start, not the success/deferred-finish path — a failed run must still carry replay_input (those are the ones you most want to replay):

  pythonCopy code
  LLMObs.annotate(span=span, metadata={"replay_entrypoint": "<stable id>", "replay_input": <extractor>})
  

  No replay_output — the original trace is the baseline. (Non-Python annotate APIs differ — e.g. Go span.Annotate(llmobs.WithAnnotatedMetadata(...)); see details.md.)
• Isolation pre-flight (before writing the runner): grep the entrypoint's call path for per-span/ per-call ml_app overrides (Go llmobs.WithMLApp; Python ml_app= on a decorator or in LLMObs.annotate). Those beat the init-level -local, so the app's spans can still land in production — tracer-level config is not proof of isolation. If any exist, the app's ml_app must resolve from env so -local wins.
• b) The runner — satisfies the language-independent runner contract in details.md (load env → derive <ml_app>-local → dispatch one entrypoint on JSON → flush on every exit path incl. errors → refuse to start unless ml_app ends in -local → print the -local ml_app). Python: copy scripts/replay_runner_template.py and fill ENTRYPOINTS. Other languages: write to the contract — don't assume the Python API carries over (Go APIs + export-mode gotchas in details.md), and where the language has no in-process dotenv add a run wrapper (artifact c) that sources the project env, unsets ambient provider vars, and exports the -local override. Infer + confirm the run command; follow the host repo's build-file conventions (Bazel/Gazelle → cmd/<name>/, run Gazelle, build before replay).

5. (If a change was requested) edit, then gate

Make the code changes, show the developer the diff of your changes, then an AskUserQuestion selector: Replay now / Adjust the changes first / Cancel. Only replay on "Replay now".

6. Replay

Before the first replay: warn (re-running is real — model spend + real writes), and sanitize the environment. The coding agent's own env (ANTHROPIC_API_KEY / ANTHROPIC_BASE_URL set by Claude Code, and other provider keys) can make the app's SDK bypass its configured model gateway — a fidelity gap invisible in the diff. Unset ambient provider vars by default and report that you did (don't just ask); grep the app for its own ambient-key guards. Also verify the credential's org matches the trace's org — a mismatch ships the replay somewhere you can't query (looks like ingest lag). On confirmation, record t0 and run — source the project's env file, never inline secrets (the marker tag is fine on the command; DD_API_KEY=<value> inline is blocked by the permission classifier and leaks to history/transcript — use the wrapper/env-file):

Copy code
DD_TAGS=replay_run_id:<unique-id> <run cmd or wrapper> --entrypoint <id> --input-file <path>

The runner emits under <ml_app>-local (idempotent, so replays never pollute production) and prints that name — poll for the new trace under it.

7. Wait for the new trace

• Runner subprocess timeout = max(120s, ~3 × total_duration_ms).
• Ingest poll: after it returns, poll the backend every ~5s up to ~2 min for the replay_run_id tag under <ml_app>-local (pup: --query "replay_run_id:<id>", plain key:value). Before ever reporting "not found," re-query with no tag filter (just <ml_app>-local + window): if that returns spans, your filter/parse/scope is wrong — not ingestion. A false "no trace" reads as normal and invites a wasteful re-run.
• Verify isolation on each hit — a tag match is NOT proof. --query/tag matching can return a span whose real ml_app is a different app (the --ml-app filter gets ignored). Read ml_app off every returned span and assert it ends in -local before reporting a clean replay — otherwise you report "clean replay under -local" while the trace is actually in production (which you can't undo). This false confidence is worse than the false negative. Don't hard-fail on timeout; offer to keep waiting.

8. Diff (with links to both traces)

Concise summary of how the new output differs from the old — meaningful differences only. Note live-world drift; and because any nondeterministic agent varies run-to-run, default to two replays (diff-only mode too, not just model-facing edits) and use replay-to-replay comparison — if the two local runs differ from each other about as much as from production, the delta is sampling variance, not your change. If the replay disables a side-effecting integration (dry-run), that integration's subtree is absent — exclude it from both sides before comparing span counts, or the structural diff is junk. Lead the diff with both trace links:

• Old: trace_url verbatim — but under fan-out (you replayed one branch) link the branch span, not the whole-root url.
• New (replay): must carry ml_app=<ml_app>-local or it opens empty — and the trace_url is an org-switch wrapper (…/switch_to_user/<id>?next=<encoded /llm/traces …>&flow=org_switch), so inject ml_app=<ml_app>-local into the decoded next query and re-encode; do NOT append to the outer URL (mechanics in details.md). Browser-unverifiable from here — confirm once it opens non-empty.

9. Gate — iterate, or stop on a broken harness

Harness-failure gate (before the diff): if a replay reveals the harness is wrong — trace landed under the wrong ml_app, no trace after the step-7 sanity checks, missing flush, or auth/org misrouted — do NOT proceed to a diff on bad data. Stop and present a selector to fix the harness (re-scope ml_app / add flush / fix env) and re-replay. Otherwise, after the diff, an AskUserQuestion selector: Looks good — stop here (finish; leave the edits in the working tree) / Make more changes (free-text inline → back to step 5). Re-present after every replay; end only on "stop here".

Reference

• references/details.md — trace backend + pup exact flags, the runner contract (+ Go, export mode), polling + the false-negative sanity check, the trace-link scoping fix, limitations. Read before pup / the runner.
• references/local-setup.md — making a deployed-only app locally runnable (step 3.5). Read when that gap shows.
• scripts/replay_runner_template.py — the Python runner to copy + fill.