Observability

Observability

Telemetry is written before the incident and read during it. A dashboard nobody had a question for is a screensaver; a log line nobody can correlate is a receipt for an event you cannot find.

When this fires

A service is about to carry real traffic, a new critical path is added, or a review asks how a failure here would be noticed. It fires hardest immediately after an incident where the answer was not in the data — that is the only moment the gap is precisely known.

It does not fire for a live outage (mitigate first), for a slow path you can already reproduce and measure, or for LLM/agent run traces.

Procedure

1. Write the questions down first. "Is it up, is it wrong, is it slow, which dependency, which deploy, which tenant." Each signal you add answers a named question. Instrumentation added without one becomes cost and noise that later gets sampled away.
2. Cover every request path with four numbers: request rate, error rate, duration distribution, and saturation of whichever resource is actually constrained (connections, workers, memory, queue depth). Per route, job and consumer — a global error rate hides one broken endpoint completely.
3. Emit structured events, not prose. One event per unit of work, with stable field names: timestamp, service, version, environment, correlation id, route or job name, outcome, duration, error class. A field you can filter on beats a sentence you have to regex.
4. Propagate one correlation id everywhere — inbound request, outbound calls, log lines, emitted messages, and the background job that consumes them. Most traces die at the queue boundary, which is exactly where the unexplained latency lives.
5. Instrument the boundaries you do not control: database, cache, third-party APIs, queues. Record attempt, outcome, duration and retry count separately from the caller's total. Use the tracing library the project already depends on rather than inventing a span format, and check its current API instead of recalling it.
6. Record distributions, not averages. Keep p50/p95/p99 plus count. Percentiles cannot be recovered from a mean after the fact, and the tail is what the complaint is about.
7. Bound label cardinality deliberately. User id, email, session, full URL and raw paths containing ids do not belong in metric labels — they belong in logs and traces. Write the allowed label values per metric; an unbounded label is the usual cause of a telemetry bill.
8. Redact at the boundary, before anything leaves the process. Decide per field: kept, hashed, truncated, dropped. Authorization headers, cookies, tokens, request bodies and payment data are dropped. Exporting telemetry to a third-party backend is a data-export decision — raise it, do not enable it as a default.
9. Separate liveness, readiness and dependency health. A check that returns 200 because the process is alive, while its database is unreachable, teaches the load balancer to keep sending traffic to a broken instance.
10. Instrument the silent failures by name: retries, timeouts, queue depth and age, dead letters, dropped messages, circuit-breaker state, cache hit rate, job lag, clock skew. Nothing pages when a backlog grows unless something counts it.
11. Carry version and deploy identity in the telemetry so "which release" is answerable by filtering rather than by asking in chat.
12. Alert on symptoms users feel — error rate, latency against the stated objective, queue age — not on causes like CPU. Each alert names the first thing the responder should do and links the query behind it. An alert nobody can act on gets muted, and the mute outlives the reason.
13. Propose thresholds; do not arm them. Creating alert rules, dashboards, retention or sampling policy changes shared production configuration and usually costs money. Write the rule and the query, then stop and ask before creating anything in the account.
14. Drill it before you need it. In a non-production environment force an error, a dependency timeout and a slow request, then answer the step-1 questions using only the queries and dashboards. Every question you could not answer is the gap — fix it now, not at 3am.

Checklist

• The questions this telemetry answers are written down
• Rate, errors, duration distribution and saturation exist per route and per job
• Events are structured with stable field names, including version and environment
• A correlation id crosses process, HTTP and queue boundaries end to end
• External calls and retries are timed separately from the caller
• Percentiles recorded, not averages
• Metric labels bounded; no user-level identifiers in label sets
• Redaction decided per field; third-party export raised rather than assumed
• Queue age, retries, timeouts and dead letters are counted
• Alerts are symptom-based and each links its query and first action
• The failure drill was run and the unanswered questions listed

Failure handling

• All signals green, users complaining — you are measuring the wrong point. Move measurement closer to what the user experiences, past the load balancer, CDN and client.
• Telemetry cost spiked — look for an unbounded label or a debug-level log left on in production before touching sampling.
• Trace stops halfway — context is not propagating across an async, queue or thread boundary. Fix the propagation; a partial trace invites a confident wrong conclusion.
• Nothing was recorded for the window that mattered — sampling or retention dropped it. Say that plainly. Do not reconstruct the window from what seems likely.
• No telemetry backend available — instrument to structured local logs with the same field names and say so. Local output is not evidence about production behaviour.

Evidence to report

Name what was instrumented (code emitting signals), what was observed (a real event or query result you read back), and what remains untested — these are different claims. Quote one emitted event with its real field names, and the query that returns it. Give the drill result as a list of the step-1 questions with answered / not answered against each. List alerts proposed separately from any created, and who approved the ones created. Code that compiles but has not been seen to emit anything is instrumented, not verified.