Incident response
Mitigation comes before explanation. The cause keeps; the outage does not. Every step here exists to stop you from applying an unrecorded change you cannot undo and then calling a coincidence a fix.
When this fires
A production system is failing or degraded now and the harm is ongoing. It stops firing the moment the signal is stable again — then it is a debugging problem and a postmortem, not this.
If the symptom has already passed on its own, do not start mitigating. Preserve evidence and hand it to investigation.
Procedure
- Name the flagging signal exactly — which alert, dashboard, query or user report, what threshold, and the first bad timestamp. Everything you do later is judged against this one signal. If nobody can name it, you do not yet know that anything is wrong.
- Open the log before investigating. Append-only, UTC timestamps, one line per observation, action and decision, each action recording who ran it and what happened. A timeline written afterwards is a story assembled from memory of a stressful hour.
- Size the blast radius in numbers: who (users, tenants, regions), what (endpoints, jobs, features), how much (share and absolute count of affected requests), since when — and whether it is growing, flat or shrinking. "Lots of users" is not a blast radius.
- Say the impact out loud early, and leave comms to a human. Status pages, customer notices and broadcast messages are outward-facing: draft them if asked, stop before posting.
- Enumerate what changed before the first bad timestamp — deploys, config edits, feature-flag flips, migrations, scaling changes, dependency bumps, credential or certificate rotation, traffic shifts, upstream vendor status. List them before forming a causal theory; a theory formed first will pick whichever change it likes.
- Rank mitigations by reversibility and blast radius, not by how likely they are to be the cause. Revert the deploy, flip the flag off, drain or fail over the bad instance, scale out, shed load or rate-limit, restart. A mitigation does not need to know the cause. Writing a forward fix during an outage is the least reversible option available — treat it as the last one.
- Before applying anything, state four things: the expected effect on the flagging signal, the window you expect it in, the risk the action itself carries, and exactly how to undo it. Then stop and ask. Every production-affecting action needs its own confirmation, and "while I am in there" changes are never included in it.
- Apply one change at a time and record the time it went in. Two simultaneous changes buy an uninterpretable recovery and a mitigation nobody can safely remove later.
- Preserve evidence before it ages out. Copy the log excerpts, the metric graph with its time range, traces, failing payloads and the exact error strings into the incident log. Never delete logs, rotate files, drop tables or clear a queue to make a symptom disappear — that is irreversible data loss during the one hour it matters most; stop and ask.
- Confirm recovery against the flagging signal, over a window long enough to outlast a coincidental dip and to cover at least one full cycle of whatever failed. A green build, a passing test, a healthy synthetic check and "I am not seeing errors now" are none of them recovery evidence.
- If the mitigation did not work, undo it before trying the next one. An incident carrying three abandoned mitigations has become a second incident with no known-good state.
- Hand off explicitly: what is mitigated and by what, what is still unexplained, every change still in place and how to remove it, and where the preserved evidence is. Root cause goes to systematic debugging once the bleeding has stopped.
Checklist
- The flagging signal is named, with its threshold and first bad timestamp
- A timestamped log exists and was written during, not after
- Blast radius stated in numbers, with its direction of travel
- Recent changes enumerated before any causal theory was offered
- Mitigation options ranked by reversibility, with the undo written down
- Each production-affecting action confirmed individually before it was applied
- One change at a time, each with the time it was applied
- Evidence preserved before retention could drop it
- Recovery confirmed on the flagging signal over a stated window
- Handoff names what is mitigated, what is unexplained, and what must still be removed
Failure handling
- The signal recovered but your change could not have reached it — different region, tenant or code path — it is a coincidence. Do not close the incident on it; keep watching.
- Nothing changed in the window — not every incident is self-inflicted. Check upstream provider status, expired certificates, exhausted quotas or licences, full disks and volumes, rotated credentials, and the slowly growing resource (connections, queue, retention, ids) that crossed a threshold with no deploy at all.
- Several things broke at once — look for the shared dependency before treating them as separate incidents.
- Monitoring itself is degraded — absence of alerts is now unknown, not healthy. Say so and fall back to a signal you can read directly.
- "Just restart it" — acceptable as a mitigation when it is reversible and recorded, but a restart destroys the process state that would have explained the failure. Capture what you can first, and expect it to return.
- Pressure to apply a fix without confirmation — urgency is not authorization. State the proposed action and its undo, and wait.
Evidence to report
The timestamped log itself, verbatim — it is the deliverable, not a summary of it. The flagging signal with its query, and its values before and after with timestamps. Each action with the time applied, who confirmed it, and its observed effect on that signal. Then state plainly what is mitigated (the symptom stopped) versus what is fixed (the cause is gone) — conflating these is how a mitigation gets removed a week later and takes the site down again. List what remains unexplained, and every temporary change still holding the system up, each with an owner.
