Blameless Postmortems
A postmortem is a written record of an incident: its impact, what was done, its root causes and contributing factors, and the follow-up that prevents recurrence. Its value is proportional to what people learn from it and what changes because of it.
When one is required
Agree triggers in advance. Typical:
- User-visible downtime or degradation beyond a threshold.
- Data loss of any kind.
- On-call intervention such as a rollback or traffic reroute.
- Resolution time above a threshold.
- Monitoring failure (humans found it, not alerts).
- A single incident consuming a large share of error budget (per
sre-slos-and-error-budgets). - Anyone may request one for any event.
Blamelessness
Assume everyone acted reasonably with the information and tools they had. Focus on what in the system allowed the failure, not who.
- Don't name individuals as the cause; describe the action and the conditions that made it possible ("the workflow was rerun manually; the tool had no guard against repeat execution").
- No loaded or emotional language ("careless", "ridiculous", "can't believe we survived"). Replace adjectives with data.
- Action items fix systems and processes, not people. "Train people to be more careful" is not a fix; a guard rail is.
- If leadership slips into blame, redirect to "what warning signs existed and why were they missed?"
Procedure
- Assign one owner (plus collaborators) as soon as the incident closes. Include everyone who participated in writing it.
- Pull the facts from the incident state doc, chat logs, graphs, deploy history. Build the timeline in UTC.
- Quantify impact with numbers: duration, affected users/requests/% of traffic, latency, revenue/budget impact, team cost. An estimate with its method stated beats no number.
- Analyse causes. Separate the trigger (what set it off) from root causes and contributing factors (latent conditions that let the trigger cause damage — missing validation, no rate limit, unsafe default, monitoring gap). Keep asking why until you reach things you can change. Expect several contributing causes.
- Lessons learned: what went well, what went poorly, where we got lucky (luck marks a hidden risk).
- Action items — see criteria below. Group them by theme for large incidents.
- Review with the checklist below; senior engineers review high-impact postmortems.
- Publish within about a week, widely by default (whole engineering org; customers when appropriate). Delay lets memory fade and lets others fill the silence.
- Track action items to closure in the normal bug tracker and review completion rates.
Template: references/postmortem-template.md.
Action item criteria
Each action item must have:
- One owner and a tracking ticket.
- A priority — not all the same. At least one top-priority item for any user-affecting outage.
- A verifiable end state: "Alert when > X% of machines leave the pool within 10 min", not "improve alerting".
- A type, with a balance across types — not only mitigations:
- Prevent — stop this class from happening (input validation, safety checks, idempotency).
- Mitigate — reduce blast radius or duration (rate limits, kill switch, smaller rollouts).
- Detect — find it sooner (new alert, probe).
- Repair / Investigate / Process as needed.
Review checklist
- Readable by someone outside the team (background + glossary for jargon)?
- Impact quantified, with sources linked?
- Trigger and root causes both identified, at sufficient depth?
- Timeline complete, with detection and mitigation times?
- Action items meet the criteria above, vetted by the service's tech lead?
- Blameless language throughout?
- Single owner, status, dates, sharing list filled in?
- Shared beyond the team?
Building the culture
- Reward closing action items, not just writing documents.
- Celebrate good postmortems (reading clubs, weekly outage summaries, cross-team reviews).
- Use past postmortems for training (re-enact them as role-play drills).
- Add machine-readable metadata (services, trigger type, detection method, durations) so trends can be analysed across postmortems.
- Warning signs: people avoid being "involved", postmortems are late or shallow, the same incident recurs (are action items closing? is velocity trumping fixes?).
Sources
- SRE book ch. 15 Postmortem Culture — https://sre.google/sre-book/postmortem-culture/
- SRE book Appendix D Example Postmortem — https://sre.google/sre-book/example-postmortem/
- SRE book ch. 16 Tracking Outages — https://sre.google/sre-book/tracking-outages/
- Workbook ch. 10 Postmortem Culture — https://sre.google/workbook/postmortem-culture/
- Workbook Appendix C Results of Postmortem Analysis — https://sre.google/workbook/postmortem-analysis/
