Skip to content

Structure an Incident Post-Mortem and Root-Cause Report

Document outage timelines, customer blast radius, five-whys root causes, and preventative engineering tickets in both internal and client-safe formats.

Copy the prompt and paste it into the AI assistant you already use.

Steps to produce a credible incident post-mortem

  1. 01

    Anchor every event to normalized UTC timestamps

    Log alert firing, on-call acknowledgment, rollback execution, and full recovery in UTC so distributed teams reconstruct exact detection and resolution gaps.

  2. 02

    Quantify blast radius in concrete user terms

    State exact percentages of failed transactions, affected regions, and degraded endpoints rather than vague phrases like intermittent slowness.

  3. 03

    Separate proximate trigger from systemic gaps

    Trace beyond the bad config push to explain why staging checks, canary thresholds, or automated circuit breakers failed to catch it earlier.

  4. 04

    Split internal notes from customer-ready PDFs

    Strip internal hostname paths and individual names when exporting an external enterprise RCA summary while keeping technical honesty intact.

How it works in practice

Raw outage Slack log & engineering notes

14:08 UTC deploy #882 pushed unindexed query on orders table. 14:12 CPU hit 99% on primary DB. PagerDuty paged at 14:15. Rollback started 14:24, healthy by 14:31. 18% of EU checkout POSTs timed out. Staging DB only had 5k rows so query looked fast.

Formatted RCA summary & remediation table

Incident Summary: Between 14:08 and 14:31 UTC (23 mins), 18% of EU checkout requests returned HTTP 504 due to primary database CPU saturation triggered by release #882. Root Cause: Missing composite index on `orders(tenant_id, status)` undetected in staging due to low synthetic dataset volume. Preventative Actions: - [P1] Enforce `EXPLAIN` plan gate in CI against production-scale statistics (Owner: DB Infra, Due: Friday). - [P2] Lower DB CPU alert threshold from 5m to 90s sustained saturation.

Transforms chaotic war-room chat logs into an objective engineering artifact and an executive-safe enterprise RCA.

Common questions

When should a team publish an external RCA PDF vs. an internal doc?

Share a sanitized external RCA PDF whenever an outage breaches contractual SLAs or disrupts enterprise workflows, while keeping full infrastructure logs internal.

What makes an incident post-mortem genuinely blameless?

Focus the narrative on tooling gaps, missing guardrails, and ambiguous runbooks rather than human error so engineers report near-misses openly.

How many remediation items should come out of one post-mortem?

Prioritize two to four committed engineering fixes with strict due dates rather than a sprawling wishlist that sits unscheduled in the backlog.

Learn more

Related guides and background articles for this task