AWS Builder Center

Stop Grading Postmortems by Action Items: Measure Repeat-Incident Rate and Follow-Through Instead

A blameless postmortem is not a kindness extended to engineers; it is the precondition for getting the complete timeline, and a program that produces many documents while the same failure mode keeps recurring is documentation, not learning. This article argues that the honest way to judge an incident-review program is the attributable repeat-incident rate plus the on-time closure of corrective actions — not the number of postmortems written or the length of the action-item list.

Series: developer-tools (8 articles)

  1. …
  2. 4
    Stop Grading Postmortems by Action Items: Measure Repeat-Incident Rate and Follow-Through Instead This article

  1. Why blamelessness is an information problem, not a mood
The intellectual basis sits in Amy Edmondson's work on team psychological safety, published in Administrative Science Quarterly in 1999. Studying work teams, she tested the chain from psychological safety through learning behavior to performance, and found that teams holding the shared belief that they could take interpersonal risks asked more questions, admitted errors more freely, and sought feedback more often [1]. The finding that complicates naive dashboards: higher-safety teams reported more errors, not because they made more, but because they were willing to surface them [1]. Treating raw error counts as a quality measure therefore punishes exactly the teams willing to tell you the truth.
Google's own practice draws the same operational conclusion. Its guidance on a culture of security and reliability states that blameless postmortems are used to uncover fixable patterns of failure, and that hiding incidents erodes trust while repeating the same failure is the avoidable outcome [2]. The mechanism is simple. If an engineer believes the retrospective will be used to assign personal consequences, the timeline gets edited, local context gets omitted, and the investigation reconstructs a story that protects the narrator rather than one that explains the system.
  1. The accountability objection, and why it is a category error
"Blameless" does not mean no accountability. The distinction is where accountability is aimed. A blame-oriented culture asks why an individual was careless and removes a permission; a system-oriented culture asks why the architecture allowed a single unreviewed action to reach production, and holds owners accountable for changing that mechanism [2]. The accountability is real, but it attaches to the system fix and the follow-through, not to the person who happened to be at the keyboard when latent conditions aligned.
The figure below makes the causal reading explicit, and the lower bar states the boundary condition: removing personal blame does not remove the obligation to deliver corrective work.
![A four-stage causal chain from psychological safety through complete information and owned fixes to fewer repeat incidents, with a note that blameless is not no accountability.]( https://aka.doubaocdn.com/s/ULtqNWpbwX )
Figure by the author (illustrative).
  1. The evidence that process alone does not guarantee learning
A team that adopts the ritual without checking whether it works is the failure mode worth taking seriously, and here the empirical picture is sobering. An ACM study of operational retrospectives examined how postmortem artifacts are actually used in a high-performing organization, observing twelve engineers across three teams over three months and identifying distinct postmortem template archetypes [3]. One of its central observations is that practitioners use these artifacts to maintain mental maps of socio-technical systems that keep changing — meaning the value is in keeping an accurate shared model, not in producing a static report [3].
A more recent case study at a national space research center, an organization that ought to be a high-reliability exemplar, found the opposite of a mature learning loop. Through ten semi-structured interviews, plus five more across four other high-reliability organizations, the authors found that gathering lessons was informal and dependent on personal initiative, documentation was fragmented, and recurring failures persisted despite general process adherence [4]. Time pressure, documentation gaps, and team turnover — with knowledge walking out the door as people left — were the recurring barriers [4]. This is a preprint and has not been peer-reviewed, but it directly undercuts the assumption that holding a meeting and filing a template constitutes learning. Process presence is not the same as process effectiveness.
  1. A measurement contract rather than a template
The corrective move is to instrument the program the same way you would any other system. Industry practitioners converge on a small set of metrics, though the numeric targets below are practitioner conventions rather than peer-reviewed standards.
Metric
Definition
Why it is a signal
Evidence level
Attributable repeat-incident rate
Share of incidents whose tagged failure mode appeared before
Directly tests whether the class of failure is being removed
Practitioner metric; causal proof weak
Action-item on-time closure
Items with an owner and due date closed by that date
Tests whether documented intent converts to change
Practitioner convention
Near-miss reporting volume
Voluntary reports of incidents caught before impact
Rises when safety is genuine, per Edmondson's reporting effect
Supported by team-level research
Participation distribution
Unique speakers against attendees
Reveals whether a few voices dominate
Practitioner heuristic
Published practitioner guidance suggests action-item on-time closure of at least 70% and a repeat-incident rate at or below 15%, explicitly framing the lower end as the point where the ritual becomes theatre [5]. Other vendors define the same recurrence metric over a rolling twelve-month window and separate mitigative fixes from preventative ones that address the entire class of failure [6]. These thresholds are useful as operating defaults, not as scientific constants; they come from vendors and consultancies, not controlled experiments.
  1. The honest limitations
Three caveats keep the argument from overclaiming. Edmondson's foundational evidence was collected in manufacturing and healthcare teams rather than software organizations, so the transfer to engineering rests on mechanism and replication rather than a direct software sample. The repeat-incident and closure targets are industry rules of thumb, and organizations differ in how they tag root causes, which makes cross-company comparison fragile. Finally, much of the published case material is observational or vendor-authored, so the claim that a well-run program reduces recurrence is supported by mechanism and correlation rather than randomized causal evidence. A rigorous manager should treat the metrics as a way to detect a broken program, not as proof that a compliant one succeeds.


Next steps: tag every incident with a failure-mode category so recurrence is queryable, and give every action item an owner and a due date rather than leaving them as a list. Track on-time closure and the attributable repeat rate over a rolling window, and watch near-miss volume as an independent check that people still feel safe speaking up. If the documents are growing but the same failure modes return, the program is reporting, not learning.

References

  1. A. Edmondson, "Psychological Safety and Learning Behavior in Work Teams," Administrative Science Quarterly 44, no. 2 (1999); faculty record and bibliography, Harvard Business School —  https://www.hbs.edu/faculty/Pages/profile.aspx?facId=6451 
  1. Google, Building Secure and Reliable Systems, Chapter 21, "Building a Culture of Security and Reliability" —  https://google.github.io/building-secure-and-reliable-systems/raw/ch21.html 
  1. Beyond the 'Fix-It' Treadmill (postmortem artifacts and organizational learning), ACM Digital Library —  https://dl.acm.org/doi/pdf/10.1145/3380322 
  1. Learning From Software Failures: A Case Study at a National Space Research Center, arXiv preprint (not peer-reviewed) —  https://arxiv.org/html/2509.06301v2 
  1. PanDev Metrics, Incident Postmortem Template That Actually Helps (closure and recurrence targets) —  https://pandev-metrics.com/docs/blog/incident-postmortem-templates 
  1. incident.io, SRE Incident Postmortem Best Practices (recurrence metric and item types) —  https://incident.io/blog/sre-incident-postmortem-best-practices 

Series: developer-tools (8 articles)

  1. …
  2. 4
    Stop Grading Postmortems by Action Items: Measure Repeat-Incident Rate and Follow-Through Instead This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article