Warning signs get explained away
Start here
The Normal List
Recheck the problems the team has learned to live with
On a regular schedule, the team lists the problems it has decided are acceptable because nothing bad has happened yet, and asks whether each one still deserves that status.
Try this first
Ask, "What do we see regularly that would have alarmed us the first time we saw it?"
Borrowed from sociologist Diane Vaughan's study of the 1986 Challenger launch decision, where she named "normalization of deviance," and from the board that investigated the loss of the shuttle Columbia in 2003. Read the original source.
Other moves for this problem (5 of 5)
Kind of change
- Name summit fever · Name the goal fixation that overrides judgmentFor when: Hitting a milestone is starting to override judgment and safety.
- Challenge-and-response callouts + "question the unusual, listen, don't anticipate." · Say it out loud, confirm it backFor when: Critical handoffs are assumed rather than confirmed, or the team pushes past its own stop-points.
- The Known-Answer SetFor when: Nobody can say whether the AI, or the team's trust in it, has gotten worse over the past few months.
- Near-miss reporting + a Lessons Learned Center · Log the close calls, not just the failuresFor when: Only actual failures get surfaced, so warnings are missed.
- The 10 Standard Fire Orders + 18 Watch Out Situations · Danger signals that mean pauseFor when: Teams can't recognize the moments AI is likely to fail.