Learn and improve
Game days / chaos engineering
Break your own safeguards on purpose
Stage a failure on purpose to find out whether the team's checks catch it, before a real one arrives.
Borrowed from chaos engineering, the practice of deliberately breaking systems to test them, which began at Netflix, and the "game days" run at Amazon and Google. Read the original source.
One session · An hour or more to set up · Anyone on the team can start it
Try this first
Once, feed the team a plausible but wrong AI output as though it were real, and see whether the checks catch it.
Use it when
You don't know whether the team's safeguards would catch an AI failure.
Skip it when
The drill has to be bounded and safe, because a staged failure that causes real damage defeats the purpose. It also needs the no-blame frame. Run as a trap it destroys trust instead of building resilience.
How to introduce it
Stage a failure and watch what your defenses do. For AI, that means running a drill where the model is set up to be confidently, plausibly wrong, and seeing whether the checks catch it. Engineering teams inject failures on purpose for the same reason: find the weakness before it finds you.
How to show up
Design the failure to be realistic and not obvious. Run it as a no-blame test of the system, never of individuals. What the drill reveals about the checks is the output. Fix that, do not audit who missed it.
How long it takes
Some effort to design a good drill. The drill itself is short.
What makes it hard
Designing a realistic failure that stays safe takes care, and a poorly bounded one escapes into real work. People also feel tested or tricked, so say clearly that you are testing the system's defenses.
What it looks like when it's working
Drills expose real gaps, the gaps get fixed, and the team catches staged failures more reliably over time. Drills always caught easily are too obvious. Drills that feel like traps are framed wrong.
How long until it sticks
Expect a few drills before the team's checks are strengthened and the exercise feels safe.
How you know it stuck
The team tests its own defenses periodically and treats each drill as a chance to strengthen them.
The idea behind it
You do not know whether your checks work until you test them, and a drill costs a fraction of a real failure. A planted wrong answer shows you exactly where verification is thin.
Where it comes from
Blameless postmortem practice in engineering. Stage a case where the AI is confidently wrong, then find out whether anyone catches it. Teams break their own systems on purpose to find the weak points before a real failure does. Netflix runs Chaos Monkey. Amazon runs GameDays.
Also fits: Risky or hard-to-undo steps go ahead without a stop
Home problem: Reliance on AI drifts and nobody notices