Confident claims go unchecked
Start here
Think in probabilities; say the number
Replace "I'm sure" with a number, which is more honest and shows where to go find more information.
Try this first
Ask how confident, as a rough percentage.
Borrowed from former professional poker player Annie Duke, in her book Thinking in Bets. Read the original source.
Other moves for this problem (18 of 18)
Kind of change
- Premeditatio malorum · Rehearse what could go wrongFor when: About to make a big AI-assisted commitment with optimism unchecked.
- Say What You'd Check · Explain your own checks so you remember them at review timeFor when: People were trained on what to watch for in AI output, but errors are slipping through as the tool becomes routine.
- Wanna bet?For when: Someone asserts with more bluster than conviction.
- Design the Test TogetherFor when: The team is split (say, on whether an AI tool helps).
- "I don't know" is a legal moveFor when: False certainty is rewarded and people bluff rather than flag uncertainty.
- Pre-registrationFor when: Any outcome risks being spun as success after the fact.
- Real IndependenceFor when: A second check is in place, but it uses the same model or the same person who made the work.
- The CheckerFor when: AI-authored work needs a human check but authors are checking their own work.
- Provenance by DefaultFor when: People are signing off on AI-assisted work without knowing which parts the AI wrote or how sure it was.
- The 10 Standard Fire Orders + 18 Watch Out Situations · Danger signals that mean pauseFor when: Teams can't recognize the moments AI is likely to fail.
- The Braintrust · Peers give blunt notes, the owner decidesFor when: Fluent output invites approving it without checking, or feedback has curdled into orders and ego threats.
- Words of Estimative Probability · Agree what "likely" means in numbersFor when: "Likely" and "probably" mean different things to different people, and AI output can sound more certain than the evidence warrants.
- Name summit fever · Name the goal fixation that overrides judgmentFor when: Hitting a milestone is starting to override judgment and safety.
- The Key Assumptions Check · List what the plan assumes and test eachFor when: A plan rests on unexamined assumptions.
- Judge the Bet, Not the ResultFor when: An AI-assisted call happens to pay off and hides a bad process.
- Verification before disseminationFor when: Speed is being rewarded over accuracy.
- Weight by Track Record · Weight views by track record on the topicFor when: A contested decision where the loudest voice isn't the most credible.
- The Bar RaiserFor when: The quality bar slips under delivery pressure and the shippers are also the judges.