Sense and decide
Real Independence
Check AI output against a source that fails differently: another kind of model, a person, or a known-true reference. Anything less gets called a rough check.
Developed by Superadditive in our own review of the research. It draws on reliability engineering, where backup systems only help if they fail for different reasons.
Changes a rule or role · Minutes to start · Anyone on the team can start it
Try this first
Ask one question before you trust a check. Could this checker be wrong in the same way? If yes, it is not confirmation.
Use it when
A second check is in place, but it uses the same model or the same person who made the work.
Skip it when
Spend real verification on things that matter. Trivial output does not need it. The aim is not checking everything twice, it is being honest about what a check proves.
How to introduce it
When you check AI output, make sure the checker fails differently from the thing it's checking. A different kind of model, a human expert, a known-true reference. If it shares the same blind spots, it has not confirmed anything, and you should say so.
How to show up
Press the team on whether a check is independent. Make them say plainly when it isn't. Stop agreement between two similar sources from passing as proof.
How long it takes
No setup. It's a discipline applied when verifying.
What makes it hard
The fast check is another model, which usually shares the first one's blind spots. Resisting that convenience is the discipline. People want agreement, so they read a second matching answer as proof. Name the pull.
What it looks like when it's working
Important output gets checked against something that fails differently, and the team can tell confirmation from two tools agreeing. Trusting a second model's agreement means the idea has not landed.
How long until it sticks
A few instances before people reliably notice when a check isn't independent.
How you know it stuck
People ask whether the checker shares the blind spots, and label weak checks honestly.
The idea behind it
Two things that fail the same way cannot confirm each other. A matching second answer from a similar tool is reassurance, not evidence, and reassurance is worse than nothing.
Where it comes from
our review. Checking one model's work with another that shares its blind spots looks like verification and is not. A second check counts only if it fails differently from the first. A different kind of model, a person, or a known-true reference. Anything else is a rough check.
Evidence
No outside source. This move came out of our own review of the research, so treat it as reasoned judgment rather than tested practice.
Persuasion bombing: why validating AI gets harder the more you question it, HBS AI Institute In a study of more than 70 consultants using GPT-4, pushing back on the model's answers often made the model more persuasive rather than more correct. This is why a check needs to come from a separate source instead of from arguing with the same model.