Learn and improve
How Did You Know?
Walk an expert's decision backward
Walk an expert backward through a real case to surface the cues their judgment actually ran on.
Borrowed from the Critical Decision Method, an interview technique from naturalistic decision-making research by Gary Klein, Robert Hoffman, and colleagues. Read the original source.
One session · An hour or more to set up · Anyone on the team can start it · Start here for this problem
Try this first
One expert, one real case, one question repeated at each turn: how did you know?
Use it when
You are about to hand a step to a model, or already have, and the description of that step is thinner than the judgment it takes to do it.
Skip it when
It needs a specific case with a real decision in it. On routine work with no judgment calls there are no cues to find, and the interview turns into a process description.
How to introduce it
Pick a case you handled where the right call was not obvious. We are going to walk it backward, and at each point where you decided something, we will ask what you noticed, what that told you, and what else it could have been. The goal is the things you saw that a competent beginner would have walked past.
How to show up
Stay on the timeline and resist the pull toward general principles. When the expert says they just knew, that is the moment to slow down and ask what they saw. Ask what would have had to look different for them to decide the other way.
How long it takes
Ninety minutes per case. Two or three fit in a half day.
What makes it hard
The expert will jump to general advice, and the whole value sits in the specifics, so you have to keep pulling them back to the case. Watch for the room filling silences with their own theories about what the expert must have seen.
What it looks like when it's working
The expert says something they have never said out loud before and looks surprised by it. A transcript full of things everyone already knew means the probing was too gentle.
How long until it sticks
Three or four interviews before a facilitator stops accepting the first general answer.
How you know it stuck
The team runs one of these before any decision to hand a judgment step to a model.
The idea behind it
Expertise sits mostly in what someone notices rather than in what someone can explain. The point of the probing is to reach the part that was never put into words.
Where it comes from
Naturalistic decision making. You cannot decide whether a model should take a step until you know what the human step contained. Experts usually cannot say until someone probes. One expert picks a real case they handled well and walks it backward on a timeline. At each decision point the room asks what they noticed and what it ruled out.
Pairs with
The Control Map, which needs this as an input. The Known-Answer Set, which turns the cues into test cases.
Also fits: Handoffs drop, between people and between people and AI
Home problem: Know-how lives in one person's head