Learn and improve
Track the Overrides
Keep a record of when people overrule the AI and who was right
Log each time someone overrules the AI and check later who was right, so the team learns whether its judgment is improving the AI's recommendations.
Borrowed from a 2026 field experiment on loan decisions by Wang, Zhang, and Lu, published in Management Science. Read the original source.
Ongoing habit · Minutes to start · Anyone on the team can start it
Try this first
For the next month, write down each time you overrule the AI and why. At the end of the month, check which of those decisions turned out to be right.
Use it when
People review AI recommendations before acting on them, but nobody knows how often they overrule the AI or whether their overrides turn out to be right.
Skip it when
Skip it when outcomes take too long to become known, or when the AI's recommendations are so low-stakes that a wrong one costs nothing.
How to introduce it
Set up a simple log. Each time someone decides against what the AI recommended, they write down the case, what the AI said, what they decided instead, and one sentence on why. When the outcome becomes known, record whether the person or the AI was right. Once a month, review the log together. Count how often people overrule the AI and how often those overrides turn out to be right. Look for kinds of cases that produce most of the good overrides or most of the bad ones.
How to show up
Make it clear that overruling the AI is expected when there is a reason for it. Do not treat wrong overrides as failures to punish. If people are punished for wrong overrides, they will stop overriding. Treat the pattern across many cases as the thing to learn from.
How long it takes
A few seconds to log each override. Thirty minutes a month to review the log.
What makes it hard
Logging feels like extra work, so people skip it on busy days. Outcomes can also be unclear. Keep the log to one line per case, and agree in advance what counts as the right decision.
What it looks like when it's working
The team knows how often it overrules the AI and how often those overrides are right. Over time, people overrule the AI more in the kinds of cases where they tend to be right, and less where they tend to be wrong.
How long until it sticks
Two or three monthly reviews before the log shows patterns worth acting on.
How you know it stuck
People log overrides as part of the work, and the monthly review changes how the team uses the AI.
The idea behind it
A person reviews AI output so that the person can catch what the AI gets wrong. That only works if the person disagrees at the right times. When people almost never overrule the AI, it could mean the AI is excellent. It could also mean people have stopped looking closely. A record of overrides and outcomes shows which one is true.
Where it comes from
Decision research. A field experiment on loan decisions, published in Management Science in 2026, compared decisions made by people alone, by an algorithm alone, and by the two together. The combination did best. The gain came from the moments when people disagreed with the algorithm and were right to disagree. How often people agreed with it did not explain the gain. If a team always accepts the AI's recommendation, the person reviewing it adds nothing. If a team overrules it without good reason, decisions get worse. A record of overrides and their outcomes is the only way to see which of these is happening. The study covered loan decisions, so applying it to other kinds of work is our judgment. The team keeps a simple record of each time someone overrules an AI recommendation. Later, once the outcome is known, the team checks whether the person or the AI was right.