Sometimes it's great, sometimes it's off. No idea why.
Nobody knows where the AI can be trusted
The AI gets used the same way for every task, without anyone mapping where it is reliable and where it isn't. Its errors may cluster in the kinds of tasks it handles poorly, and nobody sees the pattern.
- The AI gets used for everything because it's easier.
- Nobody can say which tasks it's good at and which it's bad at.
- Quality problems cluster around certain AI steps, and nobody has noticed.
Who: two people who use the AI in this workflow every day. Look at: 20 recent AI outputs from different kinds of tasks, and mark each one right or wrong. Count: errors by kind of task. You're probably looking at this pattern if errors cluster in kinds of tasks nobody had named. Probably not if errors are rare and spread evenly.
Plausible but wrong AI output. That pattern is about a single fluent answer that nobody catches. This pattern is about not knowing where those answers are likely to come from.
Pull 30 recent AI outputs. Have two people read them and write one line about each one that has a problem. Then count which problems repeat.
Set aside a handful of known-answer cases and run them now and then, against the model and against the team.
For the next month, write down each time you overrule the AI and why. At the end of the month, check which of those decisions turned out to be right.
In a field experiment with consultants, people using AI on a task outside what it handled well were 19 percentage points less likely to get the right answer, and their output looked just as confident (Dell'Acqua and colleagues, 2023). Emerging.