It looked right. The writing was really good.
Plausible but wrong AI output
The AI can produce something fluent and confident that is still wrong. Fluency does not reliably tell you whether the output is correct, which can make errors easy to miss.
- Nobody can describe what a wrong answer from the AI looks like.
- AI output gets edited for style but not checked for substance.
- Mistakes that reach customers trace back to AI output nobody checked.
Who: you, with someone who knows the subject. Look at: five recent pieces of AI-assisted work that went out. Check one factual claim in each against its source. Count: claims that were wrong. You're probably looking at this pattern if you find errors that went out unchecked. Probably not if the claims hold up and someone had clearly checked them.
Reviews that go through the motions. In that pattern, a review step exists but doesn't catch anything. In this pattern, there may be no real check on the AI's work at all. Also the Change Barriers entry Misplaced trust.
On an important piece, name one person who is not the author to verify each key claim against its source.
Ask one question before you trust a check. Could this checker be wrong in the same way? If yes, it is not confirmation.
Ask each person to write three sentences on what the AI tool tends to get wrong in their work and how they would catch it. Put the most useful points where the output gets reviewed.
People tend to accept what an automated system tells them and miss its errors, especially when they are busy (Parasuraman and Manzey, 2010). Established. In one field experiment, consultants using AI outside what it handled well got the answer right 19 percentage points less often, with output that looked just as confident (Dell'Acqua and colleagues, 2023). Emerging.