Learn and improve
Check the Felt Speedup
Measure the time AI saves instead of estimating it
Time real work with and without AI, and compare the result to how much time people believe they are saving.
Borrowed from a 2025 randomized study by the research group METR on AI and software developer productivity. Read the original source.
One session · An hour or more to set up · Anyone on the team can start it
Try this first
For the next ten tasks of one type, flip a coin to decide whether to use AI. Write down how long each task takes from start to finish.
Use it when
Decisions about AI tools, staffing, or deadlines rest on how much time people say AI is saving, and nobody has measured it.
Skip it when
Skip it when tasks vary so much that no two are comparable, or when the team has too few tasks of one type to compare over a few weeks.
How to introduce it
Pick a type of task the team does often, where AI is now part of the work. Over two to four weeks, decide at random for each task whether it is done with AI or without it. A coin flip is enough. Record how long each task takes from start to finish, including the time spent checking and fixing. Before you start, have each person write down how much faster they expect AI to make them. At the end, compare three numbers: the speedup people expected, the speedup people report, and the speedup you measured.
How to show up
Present it as a way to find out, not as a test of whether people use AI well. Be ready for any result. If AI turns out to be slower on this task, treat that as useful information about where it helps and where it does not.
How long it takes
About an hour to set up. The measurement runs for two to four weeks inside normal work. The comparison at the end takes under an hour.
What makes it hard
People resist doing tasks without AI once they are used to it. The timing also has to include checking and rework, which people tend to leave out. The coin flip matters. If people choose when to use AI, they will tend to use it on the easier tasks, and the comparison will be unfair.
What it looks like when it's working
The team can say, with numbers, whether AI makes a given task faster or slower. Claims about time saved start to come with measurements attached.
How long until it sticks
One full round is usually enough to change how the team talks about time savings. Repeat it when the tools or the work change.
How you know it stuck
When someone says a new tool saves time, the next question on the team is how that was measured.
The idea behind it
People are poor judges of their own speed. AI work can feel faster because the machine produces text or code quickly. The checking and correcting afterward can take longer than doing the work by hand, and people tend to discount that time. A measured comparison is the only way to know which effect is larger for your team's work.
Where it comes from
Productivity research. In 2025 the research group METR ran a randomized study with 16 experienced software developers. They worked on 246 real tasks, and each task was randomly assigned to be done with AI tools or without them. Before the study, the developers predicted that AI would make them 24 percent faster. Afterward, they believed it had made them 20 percent faster. The measurements showed that they took 19 percent longer with AI. The study was small and covered one kind of work. It still shows that people's sense of their own speed can point in the wrong direction entirely. Before accepting that AI is saving time, the team measures it. People time the same kind of real task done with AI and without AI. Then they compare the measured difference with how much faster they believed they were.