In early 2025, 16 experienced software developers agreed to be timed.
They worked through 246 real tasks on open-source software projects. For some tasks they were allowed AI tools, and for the rest they were not. Before they started, they predicted AI would make them 24% faster.
When they finished, they still believed it had made them about 20% faster.
The study had timed every task. With AI tools, they took 19% longer. ⏱️
These were experienced people describing their own work, right after doing it. But their sense of it ran the wrong way. METR, the research group that ran the study, later summed up the gap: people “overestimated AI’s effect on their time spent on tasks by 40 percentage points on average.”
If you run a business of 20 or 200 people, you have probably had a version of this conversation with yourself:
“We added AI. Are we better off? I think so. But how would we know?”
The team likes it. Drafts come faster. The person who uses it most says it saves them hours every week, and they mean it.
The developers meant it too, and they were 19% slower.
Author’s note: I use AI to help write this newsletter, so this edition is about my business too. The drafts come fast. Getting to one I’m willing to put my name on does not: the last 5 editions took too many drafts each. So I’m running the Prove-It Test on my own process, against the result I set: an article I’d publish in 5 drafts or fewer.
“I think so” is doing the job of a number
That feeling is making real decisions. It decides whether you renew, whether you add seats, and what you expect the next AI purchase to do.
This is AI measurement debt: the cost of launching before you have defined what success looks like. When nothing was written down before the tool went in, the scorecard defaults to how the tool feels to the people using it. That feeling tends to overrate the tool. In METR’s research, the gap averaged 40 percentage points.
Every renovation gets judged against the before photo. Without one, “it looks better” is all anyone has. 📸
An AI tool works the same way. If the work was never measured before the tool arrived, the only record of before is memory, and the developers showed what memory does with it.
One survey says AI is working. Another found no impact.
In August 2026, GTIA surveyed 520 small- and midsize-business decision-makers. 84% reported a positive business impact from AI. 📊
Contrast that with a February 2026 survey of about 6,000 senior executives in the US, the UK, Germany, and Australia, published by a team of economists. Asked what AI had done to employment and productivity in their firms over the past 3 years, 9 in 10 reported no impact.
The 2 surveys asked different questions of different people, so they cannot be set against each other as a scoreboard. What they share is a method. Both are people reporting how things look from where they sit.
When the answer depends on who you ask, nothing has been measured yet.
1,000 civil servants liked the tool. Their colleagues saw no change.
From October to December 2024, the UK’s Department for Business and Trade gave 1,000 staff Microsoft 365 Copilot, the AI assistant built into email and documents. Then it published an evaluation.
72% of respondents said they were satisfied or very satisfied. By their own account, most tasks got a little faster. Scheduling and image tasks got slower, and users reported editing the tool’s output for grammar, spelling, and factual errors.
But the evaluators also asked colleagues outside the trial, the people working alongside those who had the tool. Those colleagues “had not observed productivity improvements from colleagues taking part in the M365 Copilot pilot.”
The report’s finding: “The evaluation did not find evidence that time savings have led to improved productivity.“
Could anyone show a change?
That department had an evaluation team, a comparison group and a published method, but its time savings still came from diaries the staff filled in about their own work.
The Department for Work and Pensions ran a larger trial, comparing 1,716 staff who had the tool with 2,535 who did not. Users reported saving 19 minutes a day, but its report is plain about where that number came from: “All outcomes (such as time saved or job satisfaction) were based on what participants reported.” And: “No pre-intervention measures were collected before Copilot was introduced.”
Your business has a before. Your records hold it.
Those departments asked thousands of people how it went because a survey was the instrument they had. A business of 20 or 200 people keeps a better before photo, because it keeps its records for other reasons: invoices, job tickets, the call log, the sent folder, the bank statement, the count of quotes that went out and the count that came back signed.
“We added AI. Are we better off? How would we know?”
The Prove-It Test: 5 steps from “I think so” to a number
The test runs on one tool.
Using AI now: start with the tool that renews next. Its renewal date is when you decide whether to keep paying for it.
New to AI: use the first tool you’re considering, and run steps 1 to 3 before it arrives. Then, on the day it goes live, you know what good looks like and where you’re starting from.
Bring in the people who use it most, or will, and the records you keep. Then work through 5 steps, in order. 📋
Steps 1 to 3: the result and the baseline
- Write down the result you want. Bernie Smith of Made to Measure KPIs has teams state the outcome “as though it has already happened.” Mine: “Every edition reaches publish in 5 drafts or fewer.” Yours might be “Our quotes go out the same day, and more of them come back signed.” If your first answer is an activity, “so we can write proposals faster,” ask “so that?” until you reach a result a customer or your profit and loss would notice.
- Map how the work runs today. Before any numbers, draw the journey: who does the work, each step they take, where it stalls, and where people get frustrated. Nielsen Norman Group defines a journey map as “a visualization of the process that a person goes through in order to accomplish a goal.”
- Put your records on the map, starting where it hurts. The map shows where the work stalls. Your records show how much. Nielsen Norman Group recommends layering data such as “customer-support call logs” onto a journey map to back up what people told you. At the step where the pain is, count the same kind of work, for the same customers or cases, over a set period. If there is no record, start counting today, and run the count for the same set period before you judge the tool.
Steps 4 and 5: the test and the decision
- Measure it now, 3 ways, against the same work done without the tool.
- Quality: did the work come out right, or did someone have to fix it?
- Time: how long did it take, start to finish?
- Experience: what do the people doing the work say about it?
Quality and time come from your records. Experience comes from asking, and it goes in its own column. In an analysis of 298 designs, Jakob Nielsen found that how people rate a system and how well it performs agree about 70% of the time, so each tells you something the other can’t. Where the whole team uses the tool, your baseline from step 3 is the comparison. - Check what it cost somewhere else, then decide. Every number you improve sits next to one that can get worse. Andy Grove, who ran Intel, argued in High Output Management for pairing every indicator with one that catches its side effect. If proposals go out faster, check revisions and the win rate. If the chatbot answers more customers, check repeat contacts and refunds. If drafts come faster, check how long a senior person spends fixing them. Then the team decides together: keep it, change it, or stop it, and names the date the test runs again.
Small and proven beats big and felt
The people who use the tool most are part of this for a reason. They know which part of the work got faster and who picked up the rest. They also need to be safe when the number comes back smaller than the story.
If you built the AI inventory from Edition 07, it is your list of tools to run through the Prove-It Test, starting with the one that renews next.
What else did the tool do?
Fewer late nights before a deadline and a customer who gets an answer without waiting on hold are outcomes too, and they only show up in your records if someone asks about them.
Kat Zhou, who created Design Ethically, calls this layers of effect. Primary effects are the ones a product is built for, “always intended and known.” Tertiary effects are “either unintended or unforeseen,” and she is plain that they “can be good or bad.” The late nights that stopped are a tertiary effect. So is the customer who now waits longer because the tool sends them in a circle.
Step 5 of the test, above, covers both kinds. Ask the people who use the tool, and the people on the other end of it, what changed for them. Write down what they say, with the date, next to the numbers from your records.
The developers in that study were sure. They learned otherwise because every task was timed.
The people on your team who swear by the tool may be right. The Prove-It Test is how you find out, and how you back them when they are.
Where does AI measurement debt sit in your organization?
Start with the AI Debt Diagnostic. 9 questions, about 5 minutes, free. It scores your organization across the 9 debts and shows where to start.
Every edition of this series is at digitalbard.co/perspectives.