In April, more than 85,000 Meta employees showed up on an internal leaderboard called “Claudeonomics.” The ranking criterion: how many tokens they burned. 60 trillion in thirty days, with some people leaving agents running for hours just to climb the board.
This week, an internal Amazon memo made public called token volume and lines of generated code “deeply flawed” metrics. Amazon had already shut down its own leaderboard in May, after realizing teams were gaming the meter. Microsoft told its engineers back in August that tokenmaxxing “is not what we are optimizing for.”
We’ve seen this movie before. In the ‘90s, we counted lines of code. Later, story points. Every time, the metric ended up being managed for its own sake, and stopped saying anything about the value produced. That’s Goodhart’s law: when a measure becomes a target, it ceases to be a good measure. Agents don’t change the law. They just run it much faster.
Helping companies transform their tech and product organizations is my job. And as soon as agents come up, the same question lands on the table: how do we know this is creating value? My read is that we’re very often looking at the wrong counter.
90% adoption. Now what?
DX, a developer productivity research firm, looked at more than 500 organizations for its Q2 2026 report. AI adoption is above 90%. 52.7% of code is now AI-generated, up from 19% a year earlier. The median pull request grew from 42 to 72 lines. Median quarterly AI spend went up 28x.
And on the other side? Maintainability is up 3.8%. Confidence in changes shipped to production is down 6.1%. The summary fits in one line: adoption is done, the ROI isn’t there.
Meta learned this at scale with Project OT, which aimed to hand up to 60% of some teams’ work to agents. More code, not more features. The second planned wave of layoffs was called off. Ars Technica put it plainly in October: AI coding agents generate more code, but not more software.
This isn’t a tooling problem. It’s a problem with what we choose to look at.
More code is inventory first
In lean thinking, overproduction is considered the worst kind of waste. Not because it’s expensive in itself, but because it hides all the others. Producing faster than the rest of the chain can absorb doesn’t create value. It creates waiting.
Code that’s generated but not reviewed, not merged, not used is exactly that: inventory. And once merged, it becomes a liability to maintain. James Shore puts it well: the AI we need is the one that lowers the total cost of owning code, not the one that writes it faster. David Crawshaw points to another side effect: the AI productivity of whoever opens a pull request can wreck the productivity of whoever reviews it.
Simon Willison, commenting on Andrej Karpathy, describes a software version of the Jevons paradox. When producing code becomes almost free, scarcity moves to judgment and verification. Put differently, the bottleneck has moved. It’s no longer at the keyboard. It’s in review, in decisions, in getting to production.
And optimizing a step that isn’t the bottleneck is the first thing you learn not to do when you look at a value stream end to end.
Token price says nothing. Cost per accepted task says a lot.
The unit that makes sense is cost per accepted task: a task that passes the tests, gets reviewed, merged and shipped. And that cost includes everything. Tokens, human time, rework, post-release bugs, and failed attempts. Florian Bruniaux adds a point I find essential: separate waiting time from working time.
The experiment published by Martin Fowler and Naresh Jain shows why this changes everything. The same service was built ten times per configuration and checked against 178 acceptance tests. The strong model succeeded 10 out of 10, with or without a harness, and the harness made it about seven times more expensive. The cheaper model went from 3 out of 10 without a harness to 10 out of 10 with one. Once harnessed, it came out about a third cheaper per run than the strong model on its own. Small sample, but the lesson is clear: the right answer depends entirely on the unit you look at.
Mendral found the opposite when switching to a frontier model: fewer tokens, fewer retries, and a lower bill. The most expensive model per token can be the cheapest per successful task. Max Woolf reminds us that in agentic coding, cache reads make up 96 to 99% of tokens. The sticker price per token tells you almost nothing about real cost.
Cloudflare shows what managing by unit of work looks like. The cost of an AI code review went from $1.19 to $2.46, then down to $0.77 after optimization, with median duration cut from 4:09 to 1:32. We’re no longer talking about consumption. We’re talking about the price of an outcome.
A metric on its own always ends up lying
Anthropic’s Claude Code team just published its numbers from June to September. Merged pull requests more than doubled. Over the same period, the share that had to be reverted fell by more than 40%. And the share of pull requests with substantive review comments went from 16% to 54%.
These are internal metrics, so self-reported. But the shape is right: a throughput indicator, always paired with a quality one. The doubling alone wouldn’t mean much. The drop in reverts next to it is what gives it meaning.
The UK’s National Cyber Security Centre documented the reverse in security operations centers. Push hard enough on detection and response times, and the quality of alert triage degrades. Teams move fast, on the wrong things. Goodhart again.
We don’t need to invent anything here. Since Accelerate, the DORA metrics have come in pairs: deployment frequency and lead time on one side, change failure rate and time to restore on the other. We just tend to forget that whenever a new tool shows up with its own counters.
The work nobody counts
According to Workday, employees spend 6.4 hours a week checking and fixing what AI produces. Harvard Business Review has a word for it: “botsitting.” It’s real work, and it shows up in none of the announced gains.
HBR also documents the flip side. Supervising several agents in parallel is exhausting. Getting rid of repetitive tasks is a relief. Annie Vella, who rereads fifty years of automation research through the lens of coding agents, describes engineering work sliding toward supervision. Anthropic, looking at its own teams, reaches the same conclusion: human attention is becoming the bottleneck.
That’s where the manager’s role changes. Protecting your team’s attention becomes a steering topic, not just a wellbeing one. And the bill follows dynamics few leadership teams see coming. At Rippling, token consumption was growing 80% a month, and 10 to 15% of employees generated 60% of the bill. The company ended up building its own per-employee ROI tool.
“It’s 1997”
Torsten Slok, Apollo’s chief economist, made waves a few days ago. Across what he calls the “S&P 493,” the S&P 500 minus the tech giants, neither margins nor revenue per employee have moved. Zero visible financial payoff.
Laurie Voss answered with a line I find spot on: “It’s 1997.” We’ve been through the productivity paradox with IT before. The gains came, but sector by sector, and only for those who had reorganized how they work. PwC’s 2026 study points the same way, with figures to take as self-reported: 20% of companies capture 75% of the gains, and they’re the ones aiming for growth, not just productivity.
This is where I come back to a conviction I’ve held for a long time. You don’t create value by plugging agents into an unchanged organization. Conway’s law applies to agents too: the system you build mirrors the structure that builds it. If flows, ownership and bottlenecks don’t move, agents just speed up producing the same thing into the same traffic jams.
To be fair, there are more skeptical voices too, like Ed Zitron, who argues inference costs are growing faster than measured gains. I’m not sure he’s right. But as long as we measure consumption rather than outcomes, nobody can prove him wrong.
What I propose to do, concretely
No miracle framework here. But these are the workstreams I propose to tech and product leadership teams that want to stop counting tokens.
Map one value stream end to end, from intent to customer usage, before adding agents to it. Find where work waits, and where the real bottleneck sits.
Write down the intent behind each initiative: what problem, for whom, and how we’ll know it’s solved. Addy Osmani calls the gap “intent debt”: goals that are never written down cap what agents can deliver. That’s the whole point of the Intent-Driven Engineering approach I’m working on.
Instrument cost per accepted task on one or two pilot streams, including tokens, human time, rework and failed attempts. Better one accurate number on a narrow scope than a company-wide dashboard counting consumption.
Pair your metrics. Every throughput indicator comes with its quality indicator: share of reverted changes, post-release incidents.
Measure review and supervision time, and make it a management topic: how review is organized, how the team’s attention is protected.
Take tokens consumed, share of AI-generated code and adoption rate out of value steering. They belong in FinOps tracking, nowhere else.
Rework the organization where the flow gets stuck: ownership, team interfaces, the role of reviewers. That’s where the gap opens between the companies capturing the gains and everyone else.
What to remember
Tokens, code volume and adoption rate measure effort, not value. Meta, Amazon, Microsoft and Accenture are all walking them back.
Adoption is above 90%, the ROI isn’t following: more code, not more software.
Code that isn’t reviewed, merged or used is inventory. The bottleneck has moved to review and decision-making.
The right unit is cost per accepted task, failed attempts and human time included. The most expensive token can buy the cheapest task.
A throughput metric without a quality metric always ends up gamed.
Supervision time is a real cost, missing from every announced gain.
The gains go to companies that reorganize their flows, not to those that plug agents into what already exists.
No written intent, no way to measure value.
We can’t measure everything yet, and that’s fine
OCTO, a French tech consultancy, published a candid write-up in October: eight projects run by agents, with no human code review. Of the four quality questions they asked, two still have no metric available. That’s not an admission of failure. It’s the real state of the art.
Many of the figures cited here are self-reported, coming from the companies’ own communications. Published, verifiable measurements are still rare. But that’s no reason to fall back on whatever is easy to count.
The question I’ll leave you with
Moving an organization from measuring effort to measuring value is exactly the kind of transformation I work on: rethinking flows, setting the right metrics, bringing teams along.
So, fellow tech and product leaders: what’s really on your agents’ dashboard today? And what would you take off it?
If you’ve lived through this, on either side of the table, hit reply — I read everything.
Talk soon, Nicolas
PS: post written with Claude, of course :)
Sources
Meta’s “Claudeonomics” leaderboard (The Decoder)
Microsoft: “Tokenmaxxing is not what we are optimizing for” (404 Media)
Companies scrambling to stop employees maxing out AI budgets (TechCrunch)
Meta cancels Project OT (The Next Web)
AI coding agents generate more code, but not more software (Ars Technica)
James Shore: You need AI that reduces your maintenance costs
Florian Bruniaux: Du code au merge (in French)
Martin Fowler and Naresh Jain: Are harnesses worth the cost?
Workday: 6.4 hours a week monitoring AI (Le Monde Informatique, in French)
HBR: How much time do your employees spend botsitting? and When using AI leads to brain fry
Rippling built an employee ROI tool (TechCrunch)
Apollo: AI delivers zero financial payoff to most companies (TechCrunch)
OCTO: on faith or on evidence? (in French)