When Metrics Reward AI Usage Instead of Real Work

Amazon recently shut down an internal AI usage leaderboard called KiroRank after employees reportedly started optimizing for the leaderboard instead of useful work. The system ranked people by AI token usage. Some employees figured out the obvious loophole: if the metric rewards more tokens, then the winning move is to burn more tokens.

That is how you get engineers asking AI agents to perform low-value work just to climb a chart. Not because the work mattered. Not because the output shipped. Because the scoreboard made usage look like productivity.

Amazon senior vice president Dave Treadwell reportedly told employees: "Please don't use AI just for the sake of using AI." That sentence is almost too clean. It captures the whole failure. The company wanted to encourage AI adoption, but the metric pushed people toward AI theater.

This is not really just a story about Amazon. Meta is pushing AI usage into employee goals too. The broader story is what happens when companies cannot measure real value, so they measure whatever is easiest to count.

The Problem Wasn't AI. It Was the Incentive.

Token usage is easy to track. It is objective. It updates automatically. It can be turned into a leaderboard, dashboard, adoption report, or executive slide.

It also tells you almost nothing by itself.

A developer can spend 500 tokens to find the right answer. Another developer can spend 500,000 tokens generating bad code, useless review comments, fake summaries, and agent loops that never produce anything worth shipping. If the company only tracks token volume, the second developer looks more "AI-native" than the first one.

That is the same old measurement failure in a new costume. Lines of code were never a good productivity metric. Number of commits was never a good productivity metric. Number of meetings attended was never a good productivity metric. Now token usage is becoming the AI version of the same mistake.

The metric is not neutral. Once people know what is being measured, they start optimizing for it. If the metric is disconnected from value, the organization gets more metric and less value.

Why This Keeps Happening

The uncomfortable part is that management has a real problem here. AI infrastructure is expensive. AI tools are being pushed across engineering orgs. Executives need to show that the spending is producing output.

So they look for measurable signs of adoption. How many engineers are using AI? How often? How many tokens? How many AI-assisted actions? How many generated pull requests? How many summaries, reviews, or agent calls?

Those numbers are convenient because they are visible. But visible does not mean meaningful.

The real questions are harder — and they don't line up with the easy ones:

What's easy to count	What actually matters
Tokens used / AI-assisted actions	Did the feature ship faster?
Generated pull requests	Did the defect rate go down?
AI review comments	Did review quality improve?
Agent calls	Did incident frequency decrease?
Adoption % (how many engineers, how often)	Did latency, reliability, or conversion improve?
Summaries produced	Did AI remove a real bottleneck — or just make more artifacts?

The right-hand column is harder to measure because it requires context. It requires judgment. It requires knowing what the team was actually trying to accomplish.

That is why companies fall back to easy numbers. Easy numbers scale. Judgment does not.

This Connects to Layoffs Too

I wrote earlier that layoffs are not always random. A lot of them happen because companies are bad at measuring real value. When leadership cannot clearly see who creates leverage, they use proxies: title, headcount, project visibility, manager perception, cost center, or whatever dashboard exists.

The Amazon AI leaderboard is the same failure pattern, just smaller and cleaner.

The company wanted to know whether engineers were becoming more productive with AI. Instead of measuring shipped outcomes, it measured AI consumption. Engineers responded to the incentive. The result was not better engineering. It was more token burn.

This is why "data-driven management" can become dangerous when the data is shallow. A bad metric does not become good just because it is quantitative.

Meta Shows Where This Is Heading

Amazon isn't alone. Meta shows where this is heading: AI usage is becoming a measurable workplace expectation, not just an encouragement. Reporting on internal Meta documents describes concrete targets — one engineering org's first-half-2026 goal that 65% of engineers write more than 75% of their committed code with AI tools, another division aiming for 50–80% AI-assisted code, and a company-wide push for 80% adoption of tools like DevMate, Metamate, and Gemini among mid-to-senior engineers. The stated ambition is to become "AI-native."

Meta employees have reportedly gamed those usage tables the same way Amazon's did — driving up consumption without producing more useful output. I'd still treat the specifics as reported rather than confirmed, but the direction is unmistakable. Once AI usage becomes a goal, employees naturally ask what exactly is being measured: AI-written code? AI-assisted diffs? Agent calls? Review comments? Time saved? Output shipped?

If the metric is tied to real outcomes, it can push better engineering. If it is tied to raw activity, it recreates the failure Amazon ran into.

Meta also shows the downstream pressure AI creates. A recent Meta code-review paper reports that significant lines of code per human-landed diff grew 105.9% year over year — enough of a surge that Meta built RADAR, an automated risk-aware code review system, just to keep up. That is the more serious version of the problem: AI can increase output volume faster than the organization can verify quality.

So the question is not just "are engineers using AI?" The question is whether AI-created output can be reviewed, validated, landed, and operated safely.

That is the difference between AI productivity and AI throughput theater.

What a Better AI Metric Looks Like

A better metric would not ask, "How much AI did you use?"

It would ask, "What useful outcome did AI help you produce?"

For software teams, that means measuring AI usage against engineering outcomes:

Cycle time: Did AI reduce the time from task start to merged code?
Review quality: Did AI catch real issues, or did it generate noisy comments?
Defect rate: Did AI-assisted changes cause fewer bugs or more bugs?
Operational reliability: Did AI help reduce incidents, regressions, or manual toil?
Business impact: Did the work improve revenue, retention, activation, cost, or user experience?
Cost efficiency: Did the value of the output justify the model and infrastructure spend?

Token usage can still be useful, but only as a cost input. It belongs in the denominator, not the numerator.

Amazon itself seems to have reached the same conclusion. After scrapping the leaderboard, it reportedly replaced raw token counts with a metric it calls "normalised deployments" — an attempt to track meaningful AI-driven work instead of consumption. Whether or not that exact metric is the right one, the instinct is correct: stop counting the input, start counting the result.

The goal should not be:

More tokens = more productivity

The goal should be:

More useful output per dollar of AI spend

That is a very different management philosophy.

The Engineer's Version of This

For individual engineers, the lesson is simple: do not sell yourself as someone who "uses AI a lot."

That is weak positioning. Anyone can burn tokens.

The stronger version is:

I use AI to reduce cycle time, improve debugging, strengthen tests,
speed up code review, and ship reliable features faster.
I validate the output instead of blindly trusting it.

That tells a company you understand the tool and the failure mode. AI is not the achievement. The shipped result is the achievement.

This also matters in interviews. If someone asks how you use AI, the answer should not be a list of tools. It should be a workflow:

where AI helps,
where you keep human review,
how you verify correctness,
how you measure whether it actually improved the work.

That is the difference between AI adoption and AI theater.

The Real Lesson

The Amazon leaderboard failed because it rewarded the wrong thing. It made AI usage visible but did not prove AI value.

That is the deeper problem many companies are running into. They want AI to improve productivity, but productivity is hard to measure. So they measure activity. Then employees optimize for activity. Then leadership gets a dashboard full of numbers that look scientific and mean very little.

This is how bad incentives scale.

The companies that handle AI well will not be the ones with the highest token charts. They will be the ones that can connect AI usage to shipped work, business outcomes, and engineering quality.

Everything else is just expensive noise.