Counting the Wrong Thing: Why Developer Performance Metrics Break Down When AI Changes the Unit of Work

Most engineering orgs still run performance conversations off some mix of the same handful of numbers: story points delivered, pull requests merged, commits pushed, tickets closed, sometimes still lines of code. These numbers keep updating every sprint. They look objective. And increasingly, they're measuring something that has quietly stopped meaning what it used to.

This isn't a claim that metrics are bad or that engineering performance can't be measured—it's narrower than that. Two specific things break, for two specific and different reasons, and conflating them leads to the wrong fix. One is a data-quality problem. The other is a unit-of-measurement problem. Only one of them can be patched with a better dashboard.

What These Metrics Actually Assume

Velocity and story points assume a ticket is a stable, comparable unit of effort. PR count and commit count assume output volume tracks with contribution. Lines of code makes the same assumption more crudely. Tickets-closed and cycle-time-per-ticket assume the ticket is the right grain to measure delivery at. Every one of these depends on a single hidden premise: that the unit of work is standardized and scoped to one role, moving through a queue where a person's throughput can be sensibly compared to the next person doing the same kind of work.

That premise held reasonably well in the assembly-line model of software delivery, where a PM's ticket, a frontend engineer's ticket, and a QA engineer's ticket were each a known, bounded slice of a larger feature. It's the same queue this site has argued AI is dismantling from the delivery side—multi-functional builders carrying features end-to-end, and sprint ceremonies built to coordinate a queue that's shrinking. Performance metrics are the last place that queue's assumptions are still load-bearing, because nobody redesigned the review process when they redesigned the delivery process.

Failure Mode One: Gamed, Not Wrong

The first failure doesn't require any change in how a team is organized. If an engineer keeps the exact same scoped role—say, backend feature work—and simply uses an AI assistant to produce more of it, PR count, commit count, and story points all keep functioning as metrics. They just stop functioning as good ones, because the effort-per-unit-output relationship they were calibrated against no longer holds. Drafting a PR used to cost enough time that volume was a weak but real proxy for effort. AI collapses that cost, so volume becomes cheap to produce regardless of whether the underlying judgment kept pace.

This is a data-quality failure, not a category error: the number still describes something real, it's just no longer the thing anyone thinks it's describing. An engineer who ships three times as many PRs this quarter might be doing three times the valuable work, or might be shipping AI-drafted changes with proportionally less review depth per change—and from the dashboard, those look identical. The failure is invisible exactly when it matters most, because the metric goes up either way.

Failure Mode Two: No Longer Comparable

The second failure is structural, and it only shows up once work actually goes cross-functional. If one builder now carries a feature end-to-end—scoping, backend, frontend, tests, and shipping—their "one PR" or "one ticket" is not the same unit as a specialist's PR or ticket anymore. It might represent what used to be five people's five tickets. There is no valid arithmetic that makes that number comparable to a QA engineer's ticket count, because the two numbers no longer measure the same thing at all.

The sharper problem sits with the specialists who shift roles rather than disappear. The specialists who remain valuable move from gatekeeping individual artifacts to designing the guardrails builders operate inside—the security engineer who used to review every PR now defines the policy-as-code checks that run automatically; the QA engineer who used to execute test scripts now builds the harness every builder uses. Both of them will show a collapsing ticket count and PR count on paper, in the same quarter their judgment work got measurably more valuable. A performance process still anchored to legacy throughput numbers will read that as decline. It's the opposite.

Same Quarter, Two Engineers

Put both failures side by side on the same review cycle and the mechanism becomes concrete. This is illustrative—two composite engineers, not real performance data—but it's the shape these conversations actually take once AI is in the workflow.

Engineer A is a senior QA specialist who spent the quarter building an automated tenant-isolation gate and catching three authentication-boundary defects in review before they merged. Their ticket count is down sharply—because they're not filing tickets anymore, they're preventing the need for them. Engineer B is a cross-functional builder shipping small features solo with AI assistance; their PR count nearly quadrupled, and on a throughput dashboard that's the strongest quarter on the team. It's also the quarter two of those PRs needed an emergency rollback, because review time didn't scale with PR volume and a chunk of reviews became rubber stamps under time pressure. Both stories are invisible to the metric that would normally anchor their review.

What the Industry's Own Data Already Shows

This isn't a hypothetical risk. DORA's 2025 State of AI-assisted Software Development report—the closest thing the industry has to an authoritative annual benchmark on delivery performance—found that 90% of surveyed developers now use AI at work and more than 80% believe it has increased their productivity. For the first time since DORA began tracking AI adoption, 2025 showed a positive relationship between AI adoption and software delivery throughput. It also showed AI adoption continuing to correlate negatively with delivery stability. The report's own explanation is direct: AI accelerates development, and that acceleration exposes downstream weaknesses—without strong automated testing, mature version control, and fast feedback loops, more change volume produces more instability, not more value.

That is failure mode one, measured at industry scale: throughput went up in exactly the way a PR-count or velocity dashboard would celebrate, while the metric that would have caught the cost—change failure rate, one of DORA's other three keys—moved the other direction. A team reading only the number that improved would have drawn the opposite conclusion from the one the fuller data supports.

The SPACE framework, first proposed in 2021 and still the most cited corrective to activity-only measurement, made the underlying argument years before AI coding assistants existed: activity is one of five dimensions of developer productivity, not a stand-in for the other four, and optimizing it alone reliably produces the wrong behavior. SPACE doesn't escape the AI era unscathed, though—several of its own activity and efficiency sub-metrics (commit frequency, PR throughput, time-in-review) are derived from the same git and ticketing data as the legacy metrics they were meant to correct, and are just as gameable once drafting cost collapses. A framework built to broaden the picture still needs its narrowest inputs re-examined.

What to Measure Instead

Rather than patch the old metrics, it's worth sorting what a performance process actually needs into three tiers, and being explicit about which tier each existing metric belongs to.

System and outcome signals sit at the top: cycle time from decision to production, change failure rate, and incident rate tied to specific changes. These are the same measures this site has already argued should replace ticket-count velocity at the team level, and they resist gaming for a structural reason—they describe what happened to the system, not how much activity one person generated inside it. Judgment-quality signals sit below that: review catch rate (did this person's review find something that would otherwise have shipped), postmortem contributions, and architectural decisions that measurably avoided rework. These require a human to actually read the substance of the work rather than count its artifacts, which is precisely why they resist the same gaming—there's no volume knob to turn. The retired tier is explicit: per-person ticket counts, PR counts, and lines of code, and especially the practice of splitting one cross-functional feature's credit back into per-discipline percentages after the fact. That last one doesn't just fail to measure the work—it actively pressures people to re-fragment work they'd otherwise do end-to-end, just to generate a legible number.

Where This Breaks

The proposed replacement has its own costs worth naming rather than assuming away. Judgment-quality signals take real manager time to evaluate honestly, and they're more exposed to halo effects, recency bias, and simple visibility—the engineer whose good catches happen in public review threads will look better than the one whose catches happen in a quiet Slack thread, regardless of actual impact. Structured rubrics and calibration sessions across managers reduce this but don't eliminate it. System-and-outcome metrics carry a different risk: because they're measured at the team or system level, they can hide a genuinely underperforming individual inside an otherwise healthy system, at least until an explicit review looks for that specifically. Neither of these is a reason to keep the old metrics—the old metrics have the same bias problems plus the gaming problem—but a team adopting this framework should build the individual-level check in on purpose rather than assume outcome metrics alone will surface it.

A Decision Rule, Not a Dashboard

Three questions can sort an existing performance process the same way the sprint-ceremony framework sorted standups and story points:

1. Does this metric assume a standardized, role-scoped unit of work? If the team has any cross-functional builders, that assumption is already false for them, and the metric should be retired for that population rather than averaged against specialists it still fits.
2. Can this metric go up through volume alone, without a human reading the substance behind it? If yes, treat a rising number as a prompt to look closer, not as a conclusion—exactly the DORA pattern of throughput up, stability down.
3. Does this signal describe the system's outcome or one person's activity inside it? Outcome signals are harder to game and belong at the top of a review; activity signals belong nowhere near a promotion decision on their own.

This is a hypothesis, and it's falsifiable the same ordinary way the sprint argument was: run one review cycle where a team's performance conversations lean on outcome and judgment-quality signals instead of individual ticket and PR counts, alongside a comparable team that keeps the legacy metrics, and track defect-escape rate, retention, and manager-reported confidence in the resulting ratings. If the pilot team's ratings turn out less defensible, or good engineers start leaving because the process feels vaguer than a number on a dashboard, that's evidence the old metrics were doing more real work than this argument gives them credit for—and the fix should narrow back toward whichever specific number still held up, not discard measurement altogether.