Picture a mid-sized company's finance operations team in late 2026. A coding agent built most of their internal tooling this year, and the engineers who use it are happy. So the CFO asks the obvious question: if an agent can write and fix code, why can't one handle accounts-payable exceptions, the invoices that don't match a purchase order and today take four people to clear?
The pilot goes well. The agent reads the invoice, checks the ERP, emails the vendor, and proposes an adjustment. Then the controller asks three questions before it goes to production. If it posts a wrong adjustment, can we reproduce exactly why? Can we show the auditors what it was allowed to do and what it actually did? And what does it cost to prove it works, every time we change the prompt or the model? Nobody has a cheap answer. Six months later the project is "being re-scoped."
This article explains why that pattern is common, separates what the evidence supports from what is still opinion, and works through a cost model showing which design choices decide whether the numbers ever turn positive.
Where Agents Are Working, and Why There
The ROI problem is real, but it is narrower than "AI doesn't pay." Start with what the 2026 data actually says.
- Value from AI in general is showing up. In KPMG's Q3 2026 AI Pulse (314 U.S. leaders at $1B+ companies, fielded July–August 2026), nearly six in ten leaders report measurable business value from AI initiatives.
- Profit impact is flat. McKinsey's State of AI 2026 (1,719 respondents, fielded May–June 2026) found 37% attribute any EBIT impact to AI, essentially unchanged from a year earlier. Its "high performers," attributing 5% or more of EBIT to AI, are about 6% of respondents, also flat.
- Agents are scaling in specific places. The same McKinsey survey found 40% of large organizations report scaling AI agents, up from 27%. They are scaled most often in IT, knowledge management, and software engineering.
- Many agent projects won't survive. Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, citing "escalating costs, unclear business value, and inadequate risk controls." That is a forecast, not an observed rate.
Look at where agents are scaling: IT, knowledge work, software. In each, a skilled person reviews the output before it matters. A coding agent proposes a diff and an engineer merges it. A research assistant drafts and the analyst decides what to keep. A personal assistant books the restaurant you asked for, under terms of service you accepted. In all of these, the human in the loop is also the accountable party. That person absorbs the verification cost as part of their normal work, and nobody records it as an AI cost.
A business process removes that person by design. That is the point of automating it. Once they're gone, the organization owns every outcome: the wrong payment, the leaked customer record, the approval that should have been a rejection. It has to answer to auditors, regulators, and customers, and terms of service don't cover that. All the verification the human used to do quietly now has to be engineered, tested, and paid for explicitly. That is the accountability gap, and the rest of this article is about what it costs.
Four Walls Between a Pilot and Autonomous Operation
Moving a process from a deterministic workflow (fixed code paths, the same input gives the same output) to a probabilistic agent (a model chooses the next step) runs into four problems. None is new on its own. What's new is that the agent version hits all four at once.
1. Reproducibility, and context that degrades over long runs
Auditors and incident reviewers assume you can rerun a failure. With LLM agents, that assumption is weak at two levels.
At the inference level, "temperature 0" does not guarantee identical output. Thinking Machines Lab reported in 2025 that 1,000 runs of the same prompt on Qwen-3-8B at temperature 0 produced 80 distinct completions. The root cause is dynamic cloud batching combined with GPU parallel floating-point non-associativity: because parallel summation orders fluctuate across concurrent requests, micro-variations in intermediate logits can flip token choices. While specialized batch-invariant kernels can enforce determinism if you host your own models and absorb the throughput penalty, standard multi-tenant cloud APIs offer no such guarantee. In production, temperature zero is simply not deterministic.
At the agent level, the problem is harder. Anthropic's engineering team, describing its own multi-agent research system, wrote that agents "are non-deterministic between runs, even with identical prompts" and that they "are stateful and errors compound." A different tool result early in the run sends the rest of the run down a different path.
The reliability math makes this concrete. The τ-bench benchmark introduced pass^k: the chance an agent gets the same task right on all k tries. In Sierra's 2024 results, a GPT-4o agent that solved under 50% of tasks on average fell to about 25% on pass^8 in the retail domain. Newer models score higher, but the gap between "can do it" and "does it every time" is the gap a business process has to close. A simple illustration: if each step in a 10-step process succeeds 95% of the time, independently, the whole run succeeds about 60% of the time (0.9510 ≈ 0.60). At 99% per step it is about 90%.
Long runs add a second effect, often called context rot or context pollution. Chroma's July 2025 study of 18 models found performance "grows increasingly unreliable as input length grows," and that even a single distractor, text similar to but not the right answer, lowers accuracy, with the effect growing as context gets longer. An agent that keeps every past tool result, failed attempt, and intermediate draft in its context is building its own distractors. The longer it runs, the more of its context is noise.
2. Assurance and auditability
When a deterministic workflow posts a wrong adjustment, the investigation has a finite search space: the input, the rule, the code version. When an agent does, there are more suspects. Was it the data, the retrieval, the prompt, a tool returning something unexpected, or the model taking a different path this time? A trace log tells you what happened. It doesn't tell you whether it will happen again, and without reproducibility you can't test the fix against the original failure.
Regulation is moving in the same direction. The EU AI Act requires record-keeping and human oversight for high-risk systems. The AI Omnibus (Regulation (EU) 2026/1744, in force since July 27, 2026) moved the deadline for stand-alone high-risk systems such as credit scoring and hiring from August 2, 2026 to December 2, 2027. That's a delay, not an exemption. Outside regulated categories, internal audit and SOX-style controls ask the same question: who approved this action, and on what basis?
3. The cost of controls: manual design for the sake of automation
There is no shortage of mitigations. Each one is reasonable:
- Lower randomness: fixed temperature and seeds, batch-invariant serving.
- Pin an open-weight model so its behavior doesn't shift under you, and so evaluation runs on your own hardware cost less.
- Embed the agent inside a workflow so it can't leave approved paths.
- Guardrails on inputs, outputs, and tool calls.
- Strict grounding: answers must cite retrieved, approved sources.
- Formal policies the output is checked against.
Notice what the list does. Almost every item pushes the agent back toward a deterministic workflow, and every item has to be designed, built, and tested by someone. The result is the contradiction in this section's title: to automate the process, you hand-specify more of it than the old workflow ever needed. The old workflow left the judgment calls to clerks. The agent version has to write those judgment calls down as policy, guardrail, and test case. That is often the first time anyone has written them down at all.
Then you have to prove the controls work, and prove it again after every change to the prompt, model, tools, or policy. Because of pass^k, one passing run proves little. You need many runs per scenario, and that is the next wall.
4. Running cost and talent
Agents are token-hungry by design. Anthropic reported that its agents typically use about 4× the tokens of a chat interaction, and multi-agent systems about 15×. In its research evaluation, token usage alone explained 80% of the variance in performance. Put plainly, the usual way to make an agent smarter is to let it spend more, and each extra loop also adds context that can rot. Evaluation multiplies the bill: 500 test scenarios run 5 times each is 2,500 full agent runs per release.
Organizations are reacting. In KPMG's Q3 2026 survey, 74% now include cost reviews in AI approvals and 43% set usage or token budgets. In its Q4 2025 survey, 65% of leaders named agentic system complexity as the top barrier, and 76% said they would pay up to 10% more for candidates with strong AI skills. The specific skill that's scarce is narrower than "AI skills": someone who can design an agent's boundaries, write its evals, read its traces, and explain a failure to an auditor. Few teams have that person, and most can't hire one quickly.
What Hyperscaler Packages and Neuro-Symbolic AI Fix, and What They Don't
The platforms have seen these walls and are packaging answers. Two examples from AWS show the direction:
- Deterministic action controls. Amazon Bedrock AgentCore Policy, announced December 2025, blocks unauthorized agent actions using "deterministic controls that operate outside of the agent code." AgentCore Evaluations continuously inspects agent quality based on behavior.
- Formal verification of outputs. Automated Reasoning checks in Bedrock Guardrails (generally available August 2025) translate a policy document into formal logic, then check model outputs against it, which AWS says delivers "up to 99% verification accuracy."
The second is neuro-symbolic AI in production: an LLM paired with a symbolic system (formal logic, rules, or a knowledge graph) that can be checked rigorously. It is the most promising answer to the assurance wall, and it shows the dilemma clearly. AWS's own launch post says the policy must be built from your documents into rules, variables, and types, then tested and refined with annotated scenarios before it is reliable. The rigor is real, but someone has to model the domain first. When the business rules change, someone has to update the model and rerun the tests.
Knowledge graphs have the same shape. Microsoft introduced LazyGraphRAG because full GraphRAG's up-front indexing costs "may be prohibitive for some users and use cases." The lazy variant's indexing cost is 0.1% of full GraphRAG, which tells you how expensive the full version was. And indexing is the easy part. Designing a schema your business can trust, and keeping it current, is work for people who understand both the domain and the graph. Most organizations don't have them.
So symbolic methods move cost; they don't remove it. Cost shifts from runtime verification to up-front modeling and ongoing maintenance. That's often a good trade for a stable, high-stakes domain such as lending rules or eligibility checks. It is a poor trade for a process whose rules live in people's heads and change every quarter. In neither case does the system tune itself. The more rigorous the guarantee, the more explicit design it requires, which is the opposite of "fully autonomous."
A Worked Cost Model: Same Process, Two Designs
Back to the AP exception team. Every number below is illustrative: invented to show the structure of the decision, not measured at any company. Token prices are round numbers ($3 per million input tokens, $15 per million output), not any vendor's current list price.
Baseline: 6,000 invoice exceptions a month, cleared by 4 clerks at a fully loaded $8,000 a month each: $32,000/month, about 6 minutes per exception.
Design A, the autonomous agent. One long-running agent handles each exception end to end: investigates in the ERP, emails the vendor, posts the adjustment. Average run: 300,000 input tokens (context accumulates across loops) and 10,000 output, about $1.05. It succeeds on 85% of exceptions. The other 15% need a person to find and unwind what it did, which takes longer than doing it right the first time.
Design B, a workflow with a bounded agent step. A deterministic workflow handles intake, matching, posting, and logging. A short-lived agent does one thing: classify the exception and draft a proposed resolution, starting from a fresh context of about 25,000 tokens (about $0.10 a run). A policy check runs on every proposal. A person approves anything over a value threshold or anything that contacts a vendor, about 30% of cases, at 2 minutes each. About 5% of proposals are wrong and get redone at 8 minutes each.
| Monthly cost line | A: Autonomous agent | B: Workflow + bounded agent |
|---|---|---|
| Inference (6,000 runs) | $6,300 | $600 |
| Evaluation (500 scenarios × 5 trials × 4 releases) | $10,500 | $1,000 |
| Human time | $12,500 (900 unwinds at 15 min + 5% audit sample; ~250 h) | $5,000 (1,800 approvals at 2 min + 300 redos at 8 min; 100 h) |
| Agent engineering & operations | $8,000 (0.5 FTE) | $4,000 (0.25 FTE) |
| Total vs. $32,000 baseline | $37,300 (−$5,300/month) | $10,600 (+$21,400/month) |
| One-time build | $150,000; never pays back | $90,000; pays back in ~4.2 months |
The specific numbers will be wrong for your process. The structure is the point:
- Autonomy makes every cost line bigger. Longer runs mean more tokens per run, and evaluation multiplies that per-run cost by every scenario, trial, and release. In Design A, testing the agent costs more each month than running it.
- Failure cost depends on reversibility, not just failure rate. Design A's 15% failures are expensive because the agent already acted. Design B's 5% are cheap because nothing irreversible happened before a person or a policy check looked.
- Human time doesn't go to zero in either design. It shifts from doing the work to approving and auditing it. A business case that assumes zero human time is assuming the accountability gap away.
Where Business Agents Are Heading
If the evidence above is right, the realistic destination for business agents is not "fully autonomous." These three shifts are already visible, and each comes with what would prove it wrong.
1. Human-in-the-loop as the standard design, not a temporary crutch
The agent prepares, a person (or a deterministic policy) commits. Approval gates sit on actions that are irreversible, high-value, or customer-facing. This is already common practice: in KPMG's Q3 2026 survey, 49% of organizations have defined high-risk use cases where agents may not decide autonomously, and 75% of banking leaders don't let agents touch sensitive data without human oversight. The design work is choosing where the gates go so that people review a small number of decisions that matter, not rubber-stamp a queue of trivial ones. What would falsify this: sustained autonomous operation in a regulated process with audit sign-off and no gate, at scale.
2. Small, short-lived agents inside deterministic workflows
If long runs cause context rot and compounding errors, shorten the runs. Break the process into narrow steps, give each agent a single job, a fresh context, and a structured output, and let ordinary code pass results between steps. Anthropic's guidance on building agents makes the same distinction: workflows orchestrate LLMs through predefined code paths, while agents direct their own process. It recommends the simplest option that works. Multi-agent systems fit this direction only when each agent stays small. A swarm of long-running agents talking to each other multiplies the problem. What would falsify this: long-context models that hold reliability flat as runs grow, at a cost that doesn't grow with them.
3. Automated evaluation as the thing that flips ROI
Look again at the cost model. In Design A, evaluation is the largest machine cost; in both designs, human review is the largest cost overall. ROI turns positive when verification gets cheap. That means a regression suite of real scenarios run in simulation, with pass^k as the release gate instead of a single good demo, and LLM-as-a-judge scoring to reduce how much human review each release needs. Think CI/CD, applied to agent behavior. The platforms are building this (AgentCore Evaluations is one example). One caution: an LLM judge is itself a probabilistic component. It needs to be calibrated against human labels and spot-checked, or you have moved the trust problem rather than solved it. What would falsify this: eval suites that pass while production incidents keep coming, meaning the scenarios don't represent real traffic.
Where This Breaks
When Full Autonomy Can Be the Right Call
Some processes are high-volume, low-stakes, and reversible: tagging support tickets, routing internal requests, enriching CRM records, drafting first replies a customer can easily correct. If a wrong action costs little and can be undone in seconds, the accountability gap is small, and an autonomous agent with sampled auditing may pay off today.
The cost model also moves. Per-token prices for a given level of capability have generally fallen, and models keep improving on long, multi-step tasks. If per-run cost and pass^k improve enough, Design A's lines shrink. Re-run your model with current prices and your own measured reliability rather than trusting this article's numbers, or anyone's.
And Design B can fail too. Approval gates placed on every action turn reviewers into rubber stamps. A policy check that encodes last year's rules gives false confidence. Bounded agents still need someone who owns their evals.
The evidence also has limits. The survey figures above are self-reported by large organizations. Gartner's 40% is a forecast. The τ-bench figures are from 2024 models. Taken together, they support the narrower claim this article makes: the gap is between AI that helps an accountable person and AI that replaces one. They don't show that agents can't work in business processes.
The Decision Rule
Before moving any business process to an agent, answer three questions in writing:
- Who is accountable for each action, and what do they need to see to sign off? If the answer is "nobody" or "the vendor," stop here.
- Which actions are irreversible? Put a deterministic check or a human gate in front of each one, and keep the agent's job to everything before that point.
- What does it cost to verify a change? Price the eval suite (scenarios × trials × releases × cost per run) alongside inference, and count human review time honestly.
The decision rule is short. Give the agent the smallest job that still removes real work, keep its runs short, and gate every irreversible action. Then spend on automated evaluation until verification costs less than the labor the agent replaces. Autonomy is something you expand as evidence comes in, not where you start. The organizations getting returns from agents today aren't the ones with the most autonomous agents. They're the ones who decided who answers for each action before they automated it.
For the delivery-side version of this argument, where AI moved the bottleneck from writing code to trusting it, see Code Is Cheap, Judgment Is Scarce. For why AI-driven headcount cuts aren't producing returns either, see AI Layoffs Aren't Paying Off.
Sources
- McKinsey: The state of AI in 2026: On the road to ROI (Aug 25, 2026)
- KPMG: Q3 2026 AI Quarterly Pulse Survey (Sep 24, 2026)
- KPMG: Q4 2025 AI Pulse Survey (Jan 15, 2026)
- SD Times: Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027 (Jun 30, 2025)
- Anthropic: How we built our multi-agent research system (Jun 13, 2025)
- Anthropic: Building effective agents (Dec 2024)
- Sierra: τ-bench, benchmarking AI agents for the real world (Jun 20, 2024)
- Chroma: Context Rot (Jul 14, 2025)
- Analytics India Magazine: Thinking Machines on LLM nondeterminism (Nov 9, 2025)
- Mondaq: EU Digital Omnibus on AI enters into force (2026)
- Amazon: New Amazon Bedrock AgentCore capabilities (Dec 2, 2025)
- AWS News Blog: Automated Reasoning checks now available (Aug 6, 2025)
- Microsoft Research: LazyGraphRAG (Nov 25, 2024; updated Jun 6, 2025)
Make Verification Cheap
Whether the code comes from a person or an agent, the return depends on how cheaply you can prove it works. Our hands-on classes teach the verification discipline behind that: executable specs, automated quality gates, staging environments, and gated deployment pipelines.
Explore the Engineering Curriculum →