If you can’t evaluate an AI agent, why did you deploy it?

A practical framework for evaluating AI agents, setting the limits of their autonomy, and preserving the ability to replace them.

TL;DR

• Define acceptable performance before an AI system enters consequential use. Uncertainty about model behavior makes evaluation more necessary.
• Measure components, their interactions, and the complete business outcome. A convincing answer or a high average score can conceal serious failures.
• Grant autonomy within a tested scope, with enforceable permissions and meaningful human intervention.
• Retain control of business knowledge, evaluation assets, and operational dependencies so that replacing a supplier remains a practical option.

Putting an AI agent into production is a decision to delegate responsibility. Someone will rely on its analysis, accept its recommendations, or allow it to act in a business system. What evidence makes that delegation justified?

A Dataiku–Harris Poll survey, published on September 24, 2026, reports that 72% of the 685 CIOs surveyed cannot consistently measure whether all their agents deliver their intended business outcomes. This is a survey of executives’ reported experience, rather than an independent audit of the systems themselves. Nevertheless, the measurement gap is substantial.

This unease also appears in an October 29, 2026 report by Thomas Pontiroli and Mehdi Laghrari in Les Échos, a French business newspaper. Its headline includes the phrase “Nobody knows how to fire an incompetent agent” (translated from French).

Knowing that an agent is running tells you little about whether its work is acceptable. Evaluation must help determine whether deployment is justified, how much responsibility the system can carry, and when that responsibility should change.

Before granting an AI system operational responsibility, someone should have defined competent performance, assembled evidence that the system meets it, and established what happens when it falls short.

An organization that cannot evaluate an agent’s work has an unfinished deployment decision.

Deployment without evaluation can manufacture distrust

In my work with technology leaders, I repeatedly encounter AI initiatives entering use without a credible evaluation framework. This includes document assistants, systems coordinating several agents, and conversational business intelligence: asking questions about business data in ordinary language.

An evaluation is a structured test: give the system a representative task, inspect what it does, and judge the result against criteria agreed in advance.

The reasoning behind skipping this work is rarely stated outright. Models produce variable answers; quality is difficult to measure; users will tell us whether the system is useful. Gradually, a difficult engineering problem becomes an accepted blind spot.

That creates a damaging sequence. Errors reach users. Corrections are made without a reliable way to check their wider effects. Similar requests produce inconsistent experiences. Users become reluctant to rely on the service, and sponsors begin questioning the technology itself.

Some failures do reveal genuine technological limits. Evaluation is what helps distinguish those limits from incomplete business definitions, weak data preparation, faulty integration, or inappropriate permissions. Without it, an organization cannot diagnose its disappointment.

A restricted trial, with informed participants and limited consequences, can begin with modest evidence. The requirement should rise with the authority granted and the harm a mistake could cause.

The mistake is allowing a successful demonstration to become operational permission without an explicit change in the standard of proof.

Five identical answers can still be wrong

Suppose five finance employees ask an AI assistant the same question. Different answers would make the need to investigate obvious. Identical answers could still share the same mistake.

Consistency matters. But agreement alone cannot establish correctness.

Consider a question about commercial margin. Its answer depends on the agreed treatment of returns, discounts, supplier contributions, reporting periods, and the population being analyzed. Different answers may reflect different interpretations of those definitions. Making the model repeat one interpretation reliably would not settle whether it is the right one.

This is why conversational BI requires a governed business vocabulary. Technical teams often call this a semantic layer: a maintained description of what business terms mean and how they relate to the underlying data. For an executive, its purpose is straightforward. The organization decides what a metric means; the AI uses that definition.

Our ongoing work across retail, financial services, and industry repeatedly brings these boundaries into focus. The lessons here are generalized across engagements; they describe design principles, not completed deployments or measured client results.

A useful evaluation must distinguish several questions. Did the system understand the request? Did it apply the approved definition? Did it access the appropriate data? Was the calculation correct? Does the explanation accurately describe the result?

It must also recognize situations where answering immediately would be a mistake. If an ambiguity materially changes the result, asking for clarification can be the correct behavior. Where the evidence is insufficient, declining to conclude can be the correct outcome.

A system should be penalized for confident overreach, even when its answer sounds useful.

Evaluate the parts, their connections, and the business result

The distinction between a model and an operational system is essential.

A model generates or interprets content. An agent combines that capability with instructions, tools, and a process for deciding what to do next. Its performance also depends on the information supplied to it, the actions it can take, and the behavior of other services it uses.

A document assistant illustrates this without requiring an elaborate architecture. It may first search a document collection, then select relevant passages, then generate an answer. This approach is commonly called retrieval-augmented generation, or RAG. If the search misses the decisive document, improving the wording of the final answer will not repair the failure.

For more complex systems, I would organize evaluation at three levels:

The middle level is easy to overlook. One agent may identify a missing fact, while another treats its provisional output as established. Two agents may repeat the same error because they share a misleading source. An apparently successful retry may repeat an action already performed.

Consequently, system quality cannot be calculated by simply adding component scores. Those scores help locate weaknesses. Tests of the assembled workflow establish whether the pieces work together under realistic conditions.

A systematic review published in Artificial Intelligence Review on September 15, 2026, makes a related methodological point: an agent benchmark score depends on the capability being measured, the scoring method, and the environment. Among seventeen selected benchmarks examined in detail, none reported a complete standardized run-cost record. These findings concern that research sample, but they underline a practical purchasing rule: a headline score is insufficient evidence for your business process.

Build evidence that can change a deployment decision

Evaluation begins with representative work and explicit expectations. Choose real questions, documents, or tasks. Ask the relevant business experts to define acceptable results, material errors, and circumstances requiring clarification or escalation.

Include ordinary work, difficult work, and requests the system should decline. If experts disagree about the expected answer, resolve the policy or definition where possible; otherwise record the legitimate uncertainty. An evaluation dataset can expose organizational ambiguity as effectively as technical weakness.

Keep some cases separate from the examples used to improve the system. Otherwise, apparent progress may reflect familiarity with the test. Record the relevant data and system versions so that a changed score can be investigated.

The scoring method should fit the task. A financial calculation can often be checked against an independently validated computation. An extraction task can be compared with reviewed source documents. A synthesis may require a rubric: explicit criteria for completeness, factual support, and treatment of uncertainty.

Another language model can help apply such a rubric at scale. Its judgments still need calibration against expert review, especially for consequential mistakes. Anthropic’s January 2026 engineering guidance explains why agent evaluations need multiple trials and why the actual outcome must be checked separately from the agent’s account of what happened. A system declaring success is not evidence that the intended action succeeded.

Executives should then ask for results broken down by task difficulty and consequence. A high overall score may conceal poor performance on the very cases that justify the investment. Repeated tests reveal instability; the sample size limits how confidently the results can be generalized.

Measure the relationship between quality and coverage, too. A system that refuses most requests may look accurate while providing little value. A system that answers everything may conceal unacceptable error rates. The business decision concerns how much work can be handled at the required quality, with how much human assistance.

This evidence must have consequences. It should determine which uses are permitted, which remain supervised, and which are withheld. Re-evaluate after material changes to models, instructions, tools, data definitions, or permissions. Combine those tests with production sampling and incident review, because real usage will introduce situations the original test set missed.

This is consistent with the direction of Gartner’s September 8, 2026 research summary on reliable agents: reliability requires engineering across development, deployment, and operation. A launch assessment cannot carry that responsibility indefinitely.

Human supervision needs a capacity plan

Delegating work to AI also creates work for the people who review outputs, resolve exceptions, and authorize consequential actions. The design must account for how much reviewing, correcting, and coordinating those employees can absorb.

Anthropic’s October 16, 2026 study of its own workforce identifies human attention as a bottleneck. The authors estimate that reading all interactive agent output would occupy more than half a working week even for nontechnical groups. This is evidence from one unusually intensive adopter, not a forecast for every enterprise. It illustrates why supervision needs to be designed and costed.

A human approval step is useful only if the reviewer receives the information needed to decide. For a proposed change, that could mean the requested action, supporting evidence, differences from the current state, and unresolved assumptions. A large transcript or an unexplained confidence score may provide little help.

This is a recurring lesson from work on industrial processes: preparing a proposal and authorizing its execution are distinct responsibilities. The interface, permissions, and workflow should preserve that distinction. A system should not acquire execution rights merely because its recommendations are persuasive.

Accountability must follow the decision. The business process owner approves acceptable outcomes and escalation rules. The data owner approves definitions and access conditions. The AI service owner maintains behavioral evaluations. The platform owner operates the service and enforces technical controls. A designated incident owner can suspend execution and coordinate recovery.

These controls must exist outside the model’s instructions. Access restrictions and spending limits require enforcement by the surrounding software. Evaluation tests whether the controls hold; it cannot replace them.

The economics of autonomy must include that human workload. The useful economic measure is the cost of an accepted business outcome, including computation, retries, supervision, correction, and operation. Compare it with the existing process at a comparable quality level. A cheaper model that creates more review work may produce a more expensive service.

Sovereignty determines whether replacement is possible

The ability to replace an AI supplier depends on where business knowledge and operational control reside.

An organization can have multiple suppliers while remaining dependent on a proprietary platform for its business rules, stored context, integrations, or execution history. Replacing the model may be relatively simple while replacing the operational system is prohibitively difficult.

I favor a deliberate preference for European technologies and infrastructure that strengthen technological autonomy. Selection should examine the actual allocation of control: who operates the service, who can access the data, which dependencies remain external, and what the organization can maintain or replace independently.

The European Commission’s Cloud Sovereignty Framework, explained in June 2026, provides a useful broader reference. Its assessment includes legal, operational, supply-chain, technological, and data dimensions. Geography alone does not describe that entire set of dependencies.

Open standards, accessible source code, and models that can be operated on chosen infrastructure can improve control. They still require skills, computing resources, maintenance, and an understanding of licensing and supply dependencies. Sovereignty should be assessed against those operating realities.

For agentic systems, evaluation adds a particularly important capability: the ability to qualify an alternative. Retain business definitions, representative test cases, acceptance criteria, and usable execution records independently of the supplier wherever possible. Preserve the knowledge needed to rebuild the service.

Then test the exit. Run a candidate replacement against the same requirements. Measure changes in quality, supervision, cost, and response time. Identify what must be adapted and how long a transition would take.

Evaluation makes technological independence actionable. Without it, an organization may have the contractual right to leave while lacking the evidence needed to do so responsibly.

Make continued operation an explicit decision

The strongest objection is that all this takes time while businesses are under pressure to move. It does. Evaluation requires domain expertise, representative data, and engineering effort. Its depth should remain proportionate to the consequences of failure.

The practical response is to narrow the initial scope until it can be evaluated credibly. Expand authority as evidence improves. Reserve human attention for decisions where it adds value, and retain a workable fallback when the service deteriorates.

For systems already deployed, begin with the uses carrying the greatest consequences. Establish their owners, expected outcomes, permitted actions, and current quality. Restrict the scope where the evidence cannot support the authority already granted.

Withdrawing an agent’s authority can mean revoking a permission, returning a task to supervised operation, replacing a component, or retiring the service. Stopping new actions also requires checking work already in progress and identifying effects that need correction.

The standard is sufficient evidence for a defined responsibility, maintained throughout the service’s life.

Before the next agent enters production, the executive question should be explicit: what evidence earns this system the authority we are about to give it, and what would cause us to withdraw that authority?

For the organizational model behind these controls, see why enterprises need an AI Delivery Factory. For the infrastructure choices that preserve control, read AI agents as critical infrastructure.