An AI agent says it's "done": how do you verify it actually did the job?

Most teams building with AI agents hit the same uncomfortable moment sooner or later. The run finishes, the log says "completed", the status turns green, and someone asks the obvious question: did it actually do the job? Not "did it stop", but "did it succeed". Those are two different things, and confusing them is one of the most common ways AI projects quietly disappoint the people who paid for them.

This article is about that gap, and about the practical habits that close it.

Finished is not the same as succeeded

An agent is, at bottom, a loop: it reads a task, decides on an action, calls a tool, looks at the result, and repeats until it decides it's done. "Done" is a judgement the model makes, often from its own reasoning and the tool outputs it can see. It is not a guarantee.

A few failure patterns show up again and again:

• Premature completion. The agent marks the task complete after a partial result, because the next step was ambiguous or the tool returned something it misread.
• Silent tool failure. An API returns a 200 with an error message in the body, or returns an empty list that the agent treats as "nothing to do".
• Optimistic summarising. The final message describes what the agent intended to do, not what happened. The prose is confident; the side effects are missing.
• Scope drift. The agent does something plausible but not what was asked, and reports success against its own reinterpretation.

None of these look like crashes. That's what makes them dangerous: your monitoring is calm, your users are not.

Verify the outcome, not the transcript

The single most useful shift is to stop grading the agent's narrative and start checking the world it was supposed to change. Ask: what observable state should be different if this task genuinely succeeded? Then check that state directly.

Concretely:

• Define a success condition per task, in data. "Invoice record exists, matches the source document on amount and vendor, and is marked reconciled" is checkable. "Agent reports invoice processed" is not.
• Read back from the system of record. If the agent was supposed to create a ticket, query the ticketing system for that ticket. Don't trust the tool call's return value alone.
• Check counts and invariants. Row counts, totals, required fields, no duplicates. Cheap checks catch a surprising share of failures.
• Treat empty and partial results as suspicious. A zero-result response is the most misread signal in agent systems. Decide in advance whether it means "nothing to do" or "something went wrong", and make the agent say which.

This is the core of any serious AI agent verification approach: the agent's own claim of success is an input to your check, never the check itself.

Build a small evaluation set before you scale

You cannot verify what you haven't defined. Before an agent touches production volume, assemble a modest set of real tasks with known correct outcomes — twenty to fifty is often enough to start. Include the boring cases, the ambiguous ones, and the ones that failed during development.

Then measure two things separately:

1. Task success rate — did the outcome match the definition?
2. Claim accuracy — when the agent said it was done, was it right?

That second number is the one most teams never track, and it's the one that tells you how much you can trust the agent's own reporting. An agent that succeeds 80% of the time but accurately reports its own status is far easier to operate than one that succeeds 90% of the time and confidently lies about the other 10%.

Put guardrails where the cost of a wrong "done" is high

Not every task deserves the same scrutiny. Sort your agent's actions by reversibility and blast radius.

• Low stakes, reversible (drafting an email, summarising a document): let the agent run, spot-check samples.
• Medium stakes (updating a record, sending a notification): require a verification step before the action is considered final, and log the evidence.
• High stakes (payments, deletions, external commitments): human approval, always, with the agent's proposed action and its supporting evidence shown side by side.

For agent evaluation in an enterprise setting, this tiering matters more than raw model quality. A mediocre model behind good checks is safer than an excellent model with none.

Make the agent show its work

Ask the agent to return structured evidence alongside its answer: which tools it called, what each returned, and which specific result supports the claim of completion. This is not about trusting the reasoning — it's about making the claim falsifiable. A completion report that says "created record 4471, verified by reading it back" can be checked in seconds. "Task completed successfully" cannot.

Log these reports. When something goes wrong — and it will — the difference between a ten-minute diagnosis and a two-day one is whether you kept the evidence.

The honest caveat

Verification costs effort, and it never reaches zero. Agents operating in open-ended environments will occasionally surprise you no matter how many checks you build. The goal isn't a guarantee; it's knowing, at any moment, how much you can trust a green status. Teams that internalise this ship agents that quietly work. Teams that don't ship demos that impress once and erode trust afterwards.

If you're building or reviewing an agent and want a second pair of eyes on how you verify its results — or you're not sure your current "completed" status means anything — I'm happy to talk it through. No pitch, just a look at where the gaps are. You can reach me at contact@hamzabelgacem.com.