How to Test an AI Agent When the Production System Is Out of Reach

The hardest AI agent projects are rarely the ones with a clean sandbox. They are the ones where the agent has to work on a real ERP, a legacy back-office, an internal tool nobody documented, or a third-party platform you have no right to clone. The team can read the API docs and watch a few screen recordings, but it cannot reproduce the production system. Testing then becomes the weak point of the project — and the place where go-live risk quietly accumulates.

Here is a method that works in that situation, and the questions worth asking any provider who claims an agent is ready.

Accept that you cannot test everything, then choose what you must

When the target environment is out of reach, exhaustive testing is a fantasy. What you need is a deliberate choice of what to validate, ranked by cost of failure. List the agent's actions and sort them into three groups:

• Irreversible actions: sending an email to a customer, creating an invoice, deleting a record, changing a price, triggering a payment. These deserve the most scrutiny, even if they are rare.
• Reversible but visible actions: creating a draft, updating a status, assigning a ticket. Mistakes are annoying but recoverable.
• Read-only actions: searching, summarising, classifying. Cheap to get wrong, easy to verify.

Spend your testing budget from the top of that list down. A common failure mode is spending weeks polishing the quality of a summary while nobody checked what happens when the agent misreads an order number and confirms it.

Build a faithful stand-in, not a perfect copy

You do not need the real system. You need a simulation that is honest about the parts the agent depends on. Three layers usually suffice:

• A contract-level fake: a small service that implements the same API signatures as the real system, with the same authentication, the same error codes and the same quirks. If the real API returns a 409 on duplicate creation, the fake must too.
• A data replica: anonymised or synthetic records that match the real volume and messiness — missing fields, duplicate customers, inconsistent date formats. Clean test data flatters every agent.
• A record-and-replay layer: if the system is only reachable through a browser, capture real sessions once and replay them. This gives you realistic inputs without needing permanent access.

The point is not realism for its own sake. It is to expose the agent to the failure modes the real system will produce, especially the ones nobody documented.

Test behaviour, not just answers

An agent is a loop of reasoning and action. Most weak evaluations only check the final message. You want to inspect the trajectory:

• Did it call the right tools, in the right order, with the right arguments?
• Did it stop when it should have, or keep retrying after a clear error?
• When a tool returned an ambiguous result, did it ask for clarification or invent an answer?
• Did it respect the boundaries you set — no writes outside the allowed scope, no fabricated identifiers?

A practical technique is to keep a log of every tool call and score each run against a short rubric. Over twenty runs you will see patterns: the agent that always retries three times on a permission error, or the one that silently drops a field when the payload is large.

Design the failure tests on purpose

Happy paths are easy to write and easy to pass. The value is in the adversarial cases:

• Empty and malformed responses from the target system.
• Timeouts and partial failures, where the write succeeded but the confirmation never arrived.
• Ambiguous user instructions ("handle the Martin account" when there are three).
• Out-of-scope requests the agent should refuse or escalate.
• Prompt injection hidden in data the agent reads, such as a note field containing instructions.

For each, decide in advance what correct behaviour looks like: refuse, ask, retry once, or escalate to a human. Then check whether the agent does it consistently, not just once.

Plan the go-live as a controlled experiment

Since you cannot fully test in production, do not fully deploy either. Use a staged rollout with real guardrails:

• Start in read-only or suggestion mode, where the agent proposes actions a human approves. Measure how often the human corrects it.
• Enable writes only for a narrow, reversible scope first.
• Keep a full audit trail of inputs, tool calls and outputs, so any incident can be reconstructed.
• Define a rollback path and a named person who can trigger it.

This is not a compromise. It is the standard way to introduce any automation into a system you do not control.

Questions to ask your provider

If you are evaluating a proposal — including from a freelance consultant — these questions separate serious work from optimism:

• What exactly will be simulated, and what will only be validated in production?
• Which actions are irreversible, and how are they gated?
• What does the evaluation set look like, and who wrote the expected outcomes?
• How are tool calls logged, and can I inspect them?
• What is the rollback procedure, and who owns it?

A provider who cannot answer these clearly is probably planning to test on your production system and call it a pilot.

Let's talk about your case

If your agent has to operate on a system you cannot reproduce, the testing approach has to be designed early — it shapes the architecture, the permissions and the rollout plan. I am happy to look at your setup and discuss what a realistic validation plan would look like. You can reach me at contact@hamzabelgacem.com.