An AI agent that has not been tested is an AI agent waiting to do something embarrassing or expensive on your behalf. Testing an agent looks different from testing software because the agent does not have a fixed branch list. The same input on Tuesday and on Friday can produce different outputs. The test plan below is the one used internally at Gravity and is the minimum every agent should pass before it runs unattended.

The five stages: dry run, edge-case battery, cost ceiling, rollback drill, supervised production. Each stage answers a specific question. Skipping any of them means deploying an agent whose worst-case behaviour is unknown.

Why agents need a different test plan

Traditional software has known branches. Given input X, you can map every code path the program takes. An agent operates over a model that picks tools and arguments at runtime. The same prompt, run twice, can land on different tool calls. That non-determinism is what makes agents useful and also what makes them risky. The test plan exists to bound that risk.

The OWASP guidance for LLM applications calls this out specifically as Excessive Agency: agents granted broad permissions tend to use them in ways the designer did not anticipate (OWASP, "Top 10 for LLM Applications"). Pre-deploy testing is the cheapest way to discover that excess before it costs money or trust.

If you are deciding whether the agent should exist at all, the prerequisite reading is what an AI agent can actually do and the reliability framing in how we test AI agents (80 tests).

Stage 1: Dry run

The dry run is the agent doing the full task with all destructive writes disabled. It reads inbox, sheet, calendar, or whatever the input source is. It calls every tool it would call in production. The only difference is that any write is intercepted: instead of sending an email, it composes the email and writes it to a draft. Instead of updating a database row, it logs the proposed update. Instead of posting to Slack, it prints the message to a test channel.

You inspect the output. You compare it to what you would have done. If it matches in 9 out of 10 cases, move on. If it matches in 6 out of 10, the prompt or the access scope needs work before any further test.

Dry runs are how you discover the prompt was ambiguous. The agent will produce confident output that is wrong in a way you only notice because you can read it before it ships. That preview is the entire point.

Stage 2: Edge-case battery

List the inputs the agent will see in a normal week. Then deliberately give it the worst versions:

The adversarial test is non-optional. Prompt injection through user content is a documented attack class (OWASP LLM01: Prompt Injection). If the agent reads any input that arrives from outside your team, this test must pass.

EmptyNothing-to-do output OversizedSample / batch AmbiguousFlag conflict MalformedSkip and log AdversarialRefuse instruction PassMove to Stage 3
Five input categories every agent must handle before it ships.

Stage 3: Cost ceiling

Set two ceilings: cost per run and cost per day. The per-run ceiling is what stops a single bad input from spending fifty dollars in tokens because the agent looped. The per-day ceiling is what stops a chained mistake from spending five hundred dollars overnight.

Test the ceiling by feeding the agent an input that should approach it. Verify the agent halts, returns a deterministic error, and does not retry. Most platforms expose this as a budget or rate limit. If your platform does not, that is not an agent platform; it is an LLM with no governance, and it is not safe to deploy unattended.

For the longer treatment of cost economics, see how to estimate agent cost before deploying and the economics of bootstrapped AI agents.

Stage 4: Rollback drill

Most teams skip the rollback drill and regret it. The drill is simple: in a sandbox, deliberately let the agent take a wrong action that has a real-world equivalent of needing to be undone. A wrong tag in your CRM. A wrong label on an email. A wrong line in a spreadsheet. Then practice reverting it.

You time three things: detection (how long until somebody noticed), revert (how long to fix), and residual (what side effects remain after revert). If detection is more than an hour or revert is more than ten minutes, the agent does not have enough observability or its actions are too coarse to safely undo. Either fix that or shrink the agent's blast radius.

The blast-radius lesson is non-theoretical. Among the failure-mode patterns covered in AI agent failure modes, the most expensive ones are agents that wrote to systems where the revert was harder than the action.

Stage 5: Supervised production

Supervised production is the agent running on real inputs while you watch every run. Ten supervised runs is the minimum. The bar: each run completes within budget, takes only the actions allowed, and produces output that you would have approved. If any run requires manual correction, reset the counter to zero and re-test.

The rule sounds harsh and is easy to skip. Skipping it produces an agent that works fine for a week and then fails on the eighth run on a new input class you did not anticipate. Ten clean supervised runs is the cheapest signal that the next thousand will be acceptable.

For the tracking discipline that supervised production requires, see how to monitor agent activity. Once supervised runs pass, the agent graduates to limited unattended operation.

Common mistakes

Frequently asked questions

How many test runs should an AI agent pass before going live?

Plan for at least ten supervised runs across realistic inputs before letting the agent run unattended. The first five surface the obvious mistakes; the next five surface drift, edge cases, and the gap between what you described and what the agent does. If any of those ten require manual correction, reset the counter.

What should be in an AI agent dry run?

A dry run is the agent executing the full task with all destructive actions disabled. The agent reads inputs, calls tools, and prepares outputs, but writes are routed to a draft, a log, or a sandbox. You inspect the dry-run output and only enable real writes once it matches what you would have done by hand.

How do I test an AI agent for edge cases?

List the inputs the agent will see in a real week, then deliberately feed it the messy versions: empty inputs, oversized inputs, ambiguous inputs, and inputs in the wrong format. The agent should either ask, skip, or fail safely. If it confidently produces output on garbage input, the prompt is too eager.

What is a rollback drill for an AI agent?

A rollback drill is a deliberate test where you let the agent take a wrong action and then practice undoing it. You measure how long it takes to detect the mistake, how long to revert, and what side effects remain. Run the drill before going live; an agent without a tested rollback path is an agent without a stop button.

Should I write unit tests for my AI agent?

Yes if the agent is built in code. Unit tests cover deterministic helpers (parsing, formatting, math). For the agent itself, treat each test as an input fixture plus an output rubric: did the agent do what was asked, within budget, without writes that were not allowed. Hosted platforms expose this as a test-runs view rather than a test file.

Three takeaways before you close this tab

Sources