Chaos engineering for an AI agent means breaking the agent's dependencies on purpose, in a controlled test, to see whether the agent recovers the way you assumed it would. You pick a failure the agent will eventually meet in production, a tool that times out, an API that returns a rate-limit error, a search that comes back empty, then inject it into a small, bounded slice of traffic and watch. If the agent retries sensibly, falls back, or fails cleanly, the design holds. If it loops, hallucinates around the missing data, or stalls, you found the weak point in a rehearsal instead of during a real outage.
The discipline is borrowed from site reliability engineering, where it was popularized by Netflix and its Chaos Monkey tool, which randomly terminated production instances to force engineers to build systems that survive failure (see the Principles of Chaos Engineering). Agents need the same treatment, and arguably more of it, because an agent does not just call a dependency once. It reasons over the result and decides what to do next. A bad or missing result does not just throw an error; it can quietly steer the whole task off course.
What chaos engineering means for agents
Classic chaos engineering follows a simple shape: form a hypothesis about how the system behaves in a healthy state, introduce a real-world fault, and check whether that steady state holds. The "steady state" is whatever you measure to know the system is working: success rate, latency, task completion. The experiment is valuable precisely when you are confident the system will cope, because a surprise then is a genuine, previously hidden flaw.
For an agent, the steady state is usually task-level: the share of runs that finish with a correct, complete result inside an acceptable time. The faults you inject sit one layer down, at the boundary between the agent and the tools it calls. You are not corrupting the model. You are corrupting the world the model sees, so you can observe how its reasoning and your recovery code respond together.
That last point is what makes agent chaos engineering distinct. In a traditional service, a failed dependency call returns an error and your code handles it. In an agent, a failed call returns an error to a reasoning loop that may interpret it, work around it, or ignore it. Testing the recovery code alone is not enough. You have to test the agent's behavior on top of the recovery code, and the only reliable way to do that is to make the failure happen and look.
Why agents need it more than ordinary services
Three properties of agents raise the stakes on resilience, and each one is a reason to rehearse failure rather than hope.
Agents act on partial information. When a tool returns nothing or returns garbage, a well-behaved service stops. An agent often keeps going, because its job is to make progress with whatever it has. An empty search result can become a confident answer built on no evidence. Injecting empty and malformed responses tells you whether the agent notices the gap or papers over it. This is closely tied to guardrails and safety, which decide what the agent is allowed to do when it is unsure.
Agents loop. A single failed step can become a retry that re-fails, a re-plan that re-plans, or two tools that keep handing a bad result back and forth. A fault that a stateless service would shrug off can turn into a runaway loop in an agent. Chaos experiments surface these loops in a controlled window, where the rate limits you set act as a safety net while you watch the loop form.
Agents chain dependencies. A real task touches a model, one or more tools, and often a datastore. The failure that hurts is rarely the one you planned for; it is the interaction, a slow tool that pushes the whole run past its timeout, or a partial failure where one of five parallel calls comes back wrong. You cannot reason your way to every interaction. You have to inject them. The broader practice this sits inside is reliability testing, with chaos engineering as the part that targets failure rather than correctness.
The faults worth injecting
You do not need an exotic toolkit. The faults that find real bugs in agents are mundane, and they map directly to the recovery paths every agent should have. Start with these five.
- Tool timeout. Make a tool hang past its deadline. This tests whether the agent has a timeout at all, whether it retries, and whether it can proceed without that tool's result. A missing timeout is the single most common resilience bug.
- Rate-limit error. Return a 429 from a tool. This tests backoff and retry. The agent should wait and retry with increasing delay, not fail the task or hammer the endpoint. See handling agent rate limits for the recovery side of this fault.
- Malformed or empty output. Return broken JSON, an empty list, or a truncated string. This tests output validation. The agent should detect that the result is unusable rather than reason over nonsense.
- Dependency fully down. Make a tool return connection errors every time. This tests graceful degradation and error handling and rollback: can the task complete with reduced scope, or does it fail in a way that leaves no half-finished mess behind?
- Slow but successful. Return correct output, but slowly, just under the timeout. This is the sneaky one. It tests whether latency in one dependency cascades into timeouts elsewhere, and whether the agent's overall budget accounts for slow paths.
The pattern to notice is that each fault has a designed-for response. The experiment is checking whether that response actually fires, because the gap between the recovery code you wrote and the recovery behavior you get is exactly where production incidents live.
Run an experiment in five steps
A chaos experiment is a small, disciplined ritual, not a free-for-all. Follow the same five steps every time and the results stay comparable and safe.
1. State the steady state. Write down the metric that means the agent is healthy and the number that counts as normal. For example: at least 98 percent of runs complete with a valid result inside thirty seconds. You cannot tell whether a fault hurt without a baseline to compare against.
2. Form a hypothesis. Predict what happens when you inject the fault. "When the search tool times out, the agent retries twice, then completes the task using its cached context, and the success rate stays above 95 percent." A written prediction turns the experiment into a real test you can pass or fail.
3. Bound the blast radius. Decide how much traffic the experiment touches and set an abort condition before you start. A small slice and a hard stop are what make the experiment safe to run, covered in more detail below.
4. Inject and measure. Turn on the fault for the chosen slice and watch your steady-state metric against the baseline. Capture the agent's traces, not just the top-line number, so you can see how the reasoning loop reacted step by step. Good monitoring and observability is what makes this step legible rather than guesswork.
5. Fix and re-run. If the hypothesis held, you have evidence the agent is resilient to that fault. If it did not, you found a bug: a missing timeout, an absent fallback, a validation gap. Fix it, then run the same experiment again to confirm the fix holds. Resilience you cannot reproduce is resilience you cannot trust.
Chaos vs. load vs. reliability testing
These three practices get conflated, and keeping them apart helps you choose the right test for the question you actually have.
Load testing answers a capacity question: does the agent stay fast and correct as concurrent volume climbs? It keeps the system healthy and turns up the pressure. Its output is a number, the point where latency or error rate starts to bend.
Chaos engineering answers a resilience question: does the agent stay correct when a dependency misbehaves? It keeps volume normal and breaks something. Its output is a list of weak points where the agent does not recover the way you assumed.
Reliability testing is the umbrella over both, plus the correctness checks that confirm the agent does the right thing on a healthy path. A complete picture uses all three: reliability testing for "does it work," load testing for "does it work under pressure," and chaos engineering for "does it work when things break." Skipping the last one is how teams ship agents that pass every test and still fall over the first time a tool has a bad afternoon.
Blast radius and abort conditions
The difference between a chaos experiment and an outage you caused yourself is two controls: a small blast radius and an automatic abort. Neither is optional.
Blast radius is how much of the system the experiment can affect. In staging it can be everything, because nothing real depends on it. Against production it must be tiny: a single percent of traffic, one non-critical workflow, a test account. The goal is to make any surprise small enough that it is a data point, not an incident. Start narrow and widen only after the agent has earned your confidence on the narrow version.
The abort condition is the metric that means stop, plus the automation that acts on it without waiting for a human. If task success drops below your floor, or error rate spikes, or spend crosses a line, the experiment halts itself and removes the injected fault. A chaos experiment that can only be stopped by a person watching a dashboard is a chaos experiment that will, eventually, run long after the person looked away. This is the same logic as the kill switch in agent safety and guardrails: the smooth control handles the normal case, and the hard stop handles the moment something is clearly wrong.
One more habit makes the practice sustainable: run experiments as scheduled game days rather than surprise attacks. Gather the people who own the agent, announce the window, inject faults together, and watch. The shared context turns a passing experiment into team knowledge and a failing one into a fix everyone understands. For the cases a game day surfaces, the playbook in agent incident response picks up where the experiment ends.
How Gravity handles fault tolerance
Gravity is an AI agent platform, and fault tolerance is part of how the platform runs agents rather than something each user wires up. The agents are expert-built, and the failure paths that chaos engineering is designed to expose, tool timeouts, rate-limit errors, malformed responses, dependency outages, are handled inside the runtime: retries with backoff, fallbacks, and clean failure when a task genuinely cannot complete.
Because you pay per use, with $1 equal to 1,000 credits and billing only when an agent runs, the runaway-loop failure mode that chaos experiments hunt for is bounded by the same metering that bounds normal cost. A loop cannot quietly accumulate an open-ended bill, and the platform's pacing keeps a misbehaving dependency from turning into a storm of retries against itself.
The practical result is that you describe what you need in plain words and an expert-built agent runs it in about 60 seconds, while the resilience machinery, the injected-fault recovery paths a team would otherwise have to rehearse, lives in the platform. If you want the conceptual grounding before relying on it, what is an AI agent sets the foundation and the glossary defines the terms used here.
FAQ
What is chaos engineering for an AI agent?
Chaos engineering for an AI agent is the practice of deliberately injecting failures into the tools and dependencies the agent relies on, then watching how the agent reacts, so you find weak points in a controlled test instead of in production. You start from a hypothesis about how the agent should behave when, say, a tool times out, inject that fault into a bounded slice of traffic, and check whether the agent recovers, fails cleanly, or breaks in a way you did not expect.
How is chaos engineering different from load testing an agent?
Load testing asks whether the agent stays fast and correct as volume rises; chaos engineering asks whether the agent stays correct when something breaks. Load testing pushes more requests through a healthy system. Chaos engineering keeps volume normal but makes a dependency fail, slow down, or return garbage, and observes the agent's response. You want both: one finds capacity limits, the other finds resilience gaps.
What faults are worth injecting into an agent?
The highest-value faults map to the things that actually break agents: a tool call that times out, a tool that returns a 429 rate-limit error, a tool that returns malformed or empty output, a dependency that is fully down, and a slow response that sits just under the timeout. Each tests a different recovery path: retry and backoff, fallback selection, output validation, graceful degradation, and timeout handling.
Is it safe to run chaos experiments in production?
It can be, but only with a small blast radius and an automatic abort. Limit the experiment to a tiny fraction of traffic, define in advance the metric that means stop, and wire an automatic halt that fires the instant that metric crosses the line. Most teams begin in staging, prove the agent recovers, and only then run carefully scoped experiments against a slice of production where the cost of a surprise is bounded.
How often should you run agent chaos experiments?
Run a focused experiment whenever you add a new tool or dependency, change the agent's recovery logic, or after an incident to confirm the fix holds. Beyond that, a recurring game day, a scheduled session where the team injects faults and watches together, keeps resilience from rotting as the system changes. The cadence matters less than the trigger: every new failure mode the agent could meet deserves at least one rehearsal.