Throughput optimization for an AI agent is about one number: how much work it finishes per unit of time. When you have a queue of a thousand invoices to reconcile or ten thousand tickets to triage, what matters is not how fast one task runs but how many tasks per minute the system clears. The encouraging part is that most agents leave throughput on the table not because the work is hard, but because they do it one item at a time while spending most of every run simply waiting. Reclaiming that wasted waiting is where the gains live, and it can be done without touching the quality of any single result.
The trap is optimizing by feel. Teams reach for a faster model or a rewritten prompt and are surprised when total throughput barely moves, because the thing they sped up was never the constraint. Throughput work is bottleneck work: measure where time actually goes, fix the one stage that everything waits on, then measure again. This guide walks the techniques in the order you should apply them, starting with the distinction that makes the rest make sense.
Throughput is not latency
Latency is how long a single task takes; throughput is how many tasks finish per unit of time. They feel related but they are optimized differently, and confusing them sends effort in the wrong direction. An agent can be fast on each task and still have poor throughput, because it handles one task, then the next, then the next, leaving the system idle whenever a task is waiting on something external.
The reason the distinction matters is that the biggest throughput wins often do nothing for latency, and vice versa. Running a hundred tasks in parallel does not make any single task finish sooner, but it multiplies how many finish in a minute. So before optimizing, decide which you actually need. For an interactive request where a user is waiting, latency rules. For a batch of work to grind through, throughput rules, and that is the case this guide addresses. Both are facets of performance that show up in monitoring and observability, and keeping them separate in your head is the first optimization.
Find the bottleneck first
Every system has exactly one constraint at a time: the slowest stage that everything else waits on. Throughput is governed by that stage and nothing else, which is why the first move is always to find it, not to guess at it. Speeding up a stage that is not the bottleneck produces a faster non-bottleneck and the same total throughput, effort spent for no result.
For agents, the bottleneck is rarely local computation. It is almost always waiting: time spent on model calls, on tool and API responses, on database queries. To find it, measure where a run spends its time across stages and look for the longest cumulative wait, or for the dependency with the lowest rate ceiling. The discipline here is the same one behind load testing at scale: you deliberately push volume through the agent and watch which stage saturates first. That stage is your constraint, and it is the only thing worth optimizing until it stops being the constraint, at which point a new bottleneck appears and you repeat.
Concurrency: fill the waiting
Because agents spend most of their time waiting on external calls, the single highest-leverage technique is concurrency: running independent work in parallel so the waiting overlaps instead of stacking up. While one task is blocked on a model response, another advances; while that one waits on an API, a third runs. The idle time that a sequential agent wastes becomes useful time, and tasks-per-minute climbs.
The gain is real but bounded, and the bound is the shared dependency. Concurrency raises throughput right up until the parallel tasks saturate something they all depend on, a model endpoint, a database, an external API, after which adding more parallelism just builds a queue in front of that resource and stops helping. This is a practical reading of Little's Law from queueing theory: the work in progress a system can sustain is set by its throughput and the time each item spends inside it, so you cannot stuff in more concurrency than the slowest stage can drain. The art is finding the level of concurrency that fully uses the bottleneck without overwhelming it, and holding there. Push past it and you trade throughput for the retries and timeouts that come from hammering a saturated dependency.
Batching and caching
Two more techniques attack the work itself rather than the waiting around it. Both raise throughput by doing less total work, not by working faster.
- Batching groups many small operations into one. Where a downstream system accepts batched requests, sending one call for a hundred items instead of a hundred calls for one item amortizes the per-call overhead and eases pressure on the rate limit, since a batch typically counts as a single request. Batching turns a flood of tiny calls into a trickle of large ones.
- Caching avoids repeating work whose answer has not changed. If the agent looks up the same reference data, recomputes the same intermediate result, or answers the same sub-question across many tasks, caching that result means the expensive step runs once and the rest of the tasks read the stored answer.
Both compound with concurrency rather than competing with it: fewer and cheaper operations mean the bottleneck drains faster, which lets a given level of concurrency clear more tasks. They also lower cost as a side effect, since work you do not repeat is work you do not pay for, the same lever pulled in agent cost optimization. Throughput and cost efficiency usually move together when the gains come from removing waste.
Respecting downstream rate limits
Every throughput technique eventually runs into the same wall: the rate limits of the services the agent depends on. A model API or external tool that permits a fixed number of requests per minute is a hard ceiling, and no amount of concurrency lifts it. Worse, ignoring it backfires, bursting past a limit earns rejections, and the retries those rejections trigger consume capacity while delivering nothing, so throughput can actually fall the harder you push.
The right response is to treat the limit as the constraint to design around rather than an obstacle to beat. Pace requests so they stay under the ceiling, smoothing a burst into a steady stream the downstream service accepts, which is the core idea of agent rate limiting and the practical handling covered in handling agent rate limits. A well-paced agent running just under the limit clears far more work than one that bursts over it and spends half its capacity retrying. The ceiling is fixed; your job is to ride right up against it without bouncing off.
Speed without cutting corners
The one rule that governs all of this: raise throughput by removing waste, never by lowering the bar. Concurrency, batching, and caching are safe because none of them changes the result the agent produces. They fill idle time, amortize overhead, and skip repeated work, the output of any individual task is identical to what it would have been run alone.
Quality only suffers when throughput is bought the wrong way, by skipping a validation step, truncating the agent's reasoning, or dropping a check to shave time. That is not optimization; it is cutting corners wearing optimization's clothes, and it shows up later as errors the reliability metrics catch. Keep the validation, keep the reasoning the task needs, and find your speed in eliminating waiting and redundancy. Throughput earned by removing waste is durable; throughput earned by lowering standards is a debt that comes due as failed work.
How Gravity handles throughput
Gravity is an AI agent platform, and the throughput machinery described here, bottleneck analysis, safe concurrency, batching, caching, and pacing against downstream limits, is operated by the platform rather than tuned by each user. The agents are expert-built and run inside a runtime that parallelizes independent work and paces requests to stay inside dependency limits automatically.
For the user, that means you describe what you need in plain words and an expert-built agent returns the finished result in about 60 seconds, with the concurrency and pacing handled for you. You pay per use, $1 equals 1,000 credits, billed only when an agent runs, so efficiency gains show up as work delivered rather than capacity you provisioned and idled. To go deeper on the surrounding concepts, what is an AI agent sets the foundation and the glossary defines the terms used above.
FAQ
What is throughput for an AI agent?
Throughput is how much work an agent completes per unit of time, for example tasks finished per minute. It is different from latency, which is how long a single task takes. An agent can have low latency on each task and still have low throughput if it processes one task at a time, and the two are optimized with different techniques. Throughput is the right metric when you care about total volume rather than the speed of any single run.
How do you find an agent's throughput bottleneck?
Measure where time goes across a run and find the slowest stage that everything waits on. For most agents the bottleneck is not local compute but waiting: model calls, tool and API responses, and database queries. The stage with the longest cumulative wait, or the dependency with the lowest rate limit, is the constraint. Optimizing anything other than the actual bottleneck adds complexity without raising throughput, so always measure before you tune.
Does concurrency increase agent throughput?
Usually yes, because agents spend most of their time waiting on external calls, and running independent work in parallel fills that idle time. While one task waits on a model or API response, others make progress, so total tasks per minute rise. The limit is the slowest shared dependency: concurrency helps until it saturates a downstream service or hits a rate limit, at which point adding more parallel work just creates a queue.
How do rate limits affect agent throughput?
Downstream rate limits set a hard ceiling on throughput regardless of how much concurrency you add. If a model API allows a fixed number of requests per minute, the agent cannot exceed that no matter how many tasks run in parallel; pushing past it just triggers rejections and retries that waste capacity. The fix is to pace work to stay under the limit, smoothing requests rather than bursting, and to treat the limit as the real constraint to design around.
Can you increase throughput without hurting quality?
Yes, as long as the techniques remove waste rather than cut corners. Concurrency, batching, and caching raise throughput by using idle time and avoiding repeated work, none of which changes the result the agent produces. Quality only suffers when throughput is bought by skipping validation or shortening reasoning. The rule is to speed up by eliminating waiting and redundancy, not by lowering the bar on what counts as a finished task.