An AI agent succeeds when it finishes the task, gets it right, does it reliably, costs less per successful task than doing it by hand, and stays inside its safety rules. You can measure that with 12 KPIs in six groups: outcome (task success, cost per successful task, time saved), quality (pass rate, acceptance), reliability (consistency, latency, run errors), human load (escalation, containment), safety (violations per 1,000 runs) and adoption (repeat usage).

This is the owner's view: you have an AI agent in production and need to decide whether it earns its keep. One number on this dashboard, the quality score, needs its own method; that is the subject of my guide to AI agent quality scoring methods. For testing before launch, AI agent evaluation metrics is the counterpart.

Why measuring decides whether an agent survives

Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls (Gartner, June 2025). Cost, value, risk. You can put all three on a dashboard.

Deloitte surveyed 1,854 senior executives in Europe and the Middle East in August and September 2025. 85% had increased AI investment in the past 12 months, but just 10% said they currently see significant ROI from agentic AI. Among the companies Deloitte calls "AI ROI Leaders", most use different frameworks or timeframes for generative and agentic AI (Deloitte, 2025). The takeaway for an owner: decide up front what an agent must show, and by when, before the budget review decides for you.

Borrowed baselines won't help. The AI agent adoption survey index on this site grades 35 surveys; their "in production" figures run from 7% to 97%, and only 23% define what they mean by "agent". Your own baseline is the only number that counts.

The 12 AI agent KPIs at a glance

Here are all 12 with a formula, a starting target and an alert line.

KPIFormulaWhere I would startAlert when
Task success rateSuccessful runs / all runs90% single-step, 80% multi-step tool use5-point drop week over week
Cost per successful task(Tokens + tools + review time) / successful tasksUnder 20% of the manual costDaily spend over 150% of the 7-day average
Time saved per taskManual time minus human time with the agentPositive after reviewTime-to-verify exceeds manual time
Quality pass rateSampled outputs that pass / outputs sampled90%Under 85% on a week's sample (about 350 runs)
Acceptance rateOutputs used as-is or lightly edited / outputs delivered70%Reverts rise two weeks running
Consistency (pass^k)Test tasks passed on all k tries / test taskspass^3 of 80% on a fixed test setAny release lowers it
End-to-end latencyp50 and p95, request to finished resultp95 set by user tolerancep95 above 3x p50
Run error rateRuns failed by tool or API errors / all runsUnder 2%Above 5% in any hour with at least 100 runs
Escalation rateRuns handed to a person / all runsHigh in week one, then fallingFalls while errors rise
Containment rate (support)Conversations resolved without a person / all conversationsRead with 7-day recontact7-day recontact up 3 points on baseline
Safety violations per 1,000 runsThree separate counts per 1,000 runs: guardrail triggers, blocked actions, unapproved irreversible actionsZero unapproved irreversible actions (a gate); baseline for the other twoAny unapproved irreversible action, or a 2x jump in the other two after a release
Repeat usageUsers (or workflows) back within 7 days / all users50%Two weekly declines in a row
Targets are my starting defaults; replace them after a two-week baseline of real runs.

Outcome KPIs: task success, cost and time saved

1. Task success rate

Task success rate is successful runs divided by all runs. Define "successful" before you count: the run reached the end state the task asked for, and the result passed a check. A run that finishes cleanly with the wrong answer is a failure.

Where I would start: 90% for single-step work, 80% for multi-step tool use. Alert on a 5-point drop week over week, since one bad day is usually noise.

2. Cost per successful task

Divide total cost, including the minutes people spend reviewing or fixing, by the tasks the agent gets right. Google Cloud calls this "the metric that matters most here" and sums up the trap: "If an agent costs $0.10 per run but fails 50% of the time, your actual cost per successful outcome doubles."

An illustrative month: say one run uses 30,000 input tokens and 3,000 output tokens (agents re-send context on every step; paste a real run into the token counter for yours). Price it at the median new model from 11 major labs: $0.75 input and $2.55 output per million tokens (LLM API price history, 9 October 2026).

People's time, not tokens, is the biggest line here. I run Gravity, an AI agent platform in private alpha, and every run on it costs us model tokens, so I watch prices. On our price history, the cheapest price for GPT-4o-level capability fell 36% in 12 months, from $0.11 to about $0.07 per million tokens blended (1 October 2025 to 9 October 2026). Your reviewer's hourly rate did not. AI agent cost attribution splits the bill by team, and the AI agent ROI case studies roundup shows what payback has looked like in public examples.

3. Time saved per task

Time saved is manual time minus the minutes a person still spends with the agent: asking, checking, fixing. Google Cloud tracks it as time-to-value, and its time-to-verify test is my alert line: "if review takes longer than doing the task manually, friction outweighs value." Get the manual number first. The free can AI do this task checker estimates how much of a task an agent can do and what your own time on it costs a month, which is a fair baseline to beat.

Quality KPIs: is the output good enough to use?

4. Quality pass rate

This is the share of sampled outputs that clear your quality bar. The bar comes from your scoring method: code checks where the answer is checkable, a rubric or an LLM judge where it is not. Building that method is its own job, covered in how to score AI agent output quality.

Pick the sample at random. Hand-picked runs say more about your curiosity than about the agent.

5. Acceptance rate

Acceptance rate is the share of outputs people use without major edits. Google Cloud tracks output friction three ways: accepted as-is, accepted with edits, or reverted. Its rule of thumb: "If someone keeps 80% of the AI-generated draft, the agent succeeded." The undo or revert is the real signal, which it calls implicit rejection.

Read it next to time-to-verify: a draft accepted after a long check still cost someone the check.

Reliability KPIs: consistency, latency and run errors

6. Consistency (pass^k)

Agents are non-deterministic, so one good run proves little. Anthropic defines pass^k as all k trials succeeding and gives the arithmetic: "If your agent has a 75% per-trial success rate and you run 3 trials, the probability of passing all three is (0.75)^3 ≈ 42%."

The τ-bench paper, which introduced pass^k, found even state-of-the-art function-calling agents like gpt-4o succeeded on under 50% of tasks, with pass^8 under 25% in the retail domain. A pass^3 of 80%, my starting default, needs about 93% success per run.

7. End-to-end latency

Google Cloud says end-to-end latency replaces time to first token for agents. Track p50 (the median run) and p95 (the time 95% of runs beat) from request to finished result, and set the p95 target from what users tolerate: seconds for chat, minutes for background work. Alert when p95 passes 3x p50. Long tails often mean retries or loops; AI agent performance tuning covers the fixes.

8. Run error rate

This counts runs that failed on a tool or API error. Tag each failure by cause: your prompt or code, a tool, or the model provider. I keep a daily log of AI API status-page incidents. In the 90 days to 9 October 2026, the Claude API had 56 incidents (22 major or critical) and 94 hours with an open incident; the OpenAI API had 21 (4 major or critical) and 226 hours.

Those hours are not downtime, but they explain error spikes you did not cause. AI agent failure modes helps with the tags; AI agent monitoring and observability covers the logging.

Escalation and containment: how much work still lands on people?

9. Escalation rate

Count the runs handed to a person and divide by all runs: that is escalation rate. Expect it high in week one, then falling. An agent that never escalates and still makes mistakes is overconfident, so alert when escalation drops while the error rate rises. Adding a human in the loop shows where the handoff goes.

10. Containment rate in support

Containment is the share of support conversations resolved without a person, and the definition decides the number. Intercom counts a Fin resolution when, after Fin's last answer, the customer confirms it was satisfactory or exits without asking for more help (an "assumed" resolution); it bills $0.99 per resolution. Zendesk counts an automated resolution when the AI resolves the issue without a live agent: after 72 hours of inactivity for email and web forms, or 2 hours by default for messaging, with an LLM verifying the conversation.

An assumed resolution can include a customer who silently gave up, so pair containment with recontact within 7 days, or CSAT. Intercom says Fin's resolution rate averages 76% across 12,000+ customers (fin.ai); that figure is vendor-reported. Check that a vendor exports conversation-level data and shows how it calculates time saved, then compare the best AI agents for customer support.

Safety and adoption: guardrails and repeat usage

11. Safety violations per 1,000 runs

Keep three counts apart, each per 1,000 runs:

I keep a log of public AI agent incidents. It holds 63, from 15 February 2024 to 6 October 2026: 41 caused real-world harm, 20 of the 63 (32%) involved prompt injection, and 9 were destructive actions. Only 37% have a public postmortem. The AI agent incidents database has the list; AI agent safety and guardrails covers the controls.

12. Repeat usage

Of everyone who ran the agent this week, how many came back within 7 days? That share, counted by people or by workflows, is repeat usage. It sits in Google Cloud's adoption and usage pillar, and it is the bluntest verdict you will get. People vote with their second run.

What SLOs and alert thresholds should an AI agent have?

An SLO is a target for a measured level of service over a time window. Google's SRE book calls insisting on 100% "both unrealistic and undesirable" and says "it is better to allow an error budget ... and track that on a daily or weekly basis." An error budget is the share of runs allowed to miss the target. Here is an example set for one agent, built from my starting defaults:

Agent: inbox triage (example)     Window: calendar week, Mon to Sun

SLO   success      >= 90% of runs end in a checked success
SLO   latency      p95 end-to-end <= 90 s (background work)
SLO   cost         cost per successful task <= 20% of manual cost
SLO   run errors   <= 2% of runs fail on tool or API errors
GATE  safety       0 irreversible actions without approval

Error budget for the success SLO: 10% of runs
  2,000 runs a week -> 200 failed runs allowed
  budget spent by Wednesday -> freeze prompt and model changes

The budget turns a percentage into runs you can count. When it runs out, fixes come before features. Budgets in runs work for success, latency and run errors; cost is a ratio, so check the cost SLO once a week instead.

When an agent runs thousands of tasks a day

Scale makes small rates big. At 5,000 runs a day, 98% success still means 100 failed runs a day, or 700 a week. Nobody reads 5,000 transcripts.

Where I would start: score a random sample of 50 runs a day with your quality method, tag the cause of every failed run automatically, and rerun the pass^3 test set before each release. Read the sampled number weekly. Illustrative binomial math: 50 samples at a 90% pass rate leave a margin of about 8 points either way at 95% confidence; a week's 350 shrink it to about 3.

What an AI agent dashboard should show: KPIs, safety, reliability and cost

Four panels, a few tiles each:

Measuring what latency, errors and confusion cost you

Uptime tells you little about behavior. To see what a slow run, an error or a clarifying loop does to people and revenue, compare cohorts:

  1. Flag every run as slow (above your p95), errored, or confused (the agent asked for clarification twice, or the person rephrased).
  2. Each week, split users or accounts into those who hit a flagged run and those who did not.
  3. Compare the next 7 days (repeat usage, abandoned runs, escalations) and the next 30 days per account (revenue, renewals or support tickets).
  4. Match the cohorts on usage volume first. Heavy users hit more errors because they run more.

Tracing tools such as Langfuse or LangSmith can record runs, tokens and cost. That is the raw data every flag needs.

How I would roll it out in the first 30 days

If I were setting this up for one agent, this is the order. Measure first, judge later.

  1. Week 1: log every run with outcome, tokens, cost, latency and failure cause. Time the task by hand. Set no targets yet.
  2. Week 2: keep logging, then build a fixed test set of 20 to 30 real tasks and run pass^3 on it.
  3. Week 3: set targets from the two-week baseline, write the SLOs, wire the alerts and start sampling 50 runs a day.
  4. Week 4: first review. Compare cost per successful task with the manual cost, run the cohort comparison, then decide: keep, fix or kill.

After day 30: alerts daily, a short review weekly, a keep-or-kill call monthly.

Frequently asked questions

What should an AI agent platform dashboard show about KPIs, safety, reliability, and cost?

It should show four panels. Outcome tracks task success, sampled quality and repeat usage; safety tracks guardrail triggers, blocked actions and unapproved irreversible actions; reliability tracks run errors, p95 latency and error budget; cost tracks cost per successful task, including review time.

What is a good task success rate for an AI agent?

I don't know of a universal benchmark, so set yours from two weeks of real runs. My starting defaults are 90% for single-step work and 80% for multi-step tool use.

How do you calculate cost per successful task?

Add token costs, tool fees and the time people spend fixing and checking, then divide by the tasks the agent completed correctly. In an illustrative month, $30.15 of model cost, $400 of fix time and about $57 of spot-checks across 850 successful tasks come to about $0.57 each.

What is the difference between containment rate and escalation rate?

Escalation rate is the share of runs an agent hands to a person. Containment rate is the share of support conversations resolved without one, and it can include customers who gave up, so read it next to 7-day recontact.

What SLOs should an AI agent have?

Start with four SLOs and one gate: task success, p95 latency, cost per successful task and run error rate, plus zero irreversible actions without approval. Give the success, latency and error SLOs an error budget counted in runs per week, and check the cost SLO weekly.

How are AI agent success metrics different from scoring output quality?

Success metrics tell an owner whether a production agent is worth keeping. Quality scoring is the method behind one of those metrics, using code checks, rubrics, LLM judges and human review.

Sources