AI Agent Quality Scoring: 5 Methods, One Scorecard

Agent quality scoring turns "was that output good?" into a number you can track and gate a release on. It uses five methods: code checks, a rubric, an LLM judge, human review and a weighted composite scorecard with hard gates. Most teams end up needing all five.

This post covers how to produce the score. Tracking it next to cost and reliability is a separate job, covered in AI agent success metrics, where quality is one KPI among several. For correctness alone, see how to measure AI agent accuracy.

The five methods compared

Each method answers a different question at a different price. Build them in this order: the rubric, the code checks, two human reviewers to calibrate, then the judge and the scorecard. At run time the code checks go first on every output, because they are free and never argue.

Method 1: code checks for anything with a right answer

A code check is a deterministic grader: code that passes or fails an output with no model involved. Anthropic sorts agent graders into code-based, model-based and human, and its code-based examples include string match checks, binary tests, static analysis, outcome verification, tool-call verification and transcript analysis (Anthropic, January 2026). Anthropic rates them fast, cheap, objective and reproducible, but brittle to valid variations and lacking in nuance.

For an invoice agent that posts supplier bills, I would check five things in code:

• The output parses against your JSON schema.
• The total equals the source document's total, to the cent.
• Vendor, invoice number, due date and currency are present.
• The create-bill tool ran exactly once, with the vendor ID your records hold for that vendor.
• No email to the supplier contains a banned phrase.

My rule: if a question has a right answer, it gets a code check, not a judge. A judge comparing two totals is usually right. A subtraction is always right. Put these checks in your pre-deploy agent tests as well.

Method 2: a rubric with anchors for 1, 3 and 5

A rubric is the written definition of good: the dimensions, and what each score looks like, so a judge and two people land on the same number. Here is one for a support agent that answers refund and order emails.

Four habits keep a rubric usable:

• Four to six dimensions. Fewer blurs failures together; more, and nobody reads the anchors.
• Concrete anchors. "Wrong refund amount" is checkable. "Poor accuracy" is a mood.
• Yes/no checks inside a dimension where you can. Completeness for a three-question email is three yes/no questions.
• A 20-output pilot, by hand, before you automate.

Still choosing dimensions? The guide to AI agent evaluation metrics works as a menu.

Method 3: LLM-as-judge, and how far to trust it

An LLM judge is a second model that scores the agent's output against your rubric. It is the only method that rates open-ended quality at volume. Three modes:

• Pointwise: one output, one dimension, a score from 1 to 5.
• Pairwise: which of two outputs is better, the natural fit for A/B testing an agent.
• Reference-based: distance from a known good answer.

Evidence from five studies

Strong judges like GPT-4 reached over 80% agreement with human preferences, the level humans reach with each other, and showed position, verbosity and self-enhancement biases (Zheng et al., 2023). Wang et al. found rankings "can be easily hacked by simply altering their order of appearance": with ChatGPT judging, Vicuna-13B could beat ChatGPT on 66 of 80 tested queries (Wang et al., 2023).

Agreement can flatter a judge. Thakur et al. found high percent agreement can hide vastly different scores, up to 5 points from the human score, with even the best models well behind human-to-human agreement (Thakur et al., 2024). G-Eval reached a Spearman correlation of 0.514 with human judgments on summarization (Liu et al., 2023). Useful, and far from a human replacement.

On agent tasks, Peng et al. built RuVerBench: 2,458 deep research and agentic coding instances, each with a rubric and a human label. Even frontier judges perform strongly "but still exhibit substantial noise" (Peng et al., 2026). Weaker models are more sensitive to prompt wording, and majority voting helps with diminishing returns.

A judge prompt you can copy

One dimension, isolated, with a way out and a quoted reason. The quote shows you what the judge reacted to.

Grade ONE dimension of a customer support reply: correctness.
Ignore tone and format. Other graders score those.

Customer message: {{customer_message}}
Order record and refund policy: {{reference}}
Reply to grade: {{agent_reply}}

Score 1 to 5:
1 = states a fact that contradicts the record or the policy
3 = facts right, one detail vague or missing
5 = every amount, date and policy statement matches the record
Use 2 or 4 between anchors.

If the record lacks the facts to judge, set score to "Unknown".
Quote the sentence from the reply that decided your score.

Return JSON only:
{"score": 1-5 or "Unknown", "reason": "one sentence", "evidence": "exact quote"}

In promptfoo, the llm-rubric assertion returns a similar shape (a reason, a score from 0 to 1, a pass flag) and accepts an optional threshold such as 0.8 that the score must also reach (promptfoo docs).

Controls that keep a judge honest

• One judge per dimension, as Anthropic advises, so tone cannot leak into correctness.
• Both orders in pairwise. Randomize which answer comes first, score both orders and count a win only if it holds both ways.
• A different model family from the agent, where you can, against self-enhancement bias.
• An "Unknown" option. A forced guess looks exactly like a real score. An Unknown on a gated dimension goes to a person and never counts as a pass.
• Three votes on gated dimensions, majority wins. That is where I'd start.
• Transcripts, read by a person. Anthropic warns that you won't know if your graders work until you read transcripts and grades from many trials.

Cost per judged output

Illustrative math: one judge call reads 3,000 input tokens (task, output, one dimension) and writes 300. The median new model from 11 major labs lists at $0.75 input and $2.55 output per million tokens, per our LLM API price history.

input    3,000 x $0.75 / 1,000,000   = $0.00225
output     300 x $2.55 / 1,000,000   = $0.000765
one dimension                        = about $0.003
five dimensions, one output          = about $0.015
10,000 outputs a month               = about $150

with three votes on the two gated dimensions:
2 x 3 votes + 3 single calls = 9 calls = about $0.027 per output
10,000 outputs a month               = about $270

Check your own prompt in the token counter. I run Gravity, an AI agent platform in public beta, and every agent run on it costs us model tokens, so I watch these prices.

Method 4: human review, measured by agreement

Anthropic calls human graders the gold standard and says model-based grading "requires calibration with human graders for accuracy". That takes two reviewers on the same sample; one reviewer cannot show agreement.

For two raters, use Cohen's kappa. For three or more, use Krippendorff's alpha, which Hayes and Krippendorff proposed as the standard for any number of coders (Hayes and Krippendorff, 2007). My bar comes from McHugh: "any kappa below 0.60 indicates inadequate agreement among the raters", and 0.80 to 0.90 is strong (McHugh, 2012).

Why kappa? Percent agreement ignores guessing and "may overestimate the true agreement", McHugh notes. Illustrative: two reviewers each pass 90 of 100 replies and agree on 86.

kappa = (observed - chance) / (1 - chance)

observed agreement   86 / 100                 = 0.86
chance agreement     0.9 x 0.9 + 0.1 x 0.1    = 0.82
kappa                (0.86 - 0.82) / 0.18     = 0.22

86% agreement. Kappa of 0.22. When nearly everything passes, people agree by accident.

Calibrating humans, then the judge

Where I would start, each step gating the next:

1. Pull 100 real outputs at random. Hand-picked samples flatter the agent.
2. Two people score each one against the rubric, blind to each other.
3. Compute kappa per dimension, either on pass/fail at the gate line or as weighted kappa on the 1 to 5 scale, so a 4 against a 5 counts as a smaller split than a 1 against a 5. Where they split, fix the anchor and re-score until kappa is 0.6 or higher.
4. Settle the remaining splits into one consensus score per output.
5. Run the judge on the same 100. It scores alone only once judge-versus-consensus kappa is 0.6 or higher.
6. Each week, a person re-scores a fresh random 5% of judged outputs. I'd start at 5% and lower it once the judge holds steady. If judge-versus-audit kappa drops below 0.6, the judge stops scoring alone and you go back to steps 3 to 5.

Illustrative cost, at 2 minutes and $30 an hour, or $1.00 a review: the 100-output calibration with two reviewers is 200 reviews, about $200 once. The weekly 5% audit on 10,000 outputs a month is about 500 reviews, $500 a month. Spend people on calibration and audits, not on grading everything.

Method 5: the composite scorecard with hard gates

A composite score is one number that combines several dimension scores, each weighted by how much it matters. The catch is that an average can hide a serious miss. A gate fixes that by failing the output on one dimension, whatever the composite says.

weights    correctness 0.35, completeness 0.25, policy and safety 0.15,
           format 0.15, tone 0.10
scores     1 to 5 per dimension

weighted   = sum of (weight x score)        range 1 to 5
composite  = (weighted - 1) / 4 x 100       range 0 to 100

gates      any code check fails: FAIL
           policy and safety below 4: FAIL
           correctness below 3: FAIL
ship bar   composite 75 or more, and no gate tripped

The weights are mine for a support agent, where a wrong fact costs more than a stiff tone. Three illustrative replies:

B is complete, tidy and warm, and scores 80. Its 2 on safety means it drifted outside policy, say by implying a refund the customer cannot get. Without the gate, B ships. C fails the honest way: nothing alarming, just not good enough.

Should you score the path or only the answer?

Mostly the answer. Anthropic's advice: "It's often better to grade what the agent produced, not the path it took." My exception is any agent that takes actions. A refund agent can write a perfect summary of a refund it should never have issued.

For those, Google Cloud lists three trajectory checks a critic agent can run, and says plan adherence and argument hallucination rate "tend to surface issues fastest" (Google Cloud, February 2026):

• Tool selection accuracy: "Did the agent choose the right tool for the subtask?"
• Argument hallucination rate: "Did the agent invent parameters for a function call?" An invented order ID is a grounding failure you can catch in code.
• Plan adherence: the right tools in the right order.

I keep a log of public AI agent incidents, and it shows why. Of 63 incidents since February 2024 in our AI agent incidents database, 9 were destructive actions and 12 were wrong commitments or actions. In each of those, the problem was what the agent did, whatever its final message said. The patterns are in AI agent failure modes.

From score to a ship decision

Set the bar by what a bad output costs. These are my starting defaults.

The worked example above used 75 as a pilot bar; before that agent answers customers unsupervised, I would raise it to 80. Re-run the set on every release, model swap and prompt change, as the regression testing guide lays out. The step-by-step evaluation framework shows where the scorecard fits.

Then put the quality pass rate on a dashboard next to cost and reliability. AI agent success metrics covers that dashboard.

Frequently asked questions

What is the difference between agent quality scoring and accuracy?

Accuracy asks whether the answer is right. Agent quality scoring also rates completeness, policy and safety, format and tone, then combines them into one composite score. Accuracy becomes its correctness dimension.

Is LLM-as-judge reliable enough to score agent output?

Yes, once calibrated. Strong judges reached over 80% agreement with human preferences in Zheng et al. (2023), but they carry position, verbosity and self-enhancement biases. My default: the judge scores alone only once it matches human reviewers at a kappa of 0.6 or higher.

How many human reviewers do you need, and what agreement score is good enough?

Two reviewers on the same sample is the minimum. Use Cohen's kappa for two raters and Krippendorff's alpha for more. McHugh (2012) calls any kappa below 0.60 inadequate and 0.80 to 0.90 strong.

How do you combine quality scores into one composite score?

Multiply each 1 to 5 dimension score by its weight, with weights summing to 1, and add them up. Subtract 1, divide by 4 and multiply by 100 for a 0 to 100 score. Hard gates stop a high average from hiding a serious miss.

What score should an AI agent reach before it ships?

It depends on what a bad output costs. My starting defaults: 70 or more for internal drafts, 80 or more with safety and correctness gates for customer-facing replies, and 85 or more with every dimension gated at 4 for actions that move money or change data.

How much does LLM-as-judge scoring cost?

Very little. Illustratively, a judge call with 3,000 input and 300 output tokens costs about $0.003 at median list prices of $0.75 and $2.55 per million tokens. Five dimension judges come to about $0.015 per output, or $150 for 10,000 outputs. With three votes on the two gated dimensions it is about $0.027 per output, or about $270.

Sources

• Anthropic, "Demystifying evals for AI agents," 9 January 2026, anthropic.com/engineering/demystifying-evals-for-ai-agents
• Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," 2023, arxiv.org/abs/2306.05685
• Wang et al., "Large Language Models are not Fair Evaluators," 2023, arxiv.org/abs/2305.17926
• Thakur et al., "Judging the Judges," 2024, arxiv.org/abs/2406.12624
• Liu et al., "G-Eval," 2023, arxiv.org/abs/2303.16634
• Peng et al., "Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?" submitted 29 June 2026, revised 2 September 2026, arxiv.org/abs/2606.29920
• McHugh, Biochemia Medica, "Interrater reliability: the kappa statistic," 2012, pmc.ncbi.nlm.nih.gov/articles/PMC3900052/
• Hayes and Krippendorff, Communication Methods and Measures, "Answering the Call for a Standard Reliability Measure for Coding Data," 2007, tandfonline.com/doi/abs/10.1080/19312450709336664
• promptfoo, "llm-rubric," docs checked October 2026, promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/
• Google Cloud, "The KPIs that actually matter for production AI agents," 27 February 2026, cloud.google.com/transform/the-kpis-that-actually-matter-for-production-ai-agents
• Gravity, "LLM API Pricing History," 9 October 2026, gravity.fast/data/llm-api-price-history/
• Gravity, "AI Agent Incidents Database," updated 6 October 2026, gravity.fast/data/ai-agent-incidents-database/