LLM-as-Judge for Support Email: Where It Works and Where It Lies

An eval run came back at 4.6 out of 5 across 300 support replies. The same batch, scored by two experienced support leads, came back at 3.1.

The gap wasn't subtle and it wasn't random. It tracked almost perfectly with reply length. The judge was rewarding thorough-sounding answers that restated the customer's problem, added a paragraph of context nobody asked for, and closed with an apology. Humans read those same replies as padded and slightly evasive.

That's the honest starting point for LLM-as-judge. It is genuinely useful, it is the only way to score quality at any real volume, and it will lie to you in patterned, predictable ways unless you build against those patterns deliberately.

What the judge is actually doing

Strip away the framing and an LLM judge is a model reading a support reply and producing a score or a verdict against some rubric. Three shapes show up in practice, and they behave very differently.

• Pointwise scoring: one reply in, a number out. Easiest to build, least reliable. Absolute scores drift between runs and cluster tightly around whatever the model considers a safe middle.
• Pairwise comparison: two replies to the same ticket, judge picks the better one. Substantially more reliable, because relative judgement is an easier task than calibrated absolute judgement. Useful for A/B on prompt or model changes.
• Reference-based checking: reply plus a labelled expected outcome, judge decides whether the reply satisfies it. The most reliable of the three, and the reason a golden set built from real tickets is worth the labelling cost.

Most teams reach for pointwise first because it's the one that fits neatly into a dashboard. That's understandable and it's also the shape most likely to produce a number that looks stable while meaning nothing.

Where it works well

The judge is reliable when the question is narrow, checkable against supplied evidence, and phrased as a verdict rather than a rating.

Grounding checks against retrieved context

Give the judge the reply and the exact context the agent retrieved, then ask a specific question: does every factual claim in this reply appear in the supplied context? This works well, and it works because you've converted an open-ended quality judgement into a containment check. Agreement with human labellers on grounding sits comfortably above 0.8 in our experience, which is better than most human-to-human agreement on softer dimensions.

Policy compliance as binary rules

Did the reply promise a refund? Did it commit to a delivery date? Did it disclose the required regulatory language for an EU customer? These are yes/no questions with the policy text supplied alongside, and the judge handles them well.

What it handles badly is "was this reply compliant," asked without the policy attached and expecting the model to remember what your rules are.

Pairwise A/B on identical tickets

When you've changed a prompt and want to know whether replies got better, showing the judge both versions of the same ticket and asking which is preferable is the single most trustworthy setup available. You still have to control for position, which is covered below, but the underlying signal is real.

Triage of what a human should look at

Even a mediocre judge is a good filter. Running it across every reply and surfacing the bottom 5% for human review is a genuinely good use, and it doesn't require the scores to be well-calibrated in absolute terms. It just requires the ordering to be roughly right.

Where it lies

These biases are documented across the research literature and they show up in support email evaluation exactly as advertised. They are not edge cases.

Verbosity bias

Longer replies score higher, independent of whether the extra length adds anything. This is the one that produced the 4.6-versus-3.1 gap in the opening, and it's the most damaging bias in a support context specifically, because good support email is short. A reply that answers the question in two sentences and stops is often the best possible reply, and a naive judge will rank it below a five-paragraph version that says the same thing.

Position bias

In pairwise comparison, the option shown first wins more often than it should. The effect is large enough to flip conclusions on close calls. The fix is cheap and non-negotiable: run every comparison twice with positions swapped, and only count a result when both orderings agree. Disagreements get logged as ties, which is informative in itself.

Self-preference

Models tend to prefer text generated by themselves or by models in the same family. If your agent and your judge are the same model, your eval has a thumb on the scale. Use a different model for judging where you can, and at minimum note the dependency when you report the numbers.

Confidence over correctness

A reply that states a wrong fact firmly tends to score better than one that hedges toward a correct answer. This is the bias that should worry a support team most, because it inverts the thing you actually want. Hedging is often the correct behaviour when the agent isn't sure, and a judge that penalises it will train your prompts toward confident wrongness.

Score compression

Ask for a 1 to 5 rating and you will get 4s. Sometimes 3s. Almost never 1s or 5s. The usable range collapses to about two points, which means a change that moves quality meaningfully might move your average by 0.08 and vanish into noise. Binary rubric items don't have this problem, which is a large part of why they're the better design.

It cannot check facts it doesn't have

Obvious once stated, and still a common failure. If the reply says the order shipped on Tuesday and you haven't given the judge the order record, the judge has no way to know. It will usually mark the claim as fine, because it reads as plausible. Anything requiring a system value has to be checked in code against that system value, not delegated to a model that's guessing.

What this looks like on one ticket

A customer writes in asking why their subscription renewed at a higher price. The record shows a promotional rate that expired after twelve months, disclosed at signup.

Reply A: three sentences. States the promo rate ended on 14 June, gives the new monthly amount, links to the plan page, offers to downgrade if they'd prefer. Done.

Reply B: six paragraphs. Opens by apologising for the confusion, restates the customer's question back to them, explains the concept of promotional pricing in general terms, gives the same three facts Reply A gave, apologises again, and closes by thanking them for being a valued subscriber since 2024.

Both are factually correct. A support lead reads Reply A as good and Reply B as padded, and would prefer A on almost any queue. A pointwise judge scoring on a 1 to 5 quality scale gave B the higher mark in every run we tried, and the stated reason was consistently some version of thoroughness and empathy.

Swap the rubric to binary items and the gap closes immediately. Did it answer the question? Both yes. Did it state any fact not in context? Both no. Did it include content the customer didn't ask for? A no, B yes. Now the ordering matches the human one, and it matches for a reason you can point at.

Calibrating against human labels

A judge you haven't calibrated is a random number generator with good manners. Calibration is the step that converts it into something you can report to a leadership team without hedging.

The mechanics are straightforward. Take a stratified sample of 100 to 150 replies, have two humans label them independently against the same rubric the judge uses, adjudicate the disagreements, then compare the judge's verdicts to the adjudicated labels.

Report agreement as Cohen's kappa, not raw accuracy. Raw accuracy is inflated by class imbalance, and support evals are badly imbalanced. If 88% of replies pass policy compliance, a judge that says "pass" every single time scores 88% accurate and is completely worthless. Kappa corrects for agreement you'd get by chance.

Rough interpretation for this kind of work:

• Above 0.7: trust the dimension for tracking trends and gating releases.
• 0.4 to 0.7: useful for triage and ranking, not for reporting absolute numbers to anyone.
• Below 0.4: the dimension is broken. Either the rubric is ambiguous or the task is genuinely too subjective for a model.

When kappa comes back bad, the instinct is to blame the judge. Check the humans first. If your two labellers only agreed with each other 65% of the time before adjudication, the rubric is the problem and no judge is going to rescue it.

Recalibrate quarterly, and immediately after any judge model version change.

Designing a judge that fails less

Most of the improvement here comes from rubric design rather than prompt cleverness.

1. Decompose into binary items. Replace "rate response quality 1 to 5" with eight yes/no questions: did it answer the actual question, did it invent any facts, did it commit to anything it shouldn't, and so on. Aggregate afterwards if you need a headline number.
2. Supply the evidence. Retrieved context, relevant policy text, and the customer's prior thread. A judge working from memory is guessing.
3. Run separate judges per dimension. One call scoring grounding, another scoring policy, another scoring tone. Combined rubrics bleed, and a reply that's warm and wrong gets partial credit it hasn't earned.
4. Swap positions on every pairwise call. Count only the agreeing results.
5. Make the judge cite. Requiring it to quote the specific span of context supporting each factual claim cuts grounding errors noticeably and gives a human something to check when the verdict is disputed.
6. Control for length. Either strip length from the rubric explicitly or track reply length as a covariate so you can see when a score improvement is really just verbosity.

Item five is the one teams skip and shouldn't. Citation-forced judging is slower and more expensive per call, and it's the difference between a verdict you can defend and a verdict you have to take on faith.

Drift, and the version you forgot to pin

Here's a failure that's caught more than one team off guard. Quality scores improve 4% across the board over a weekend with no deployment on your side. Nobody shipped anything. The judge model was updated underneath the API.

Pin judge model versions explicitly. When a version has to change, re-run the previous 90 days of stored evals against the new judge before switching, so you know the size of the step change and can annotate your dashboards rather than reading a model swap as a product improvement.

Keep the raw replies, not just the scores. Without them you cannot re-baseline, and every historical number becomes uncomparable the moment anything moves.

What judging costs, and how much you need

Nobody writes about this part and it shapes the design more than the rubric does.

A decomposed rubric with separate judges per dimension and citations required means five to eight model calls per reply evaluated, each carrying the retrieved context in the prompt. At 40,000 emails a month, judging every single one on every dimension gets expensive fast, and the marginal information from reply 39,000 is close to zero.

A more sensible allocation:

• Grounding and policy compliance on 100% of replies. These are the dimensions where a single miss is a real incident, and they're the cheapest to check because the rubric items are binary.
• Tone and completeness on a 10 to 15% stratified sample, weighted toward high-value and previously-escalated tickets.
• Full decomposed rubric with citations on the golden set only, which runs on every deploy rather than on every ticket.

That mix keeps the per-ticket cost of evaluation in the low single-digit cents while still catching the failures that matter. Teams that try to judge everything on everything usually end up turning judging off entirely three months later when someone looks at the bill, which is the worst outcome available.

When to stop trusting it

Some things belong with humans and always will.

Judges are unreliable on tone in emotionally loaded threads, which is exactly where tone matters most. A bereavement cancellation, a complaint about a billing error that caused a bounced payment, a customer on their third escalation. Model graders read these as adequately polite and miss the thing a human catches instantly, which is that the reply is technically correct and emotionally tone-deaf.

They're also unreliable on anything with legal or regulatory exposure, on novel situations outside the rubric's scope, and on judgement calls where two reasonable people would disagree. If your humans can't agree, the judge's confident verdict is noise wearing a suit.

Routing rules built on judge output need the same scrutiny. Using a judge to decide which replies a human reviews is fine. Using one to decide whether a reply sends is a design we'd push back on for any category with real consequences, and the escalation thresholds discussion covers where those lines usually land.

How this runs at Robylon

Our evaluation stack scores action correctness deterministically, because whether a refund fired against order 41822 for $84.20 is an assertion rather than an opinion. Model-graded judging handles grounding, policy compliance, and tone floors, with citations required on every grounding verdict.

Judge output gets calibrated against human labels each quarter, and the kappa values are something we'll show a customer rather than summarise. That matters more than it sounds, because 60–80% autonomous resolution is only a meaningful claim if the measurement behind it holds up when someone pushes on it.

Continuous scoring runs on live traffic through the same rubric, which is how the offline picture and the production picture stay connected. More on that layer in our writeup on scoring every response for quality and compliance, and on the grounding side in preventing hallucinations in customer email.

The trade worth naming

LLM-as-judge buys you coverage at the cost of precision. A human QA process might review 2% of replies with high accuracy. A judge reviews 100% with meaningful error. Neither number is better in the abstract, and the useful configuration uses both: the judge for coverage and triage, humans for the flagged tail and for calibration.

The failure mode isn't using a judge. It's using one and forgetting it's an estimate.

Ready to see how your support replies score against a calibrated rubric? Robylon AI resolves 60–80% of customer emails autonomously with agents that take action across Zendesk, Stripe, Shopify, Salesforce and 60+ other integrations. Start free at robylon.ai