Evals for Email Agents: Building a Golden Set From Real Tickets

A support team pushed a prompt change on a Thursday afternoon. By the following week, refund replies were quoting a 14-day return window that had been 30 days since March. Nobody caught it for nine days.

The change had been checked. Someone read six replies, they read fine, and it shipped. What was missing wasn't care, it was a fixed set of tickets with known-correct outcomes that every change has to clear before it reaches a customer.

That fixed set is a golden set. Building one is the highest-leverage engineering work in an email agent programme, and it's the thing most teams postpone until something breaks in front of a customer.

A golden set is not a test inbox

The confusion is worth clearing up early, because teams often think they already have one.

A golden set is a curated, versioned collection of real support emails, each paired with a labelled definition of what a correct handling looks like. Not a correct reply. A correct handling, which includes the actions taken, the facts asserted, and the decision to resolve or escalate.

Things that are not a golden set:

• A test inbox. Useful for checking that the pipeline runs. Tells you nothing about whether the answer was right.
• Synthetic tickets written by a model. They cluster around the clean, well-formed middle of your distribution and quietly omit the messy tail, which is where agents actually fail.
• Last week's production sample. It moves every time you look at it, so you can never tell whether a metric shift came from your change or from the sample.
• CSAT. It measures how the customer felt, on the small biased slice who bothered to respond, days later.

Think of the golden set as a fixed camera angle. You're not trying to capture everything happening in the room. You want the same view every time, so that when something moves you can actually see it move.

Sampling: where the tickets come from

Most teams get this step wrong by reaching for random sampling, which feels rigorous and isn't. Random sampling reproduces your production distribution, which means 40% of your golden set becomes order-status queries and your hardest intents get two examples each.

Stratify instead. Sample deliberately across the dimensions where behaviour changes.

Stratify by intent, then by customer state

Intent is the obvious axis: refunds, billing disputes, cancellations, technical issues, account access. The less obvious axis is the state of the customer and their record when the email arrives. A refund request from a first-time buyer inside the return window is a different problem from the same words sent by a customer on their fourth refund this quarter with a chargeback on file.

The second one is where policy lives, and policy is where agents fail.

Build a coverage matrix with intent on one axis and customer or order state on the other, then fill cells deliberately. Empty cells are your known blind spots, and knowing about them is worth more than pretending they don't exist.

Deliberately over-sample the tail

Your production distribution has a long tail of rare-but-expensive tickets. Regulatory requests, angry escalation threads, multi-issue emails where someone asks five things at once, forwarded chains with three participants, tickets where the customer is factually wrong about their own order.

These are maybe 6% of volume and something like half of the incidents that get a support lead paged. Over-sample them. A golden set that mirrors production is a golden set that will pass while your worst tickets get worse.

Slice across time

Pull from at least three separate periods, including one high-volume week. Ticket language shifts around launches, outages, and peak season, and a set built entirely from a quiet February will not warn you about anything that happens in November.

How many

Between 300 and 600 labelled tickets is a workable first golden set for a team handling a few thousand emails a month. Below roughly 200 the per-intent cells get too thin to give you a signal you can trust. Above 800 the labelling cost starts to outrun the value, and you're better off spending that effort on a second set for a specific problem area.

Start at 300. You'll grow it, because every production failure becomes a new case.

Labelling is the part that actually costs something

Here's the mistake that wastes the most time: labelling the ideal reply text.

There is no single correct reply to a support email. There are dozens of acceptable phrasings and a much smaller number of correct outcomes. If you label the text, you end up building an eval that punishes an agent for saying the right thing in different words, which is both wrong and demoralising for whoever has to read the failures.

Label the outcome. For each ticket, record:

• Required actions: the concrete operations that must happen. Issue refund of $84.20 against order 41822. Update shipping address. Cancel subscription at period end, not immediately.
• Forbidden actions: often more important. Do not issue a refund. Do not confirm a delivery date.
• Facts that must appear: the return window, the refund amount, the actual order status pulled from the source system.
• Facts that must not appear: anything not supported by retrieved context. This is where hallucinated details in email replies get caught.
• Escalation decision: resolve, escalate, or either-is-acceptable. That third option matters more than people expect.
• Tone constraints: usually a floor rather than a target. No cheerfulness on a bereavement-related cancellation.

A label like this takes a trained support person about four to seven minutes per ticket. For 400 tickets that's roughly 35 hours of labelling. It is the least glamorous line item in the project and the one I'd protect first if the timeline got squeezed.

Two labellers and an adjudicator

Have two people label an overlapping subset of 15 to 20% independently, then measure how often they agree. If your own humans agree on the correct handling only 70% of the time, no eval built on those labels can distinguish a good agent from a bad one above that ceiling.

Low agreement is usually a policy problem, not a labelling problem. It means the actual rule is ambiguous, which means your agents and your human agents are both guessing. Fixing the policy is the real work; the eval just surfaced it.

One labelled ticket, start to finish

Abstract guidance on labelling is easy to nod along to and hard to act on, so here's a single case worked through. The ticket is representative of the kind that separates a serious eval set from a decorative one.

The inbound email, lightly cleaned up:

Hi, I ordered the walnut side table on the 2nd and it turned up yesterday with a cracked leg. I don't want a replacement, I've already bought something else. Can you refund me? Also please cancel my design-club membership, I only signed up for the discount.

Two intents, two systems, and a policy interaction sitting underneath both. The order is inside the return window, but the item is damaged, which in this hypothetical retailer's policy routes to a no-return-required refund rather than the standard flow. The membership was bought at a discount tied to that order.

The label looks roughly like this:

• Required actions: issue full refund of the table against the original payment method; cancel the membership effective end of current billing period; create a damaged-goods record against the shipment.
• Forbidden actions: do not generate a return shipping label; do not refund the membership fee already charged; do not cancel the membership immediately.
• Facts that must appear: refund amount, refund timing window, membership end date.
• Facts that must not appear: any claim about when the replacement stock is available, since the customer declined a replacement and the agent has no reason to look it up.
• Escalation decision: either is acceptable. A damaged-goods refund above the auto-approval threshold can reasonably go to a human, and an agent that escalates here is not wrong.
• Tone constraints: no upsell, no retention offer on the membership cancellation.

Notice how much of the difficulty is in the forbidden list. An agent that refunds the table, cancels the membership, and writes a warm reply still fails this case if it also generated a return label, because the retailer now has a shipping cost against an item they told the customer to keep.

Notice too that the escalation decision is marked as either-acceptable. Teams that force a binary here manufacture failures. Roughly a fifth of real tickets have a legitimately defensible answer in both directions, and pretending otherwise makes your pass rate noisy without making your agent better.

What to measure when you run it

Text-similarity metrics are the wrong tool here. BLEU and ROUGE scores tell you how closely the reply matched a reference string, which we've already established isn't the thing you care about.

Score against the labelled outcome instead:

• Action correctness: did every required action fire, with correct parameters, and did no forbidden action fire? This is usually deterministic and checkable in code.
• Factual grounding: is every factual claim in the reply supported by retrieved context or a system lookup? Unsupported claims fail, even when they happen to be true.
• Escalation precision and recall: track both directions separately. Escalating too much is a cost problem. Escalating too little is a trust problem, and the two need different fixes. Our guide to resolve-versus-route decisions goes deeper on the thresholds.
• Policy compliance: binary pass/fail against the rules that applied to that ticket.
• Tone floor: did the reply violate a stated constraint? Not "was it warm enough."

Report these separately, always. A single composite score is a great way to ship a change that improved grounding by 3% while quietly dropping escalation recall by 11%.

The number that gets misread

Golden-set pass rate is not your production resolution rate, and the two will diverge because you deliberately over-sampled hard tickets. A set that runs at 71% while production sits at 78% autonomous resolution isn't a contradiction. It's the set doing its job.

Track them as separate series. Teams that conflate them either panic or get complacent, and both are avoidable.

Who does the scoring

Once the set exists, something has to decide pass or fail on 400 cases every time an engineer changes a prompt. Doing that by hand is fine for the first two runs and impossible by the tenth.

Split the dimensions by how checkable they are:

• Deterministic code handles action correctness cleanly. Did the refund call fire, against which order, for what amount? That's an assertion, not a judgement, and it should never go near a model.
• Deterministic code also covers most forbidden-action checks and any fact you can string-match against a system value.
• Model-graded scoring is for grounding, policy compliance, and tone floors, where the answer depends on reading the reply against context rather than comparing two values.
• Human review stays on a rotating sample, typically 20 to 40 cases per release, weighted toward whatever the automated layer flagged as borderline.

That last layer is not optional, and the reason is that model graders have their own failure modes that are invisible unless a person checks. They reward length, they drift when the underlying model version changes, and they'll agree with a confident tone over a correct answer more often than anyone finds comfortable. We go through those failure modes in detail in where LLM-as-judge evaluation breaks down, including how to calibrate a judge against human labels before you trust its numbers.

The practical split we've landed on is roughly 60% of dimensions scored in code, 30% model-graded, and the rest by a human who reads the flagged cases. Push more into code wherever you can. Code doesn't have opinions.

Golden sets rot

This is the part nobody plans for, and it's the reason plenty of eval efforts quietly die in month five.

Your labels encode the policy as it stood on the day someone wrote them. Then the return window changes. A product gets discontinued. Legal adds a required disclosure for EU customers. Every one of those makes some fraction of your golden set wrong, and a wrong golden set is worse than no golden set, because it fails good changes and blocks them from shipping.

Three habits keep it alive:

1. Version the set, not just the code. Golden set v3.1 pairs with policy doc v7. When policy changes, someone re-labels the affected cells and bumps the version.
2. Re-sample quarterly. Retire 10 to 15% of cases and replace them with recent tickets. Keeps the language current without destroying comparability.
3. Feed production failures back in. Every incident that reached a customer becomes a labelled case. This is how the set gets genuinely hard over time rather than just larger.

Budget about a day a quarter of a support lead's time for maintenance. If nobody owns it by name, it will rot.

What a golden set will never tell you

Worth being honest about the limits, because evals get oversold and then distrusted when they miss something.

A golden set is offline and single-turn by construction. It won't catch latency regressions, because it doesn't run against your live integrations under load. It won't catch a Zendesk API change that silently starts returning stale order status, since your fixtures are frozen. It won't catch multi-turn drift, where the agent handles turn one well and loses the thread by turn four. And it can't tell you anything about tickets in a category you never sampled.

Golden sets pair with three other things: a continuous QA scoring layer running on live traffic, integration tests against real sandboxes, and production monitoring on reopen rate and escalation mix. Any one of them alone leaves a gap.

Evals also can't settle judgement calls. When two labellers disagree because the policy is genuinely ambiguous, no amount of measurement resolves it. A person has to decide.

Getting the first version done

The trap is trying to build the perfect set before running anything. A rough 150-ticket set that runs on every deploy this month beats a beautiful 600-ticket set that lands in Q4.

A reasonable first two weeks:

1. Pull your top 12 intents by volume and pick 8 tickets each, weighted toward the ones that escalate most.
2. Label required actions, forbidden actions, and escalation decision only. Skip tone and grounding on the first pass.
3. Wire it into CI so it runs on every prompt, model, or knowledge-base change.
4. Set the bar as "no regression against last run" rather than an absolute pass rate you'd only be guessing at.
5. Add the harder label dimensions in month two, once the plumbing works.

Teams already training an agent on historical ticket data have a head start here, since the sampling and export work overlaps almost completely. Keep the two corpora strictly separate, though. A ticket used for training cannot be in your eval set, or you're measuring memorisation.

How this runs at Robylon

During onboarding we validate against a customer's own historical tickets before the agent touches live email, which is where the 60–80% autonomous resolution range gets grounded in something real rather than quoted from a brochure. The validation corpus becomes the first golden set, and it stays in place afterwards as the regression suite for every subsequent change.

Because our agents take actions across 60+ write-access integrations rather than only drafting text, action correctness is scored against the actual operations attempted, not the wording around them. A reply that sounds perfect while issuing a refund against the wrong order is a failure, and the eval treats it as one.

The eval harness ships with the product rather than being left as an exercise, mostly because we watched enough teams skip it.

The uncomfortable version of the argument

If a vendor can't show you their eval methodology, they don't have one. Ask what's in the golden set, who labelled it, when it was last re-sampled, and what the pass rate is on the hardest intent. The answers separate teams doing engineering from teams doing demos, and it takes about four minutes.

Which is a good filter to run before you sign anything.

Ready to see how an email agent performs against your own historical tickets? Robylon AI resolves 60–80% of customer emails autonomously with agents that take action across Zendesk, Shopify, Stripe, Salesforce and 60+ other integrations. Start free at robylon.ai