Regression Testing an Email Agent: CI for Customer Conversations

A prompt change ships on Tuesday to fix one specific problem: the agent keeps quoting a 30-day return window that finance cut to 14. The fix works. By Friday, refund emails that used to resolve cleanly are escalating at three times the old rate, and nobody connects the two events for another nine days.

That's a regression. It is the most ordinary failure mode in production AI, and most support teams have no mechanism to catch it. They have dashboards, which tell you something is wrong after customers have already been affected. What they don't have is the thing every other software team takes for granted: a suite that runs before the change ships and refuses to let it through.

Why email agents are unusually bad at staying fixed

Ordinary software has a compiler and a type system. Break an interface and something yells at you in ten seconds. AI agents have neither.

Three properties make email agents especially prone to silent regressions.

The prompt is global state. A system prompt is one shared blob of instructions serving every intent in your queue. Add a line telling the agent to be more concise on shipping questions and you have also, invisibly, made it terser on billing disputes and complaint escalations where length was doing real work. There is no module boundary. Every edit is a global edit.

Non-determinism is the second problem. The same email, sent twice, produces two different replies. Not wildly different, usually, but different enough that "it worked when I tried it" carries almost no evidence. Manual spot-checking gives you a sample size of one against a system with meaningful variance.

And email hides its failures. A chat agent that goes wrong gets corrected in the next message, thirty seconds later. An email agent sends a wrong answer into someone's inbox, and the feedback arrives days later as a reopen, an angry reply, or a quiet churn nobody attributes to anything. By the time the CSAT dip is visible on a dashboard, the bad version has been live for two weeks.

Put those together and you get the pattern we see constantly: teams fix the same class of bug three or four times over a year, because nothing in the system remembers that it was ever fixed.

What a test case actually is here

The first thing to get straight is that "does the agent work" is not a test. A regression test case for an email agent has four parts, and all four have to be pinned down or the test is worthless.

• The input email. Verbatim from a real ticket where possible, with PII replaced by consistent tokens so the thread still hangs together. Include the subject line, the quoted history, the signature block, the whole mess. Cleaned-up inputs test a queue you don't have.
• The fixture state. What the connected systems return when the agent queries them. Order #4471 is in status shipped, the customer is on the Growth plan, their last payment failed on the 3rd. Without frozen fixtures the same test passes on Monday and fails on Thursday because someone's sandbox data drifted.
• The assertions. What must be true about the output. This is the hard part and it gets its own section below.
• The metadata. Which intent, which incident it came from, when it was added, who owns it. Suites rot fast without this, and a test nobody can explain is a test somebody eventually deletes.

A useful rule of thumb: if you can't write down what a wrong answer looks like for a given ticket, you don't understand the ticket well enough to test it yet.

Every production failure becomes a test case

This is the whole discipline in one sentence, and it is the part teams skip.

When a customer complains that the agent got something wrong, the standard response is to fix the knowledge base article or nudge the prompt, close the ticket, and move on. That fixes today. It does nothing about the version of the same bug that reappears in November after a model upgrade.

The alternative is a pipeline with a hard rule attached: no incident closes until its test case is merged. Concretely:

1. Capture the failure with its context. The original email, the agent's reply, the retrieved sources, the system responses at the time, the confidence score. If your agent doesn't emit that trace, fix that first. We wrote a companion piece on tracing and reason codes for email agents because you cannot build a test from an incident you cannot reconstruct.
2. Write the assertion the failure violated. Not "the reply should be good." Something falsifiable: the reply must not state a return window; the reply must call getOrder before quoting a delivery date; the reply must escalate when the thread contains a legal threat.
3. Confirm the test fails against current production. A regression test that passes before you've fixed anything is testing nothing. This step catches more bad tests than any review process.
4. Fix, verify the test now passes, merge both together.
5. Tag it by failure class. Hallucinated policy, missing action, wrong escalation, tone breach, multi-intent miss. After sixty or so cases the tag distribution tells you where the architecture is weak, which is more useful than any individual test.

A team running 4,000 emails a month generates maybe eight to fifteen genuine incidents a month once the obvious problems are gone. That's roughly 150 cases in a year, each one representing a real customer who was actually let down. It is the highest-signal test data you will ever have, and it costs nothing to collect except the discipline to write it down.

Assertions that survive paraphrase

Here is where most attempts at this die. Someone writes a test asserting the agent's reply equals a golden string, the model rewords one clause, and the suite goes red on 400 cases overnight. Everyone stops trusting it within a week.

The fix is to assert in layers, cheapest and most reliable first.

Deterministic assertions

These have nothing to do with the text. Did the agent call the refund API? With what amount, against which order? Did it escalate, and with which reason code? Did it query the OMS before quoting an ETA? Did it stay under three tool calls?

Anything the agent does rather than says can be asserted exactly, with zero flakiness. In our experience this layer catches around half of all real regressions, and it's the layer teams write last because it feels like it isn't testing the AI. It is. An agent that says the right thing while calling the wrong endpoint has failed.

Structured extraction assertions

Pull the facts out of the reply and test those, not the prose. Extract every number, date, currency amount, and policy reference the reply contains, then assert against the fixture: the tracking number matches, no refund figure appears that the fixture doesn't support, no date falls in the past.

This is the layer that catches specificity inflation, which is the most damaging hallucination class in email precisely because invented numbers look so much like real ones.

Semantic and negative assertions

Embedding similarity against a reference answer, with a floor rather than an exact match. More valuable in practice: negative assertions. The reply must not contain a return-window claim. Must not promise a callback. Must not name another customer. Negative assertions are cheap, stable, and map directly onto the failure classes that actually hurt.

Judge assertions, used sparingly

For tone, empathy, and completeness, a model judge is the only practical option. Just be honest about what you're buying. Judges drift with model versions, correlate imperfectly with human labels, and are the flakiest thing in your suite. Pin the judge's model version, keep a set of human-labelled cases to calibrate it against, and never let a judge alone block a deploy.

Roughly speaking: deterministic and extraction assertions should be gating. Semantic and judge assertions should be reporting.

Three suites, three cadences

One giant suite that takes 90 minutes will get skipped. Split by speed and purpose.

• Smoke suite — 60 to 120 cases covering the highest-volume intents and every past sev-1. Runs on every commit, finishes in under five minutes, gates the merge. This is the one that has to stay fast, which means guarding it jealously against people who want to add their case to it.
• Full regression — every case you've ever accumulated, 800 to 3,000 for a mature deployment. Runs nightly and before any deploy that touches prompts, model version, retrieval config, or tool definitions. Twenty to sixty minutes is fine here.
• Adversarial suite — prompt injection through the ticket body, instruction-override attempts, hallucination bait ("what's your policy on the lifetime warranty?" when no such warranty exists), and the sensitive-topic set. Runs pre-deploy and weekly. This one grows from security review rather than from incidents, and it deserves its own owner.

Keep a fourth thing that isn't a suite: a golden set of a few hundred labelled tickets used for measuring capability rather than catching regressions. Different purpose, different cadence, don't mix them. Regression suites answer "did this change break something." Golden sets answer "how good is it." Merging them produces a slow suite that tells you neither clearly.

What gates a deploy

A suite that reports numbers and never blocks anything is a dashboard, not CI. Three gates work well.

The first is the strong one: no previously passing case may newly fail. Not an aggregate threshold, a per-case rule. Aggregate pass rates hide exactly the failure you care about, because a change that fixes twelve cases and breaks four still moves the average up. Track the delta per case, and a regression stops the deploy regardless of what the headline number does.

The second is per-intent floors. Overall pass rate can sit at 94% while refund handling has quietly fallen to 71%. Set a floor for each intent that carries real money or real risk, and let the low-stakes intents float.

The third handles flakiness honestly. Run each gating case three times and require a consistent verdict, or run once and maintain an explicit quarantine list for cases with known variance. What doesn't work is pretending variance isn't there, then letting the team learn to re-run the pipeline until it goes green. That habit destroys the suite faster than having no suite at all.

Past the gate, the deploy still shouldn't go straight to everyone. Shadow-run the new version against live traffic without sending, diff its replies against the current version, and eyeball the cases where they disagree most. Then canary to 5% of volume for a day. The regression suite catches what you knew to look for; the shadow run catches what you didn't.

Fixtures, and the part everyone underestimates

The unglamorous truth is that most regression suites for email agents fail on infrastructure, not on assertion design.

Live sandbox integrations sound like the rigorous choice and turn out to be the flaky one. Stripe's test mode changes behaviour, someone's Shopify dev store gets reset, a carrier API rate-limits your nightly run at 2 a.m. Recorded fixtures — captured once from real calls, then replayed — are more boring and far more useful, with a periodic contract test against the live API to catch drift.

Freeze the clock, too. A test asserting "delivery in 3 business days" against a hard-coded date starts failing the moment the date passes, and half of a neglected suite's red is usually just this.

And run against a dedicated test tenant with its own knowledge base snapshot. If your suite reads from the production KB, every content edit becomes a potential test failure, and the people writing help articles have no idea they're breaking CI.

Where this stops helping

Regression suites are backward-looking by construction. They encode failures you've already had. They will not tell you about the failure mode you haven't met yet, which is why the adversarial suite and the shadow run matter.

They also can't catch drift in things nobody wrote an assertion for. Brand voice erodes gradually and no single test goes red. The only reliable detector we know of is a human reading twenty replies a week, in full, which is neither scalable nor optional.

Suites are gameable, as well. A team judged on pass rate will delete inconvenient tests or loosen assertions until everything is green. Review test deletions more carefully than test additions — the diff that removes a case deserves more scrutiny than the one that adds fifty.

And a passing suite says nothing about the accuracy ceiling set by your retrieval and your content. If the answer isn't in the knowledge base, no amount of testing will conjure it. That's a separate problem, covered in our guide to getting an email agent past 95% accuracy.

How Robylon runs this

Robylon treats the regression suite as deployment infrastructure rather than an optional QA layer. During the 3–7 day onboarding, historical tickets are used to build the initial case set — the same corpus used for backtesting against past resolutions seeds the first few hundred regression cases, so the suite exists before the agent sends its first live email.

After that, every escalation flagged as an agent error, every negatively-reopened ticket, and every human edit that changed a fact rather than a phrasing becomes a candidate case, tagged by failure class and reviewed weekly. Prompt changes, model upgrades, retrieval config changes, and new tool definitions all run the full suite with per-case delta gating before they reach a customer. The grounding and validation layers that keep hallucination rates under control are themselves covered by dedicated adversarial cases.

The practical outcome is the boring one worth having: the same bug doesn't ship twice. That's most of what keeps 60–80% autonomous resolution stable across model upgrades rather than something that degrades quietly between quarterly reviews. The email platform overview covers how the testing and deployment layers fit together.

Ready to stop shipping the same bug twice? Robylon AI resolves 60–80% of customer emails autonomously, with a regression suite built from your own ticket history and agents that take action across Zendesk, Shopify, Stripe, and 60+ other integrations. Explore the email platform at robylon.ai