Model Cards for Support AI: A Practical Template

The questionnaire line that stalls the deal

Somewhere around question forty of an enterprise security review, there's now a line that reads: provide model documentation or a model card for the AI system. What usually comes back is a two-page PDF with a logo on it, three accuracy percentages, and no description of what the system is allowed to do.

The reviewer sends it back. Two weeks disappear.

This is a strange gap, because the document isn't hard to write. It's mostly facts the engineering team already knows. It just requires deciding to write down the parts that aren't flattering, and that turns out to be the whole obstacle.

What a model card was originally for

The format comes from Model Cards for Model Reporting, published by Mitchell and colleagues in 2019. Nine sections: model details, intended use, factors, metrics, evaluation data, training data, quantitative analyses, ethical considerations, and caveats. The worked examples were a smiling-face detector and a toxicity classifier.

Those were single trained models with fixed inputs and a measurable output. Copy that structure onto a support AI agent and it fits badly, because a support agent is not a model.

You need a system card, not a model card

What actually sits in production is a stack. A base language model you probably didn't train and can't inspect. A retrieval layer over your knowledge base. A set of tools with write access to Zendesk or Shopify or your billing system. Routing logic, guardrails, an escalation path, and a human review programme wrapped around all of it.

The base model is the least interesting component and the one you control least. A buyer who's told "we use a frontier LLM with 94% accuracy" has learned nothing they can act on.

Here's what they're actually asking, underneath the questionnaire language: what can this thing do without a human, what data does it read to decide, and what happens when it's wrong? Answer those three and the document mostly writes itself.

Part one: what the system is

Open with identity and version. Name the system, give it a version number tied to a real release, date it, and name a human owner with an email address. Sounds trivial. It's the section that makes every later claim checkable, and an undated document is worthless six months out.

Then intended use, and more importantly out-of-scope use. Most vendors write the first half enthusiastically and skip the second, which is exactly backwards from the reader's perspective. State plainly which channels and intent categories the agent handles, which it explicitly doesn't, and which it must never handle. Something like: handles order status, returns, billing queries, and account access; does not handle legal complaints, chargeback disputes, or any thread flagged by legal hold.

Third, components and data flow. List the pieces and, critically, the sub-processors, because that's a question your buyer's privacy team will ask separately anyway:

• Base model provider, model family, and whether inference happens in a shared or dedicated environment
• Retrieval and embedding infrastructure, and where the vector store lives geographically
• Every downstream tool the agent can call, with read or write scope marked per tool
• Logging and observability systems that see message content

Enterprise reviewers are checking this against their own vendor register. Getting it right here saves a round trip later, and it overlaps heavily with the ground covered in an AI email support security review.

Part two: what it's allowed to do

This is the section that separates a real document from a brochure, and it's the one almost nobody writes.

Start with knowledge sources and grounding. Which corpora can the agent draw from, how fresh are they, who owns updates, and what does the system do when retrieval returns nothing relevant? That last question is the important one. An agent that answers anyway when it finds nothing is a different product from one that escalates, and the model card should say which you built.

Then actions and authority limits. For every tool the agent can call, write down the boundary in numbers rather than adjectives:

• Refunds: autonomous up to a stated currency amount; above that, pre-send human approval
• Account changes: address and contact updates autonomous, plan changes and cancellations gated
• Data operations: deletion and export requests always routed to a human, never autonomous
• Outbound commitments: whether the agent may promise a delivery date, a credit, or a callback, and on whose authority

A buyer reading "the agent can issue refunds" has to imagine the worst case. A buyer reading "refunds up to £50 autonomously, above that a named approver, hard cap of £500 per customer per quarter" has been handed a control they can evaluate. Specificity here shortens security reviews more than any other section.

Part three: how well it works and how it fails

Every vendor reports accuracy. Almost none report it in a form anyone can interpret, because the number arrives without the thing that gives it meaning: the evaluation set.

Say how many tickets, drawn from where, over what period, labelled by whom, and using what rubric. An agent scoring 91% on 200 tickets the vendor hand-picked is a different claim from 84% on 5,000 tickets sampled from the customer's own historical queue. The second number is lower and worth considerably more.

Report by intent rather than in aggregate, too. A blended figure hides the shape of the thing. Order status at 96% and billing disputes at 62% average out to something that looks fine and tells the reader nothing about where to be careful.

Two numbers usually missing that reviewers ask for anyway: escalation rate and the false-resolution rate. Escalation rate is how often the agent hands off, which sets staffing expectations. False resolution is the one that hurts, covering cases the agent closed that came back within a week. A system with 88% resolution and a 9% reopen rate is worse than one at 79% that closes clean, and only the second number tells you which you have.

Then the section that builds more trust than all the others combined: known failure modes. Write what actually goes wrong, with the mitigation next to each one. Where the agent misreads multi-part questions. Which intents degrade when a policy page goes stale. How it behaves with mixed-language threads or forwarded email chains with quoted history. Every real system has a list like this. A document claiming none is either untested or dishonest, and experienced reviewers read "no known limitations" as the latter.

Part four: who's watching, and what's kept

Document the oversight design rather than asserting that oversight exists. Which responses get reviewed before sending, how the post-send sample is selected, who holds the authority to pause a workflow, and how quickly a pause takes effect. This section has become a standard buyer ask on its own, and the reasoning behind designing oversight that isn't rubber-stamping is worth working through before you write it, because writing "a human reviews all responses" invites a question you may not want.

Data handling comes next: retention windows for message content, logs, and embeddings; whether customer data is ever used to train or fine-tune; residency; and the deletion path when a contract ends. Much of this will already exist in your DPA, and pointing at that document is fine. Contradicting it is not, so check.

Finish with a change log. Version, date, what changed, who approved it. This is the section that decays first and the one auditors read most closely, because a system card with no entries in eleven months tells them the governance process stopped running.

Where these documents go wrong

Four failure patterns, all common:

• Aspirational metrics. A number with no evaluation set behind it. Reviewers have learned to discount these entirely, so an unsourced 94% is worth less than a sourced 81%.
• Empty limitations. "No known failure modes" reads as untested and costs more credibility than any admission would.
• Marketing voice. The document gets routed through comms and comes back with the specifics smoothed out. Specifics are the entire value.
• No versioning. Written once for a big deal, never touched again, quietly wrong within a quarter.

There's also a subtler one: writing the card for the wrong reader. The audience is a security reviewer or a risk officer who has to decide whether to approve you, not a prospect deciding whether to like you. Those two documents read completely differently, and merging them produces something useless to both.

Publishing it

Keep two versions. A public one that covers scope, actions, oversight, data handling, and failure modes without exposing customer specifics. A full one under NDA with the evaluation numbers, the sub-processor list, and the incident history.

Publish the public one. This is a mild opinion and we'll defend it: almost no support AI vendor does, so the differentiation is nearly free, and it converts a two-week questionnaire cycle into a link. Review it on every material release, with a hard review each quarter whether or not anything shipped.

One caution. A system card is a claim about your system, and claims get checked. Don't write that refunds are capped at £50 unless the cap is enforced in code, and don't write that all deletions route to a human unless they do. The document is only worth something because it's falsifiable, which also means it's a liability if it's wrong. Pair it with real logging so the claims can be verified rather than trusted; what to log and why covers the evidence side, and SOC 2 for AI support tools covers the control framework most buyers will map your card against.

What Robylon documents

Robylon's email agent platform is designed around most of the sections above being answerable rather than asserted. Autonomous resolution runs at 60–80%, and that range is established against your own historical tickets during onboarding rather than quoted from a vendor benchmark, which means the number in your system card is your number.

Action authority is configured per tool across the 60+ write-access integrations, with thresholds and approval gates set explicitly rather than inherited from a default. Every autonomous action is logged with the sources retrieved behind it. Deployment runs 3–7 days, and the evaluation output from onboarding is generally enough to populate the metrics and failure-mode sections of a first card draft.

If you're staring at question forty right now, start with the actions table. It's the section buyers read first, and once the authority limits are written down honestly, the rest of the document tends to fall out of it.

Ready to answer the AI documentation questions in your next security review? Robylon AI resolves 60–80% of customer emails autonomously with per-tool authority limits, full action logging, and configurable human approval across Zendesk, Shopify, Stripe, and 60+ other integrations. Start free at robylon.ai