Evaluations
Create evaluations that have an AI model review conversations against your criteria, returning pass or fail, a score or written analysis, on demand or on a schedule.
An evaluation is a check that an AI model runs on your conversations. You describe what to look for, for example “Did the agent confirm the order number before cancelling?”, and choose the kind of result you want: pass or fail, a score, or a written analysis. Robylon then reviews conversations for you, so you can check hundreds instead of reading a handful.
Open it from Settings → AI Intelligence → Evaluations.
What an evaluation can return
Choose one Metric for each evaluation:
| Metric | Result | Good for |
|---|---|---|
| Binary | Pass or Fail | Rules that are either followed or not: “Did the agent ask for consent before sharing the tracking link?” |
| Rating | A whole number on a scale you set, 1 to 5 by default | Quality you want to trend over time: “How well did the agent resolve the issue?” |
| Text | Free-form written output, as bullet points or a paragraph | Open questions: “Why did this customer ask for a human?” |
Every result also comes with the model’s reason, so you can see why a conversation passed, failed or got its score.
Create an evaluation
Click Create Evaluation, describe what to check, pick a model and metric, and publish.
Start a new evaluation
Go to Settings → AI Intelligence → Evaluations and click Create Evaluation.
Name and describe it
Under Basics, enter a Name (up to 100 characters, unique in your workspace) and a Description, the short summary shown in the evaluations list.
Choose the model and metric
Under Model & Metric, choose the Model that will review conversations, then the Metric. For Binary, describe what a pass and a fail look like. For Rating, set the lowest and highest values and what each means. For Text, choose the Output Format: Bullet points or Paragraph.
Choose what the model reads
The conversation history (customer and agent replies) is always included. Under User Info, you can also include AI Internal Messages (the agent’s internal steps), Internal Notes and Additional User Info.
Write the criteria
In Additional Instructions, write exactly what the model should check and how to judge it. Robylon’s own evaluation prompt wraps your instructions, so results stay consistent from week to week.
Set up auto-run (optional)
Turn on Auto-Run This Evaluation to run it on a schedule. See Run an evaluation automatically
Create and publish
Click Create Evaluation. Test it, then Publish it to make it live.
Test an evaluation
Use Test. It runs the evaluation on your most recently updated closed chat, completed call or email ticket, so you can check the result and the reason before running it on more conversations. Adjust the instructions until the result matches what you would have decided yourself.
Drafts and published versions
Each evaluation has a draft and a live version. Save keeps your changes in the draft. Publish makes the draft the live version that runs on conversations, and saves a version you can refer back to. On an existing evaluation, View Live Version shows what is currently running.
The evaluations list shows each one’s Type, Status (Live, Paused or draft), whether Auto-Run is on, how many Runs it has done, and when it was last updated.
Conversations you can evaluate
You can evaluate closed chats, completed voice calls and email tickets. For voice calls, the call recording can be included for the model to listen to.
Run an evaluation on specific conversations
Select the conversations in Transcripts or the inbox and choose Run evaluations. Evaluations do not run automatically when a conversation closes; use auto-run for that.
Run an evaluation automatically
Turn on Auto-Run This Evaluation and choose how often it runs, how many conversations it may run on, and which conversations qualify.
- Frequency: every 12 hours or every 24 hours (the default).
- Limit: the most conversations it may evaluate per week or per month.
- Conditions: only evaluate conversations that match, for example by CSAT rating, who closed it (AI or Human), Channel, Tags or Properties. Combine conditions with and/or.
Each run picks conversations updated since the last run that this evaluation has not already reviewed, oldest first, until it reaches its limit. It does not sample at random.
Where results appear
- On each conversation: the Evaluations section of the conversation panel lists every evaluation run on it, with its result, and when and by whom it ran.
- In filters: filter Transcripts and the inbox by Evaluations to find conversations an evaluation has run on, and by evaluation result under Evaluation.
System evaluations
Robylon’s six built-in evaluations, marked System in the list, explain why conversations were handed to a human or rated low. They feed the Insights page.
| Evaluation | What it classifies |
|---|---|
| Chat Handover | Why each chat handed over to a human agent left the AI, and the fix that would have kept it automated. |
| Email Handover | The same, for email tickets. |
| Chat AI CSAT | Why chats the AI closed were rated 1, 2 or 3, from the AI’s turns only. |
| Email AI CSAT | The same, for email tickets. |
| Chat Human CSAT | Why chats a human closed were rated 1, 2 or 3, weighing the turns after the handover. |
| Email Human CSAT | The same, for email tickets. |
Each result is a reason and a fix type, for example “Customer asked for a human” with a Knowledge base fix. See Reasons and fix types. The chat evaluations cover every channel in the Chat group, including WhatsApp.
How they differ from your own evaluations:
- They run on their own. Turn on Auto-Run This Evaluation and new conversations are classified every hour. They cannot be run by hand on chosen conversations.
- Robylon maintains the prompt, so results stay comparable week to week. Click View Base Prompt to read it, and add your own Additional Instructions on top.
- Weekly suggestions come from Generate Insights On This Evaluation. See Weekly suggestions.
- Each runs on a set number of conversations a week: by default 500 for each handover evaluation and 250 for each CSAT evaluation, and never more than 2,500. Conditions can narrow the conversations they look at, but not widen them.
- They cannot be archived.
System evaluations are available on request: email support@robylon.ai.