Where tests come from
Most tests start from a real conversation.- From a past chat. Open a chat in Past chats, then click Test this reply under any agent reply. The chat before that message becomes the test’s context, the message becomes the test’s first user message, and the reply itself is never copied into the test. Add the checks that say what the agent should have done, pick a test set (it starts on the set you last opened on this agent, or New test set), and save.
- With Genie. On an agent with no tests and at least one chat, Ask Genie to set up tests opens Genie with a request for tests from the agent’s recent and thumbs-down chats already written. Send it, and approve what Genie asks to approve.
- By hand. New test on a test set opens an empty test. Write the user messages in order, save, then add its checks.
The Tests page
The Tests page is a table of the agent’s test sets. Each row shows the set’s score (the pass rate of its latest run), the change in pass rate against the version it ran on before, how many tests regressed (passed before, not now), its number of tests, the average time and cost of a test, and when and on which version it last ran. A regression shows in red, so a set that got worse stands out. Cost shows only to members who can see conversation costs. Click a row to open the set. Run on a row runs it, and the row’s menu has Ask Genie why it fails and Delete.Comparing two versions
A set opens on a comparison: a Baseline version and a Candidate version. The candidate is the latest run and the baseline the run before it on another version; pick any other run in either picker. The choice stays in the address, so a reload or a shared link shows the same comparison. Under the pickers, a summary shows the candidate’s pass rate, average time and average cost per test, each with its change against the baseline, and a verdict: how many tests regressed and improved. Show filters the tests to those that regressed, improved, stayed the same or are failing; regressed tests come first. Each test row shows its outcome on both versions, its time and its cost (each with its change against the baseline; the cost only for members who can see conversation costs), and its first failing check; click it to open the test, and use its … menu to run, duplicate or delete it. Below it, History lists each version with its pass rate on its newest complete run, oldest first; hover a version to read its changelog, and click it to compare it with the version before.A test
A test opens in two columns, under the same two pickers as its set.- On the left, the conversation. Each user message and the agent’s reply on the candidate version, rendered as a chat. Side by side in the test’s toolbar puts the baseline’s conversation beside it. A test made from a past chat shows that chat above the first message, and Open the chat opens the source. Edit messages, in the toolbar’s … menu, lets you add, remove, reorder and rewrite the user messages; leaving with unsaved changes asks first.
- On the right, the checks. Each check of this test with its result on both versions and the candidate’s reason (the judge’s reason for a judge check). A failed check offers Ask Genie why it failed. Add a check opens the catalogue, and each check can be edited or removed on its own. A check is named from what it checks (for example Mentions bank transfer); rewrite its Name in the form to call it something else, and that name shows on the set and in results.
Checks
Fourteen checks in four groups:
Tool names and arguments come from the tools the agent’s model sees. Knowledge base checks use the agent’s knowledge bases; a required document that is no longer in its knowledge base gives “could not evaluate”, never a failure.