Output Field Targeting
Every evaluator requires you to select an output field — the specific output connector from your flow that the evaluator will inspect.Only text outputs can be evaluated at this time. Non-text outputs (files, images, etc.) will not appear in the output field selector.
Per-Case Overrides
Some evaluator settings act as defaults that can be overridden directly on individual test cases. This lets you reuse a single evaluator across many cases while customizing the expected value for each one. When you open a case, overridable properties appear in the Evaluator values section of the case editor. For example, anEquals evaluator might have a default expected value of "Hello", but for a specific case you can override it to "Goodbye" without creating a separate evaluator.
Properties that support per-case overrides are marked with in the tables below.
Using Dynamic Inputs and Outputs
Some properties also support dynamic references to your flow’s inputs and outputs. These can be inserted in a compatible field using/ or the Insert variable option. They will be represented as chips.
- Input — corresponds to the value of a flow input, as mapped in the case. Useful when the expected output should match or contain the original input.
- Output — resolves to the actual value produced by the flow when the case is run. Useful when comparing one output against another.
The Inputs and Outputs values are case dependent.
Deterministic Evaluators
These evaluators apply rule-based checks. They run instantly, produce consistent results, and do not consume model tokens.Regex
Matches the output against a regular expression pattern.Is JSON
Validates that the output is a well-formed JSON object.Starts With
Checks whether the output begins with a given prefix.Does Not Start With
Verifies the output does not begin with a given prefix.Contains
Checks whether the output contains one or more substrings.Does Not Contain
The inverse of Contains — verifies that certain substrings are absent from the output.Equals
Checks whether the output exactly matches an expected string.Similar (Sequence Matcher)
Compares the output to an expected string using sequence-based similarity scoring.Sequence-based similarity — A method that compares the longest contiguous matching subsequences between the output and the expected value. A score of 1.0 means the strings are identical, while 0.0 means they share no common sequences.
AI Evaluators
AI evaluators use an LLM to assess outputs against natural language criteria. They are more flexible than deterministic evaluators but consume model tokens and may produce slightly different results across runs.Rule-Based (LLM)
Evaluates the output against free-text rules using an LLM judge. The LLM receives the test case input, the flow output, and your rules. It then assigns a score from 1 to 10 based on how well the output aligns with the rules. The score is compared against your pass threshold to determine if the evaluation passes.Criteria-Based (LLM)
Evaluates the output against multiple named criteria using an LLM judge. The LLM scores each criterion independently, giving it a score from 1 to 10 based on how well the output aligns with it. The average of the final scores is compared against the pass threshold. This is useful when you want to assess different quality dimensions separately — for example, accuracy, tone, and completeness.Default Behavior
When an evaluator’s key value is left empty (e.g., anEquals evaluator with no expected value, or a Rule-Based evaluator with no rules), the evaluator passes by default. This means you can add evaluators to a test and configure their values incrementally per case without unconfigured evaluators causing failures.
Combining Evaluators
A test can use any number of evaluators. A case passes only when every evaluator passes. This lets you layer checks — for example:- An Is JSON evaluator to verify the output is valid JSON.
- A Contains evaluator to check for required fields.
- A Rule-Based (LLM) evaluator to assess the quality of the content.