Compare on a dataset
Run two saved prompt versions on a dataset through a model connection, and read a verdict.
A hosted comparison runs two saved versions of one prompt on the same cases of a dataset. It uses the same trials and the same checks for both versions. It calls the model through a model connection, with your provider key. It ends in a verdict.
A hosted comparison tests the prompt alone: one model call for each case and trial, and judge calls when a check uses a rubric. A run from your code tests the agent end to end.
A comparison needs five things
A model connection that is Ready. The project switch Allow hosted runs set to On. The role member, admin or owner. Two different saved versions. A dataset. A draft cannot be compared until you save it as a version.
Open the form
- From a prompt. Open the compare page of two versions. See Compare two versions. Under the stored verdict, the card Compare on a dataset holds the form. It shows when two different versions of a prompt that is not archived are open. Before a comparison runs, it reads "Run one to get a verdict."
- From the Playground. Click Compare on a dataset beside Connection in a session. Two saved panes open the compare page with those versions. One saved pane opens it with that version as the candidate, and the page picks the baseline. A draft pane first opens Save the draft first. See Save a draft as a version. A session with no prompt reads "Open this session from a prompt to compare on a dataset."
The form
| Field | What it sets |
|---|---|
| Dataset | The dataset. Each option shows the name, the number of cases and the version. |
| Split | Dev, the cases that you can tune on, or Holdout. A holdout comparison counts as a look at the holdout, and each look makes it a weaker test. |
| Trials per case | 1 to 10. Under 3, a note reads "Use at least 3 trials to see flips." |
| Checks | Matches the expected output, Contains the expected output, Output matches the schema, Calls the expected tool (with a tool name) and Judge with a rubric (with a rubric name; offered when the project has a rubric). A trial passes when all the chosen checks pass. |
| Cases | All cases, or A sample of cases with a size. |
| Connection | The model connection. |
| Variables | The field of the case for each variable of the two versions. The default is input. followed by the variable name. The whole input maps a string input. A line reads "question ← input.question · found in 50 of 50 cases". When some cases lack a field, choose Leave out the 3 cases or Choose another field. |
If a version has no model, the form asks for a model name for that version only. The run records it.
The form starts from the gate settings of the prompt. Without them, it starts with the first dataset, the Dev split, 3 trials and the check Contains the expected output.
Before you start, a line gives the size and the cost bound, for example "Up to 600 model calls: 300 for each version, judge calls included. Up to $1.20 if every call uses its max output tokens. my-connection: $3.20 spent of $50.00 this month." A comparison makes at most 20,000 calls, and a project runs at most 2 comparisons at once.
Run a comparison
Click Run the comparison
If the button is disabled, the reason shows beside it, with a link to Settings → Model connections when the fix is there. For example: "Hosted runs are off for this project. An owner or admin can turn them on in Settings, Model connections."
Wait for the calls
While the tab is visible, the card reads the job every 2 seconds, for example "Running: 186 of 300 calls done." Cancel the comparison stops it before its next call.
Read the verdict
When all calls are done, the page makes the verdict. If it could not, the card reads "Not finalized. All calls are done. The runs and the verdict are not made yet." Click Make the verdict.
A run with fewer than 20 cases can still start, but it gives no verdict. The form says so first: "This run gives no verdict: a verdict needs at least 20 cases, and this split has 12." The button then reads Run it anyway.
A comparison can end without a verdict. A connection that reaches its cap ends it: "Stopped: my-connection reached its monthly cap of $50.00. 412 of 600 calls ran." A cancel ends it too: "Cancelled by you. No verdict." If nobody makes the verdict within 30 days after the comparison ended, Risicare erases its cases, and the verdict can no longer be made. Run the comparison again.
The result
- A chip with the verdict: Regressed vs v3, Improved vs v3, No change vs v3 or No verdict vs v3. A sentence says how many points the pass rate moved, and whether the change is inside or outside the noise.
- A table of Pass rate, pass^k, Flips and Failed or missing trials for the baseline and the candidate.
- The counts of cases that got worse, better and stayed the same. Each count links to the case diff. A link reads "Read the 15 cases that got worse".
- A line of facts: the dataset and its version, the split, the number of cases, the trials, the checks, the connection and the times. Then "Cost of this comparison: $0.42." and "Hosted run: the prompt alone, one model call per case and trial."
- At 7 days or more, a note reads "This verdict is 9 days old. The provider may have changed the model since then."
Run again opens the form again.
Where the runs are
When the verdict is made, Risicare writes two runs, one for the baseline and one for the candidate. Open them in Runs, the second tab of Releases. A hosted run makes no trace. Its page reads "A hosted run makes no trace. Each answer is stored with the run: open a case to read it, with its cost and time." See Evaluations.
Who can do what
A member, an admin and an owner can run a comparison. A viewer sees the form and the reason "Your role can read comparisons. A member can run them." An owner or an admin adds connections and sets Allow hosted runs.