Runs and Verdicts
Read the list of runs and the page of one run, choose a baseline, understand the verdict, and open the case diff.
Runs are on the Runs tab of Releases. The Releases item in the side navigation covers both tabs. The Runs tab has no range control and no button that starts a run. Runs start in your code.
The list of runs
Four tiles sit on top: Runs, Completed, Running and Failed or cancelled. The panel Run from your code shows an example and the gate command to copy. It makes no request.
The table lists runs newest first, 25 on a page, with the link Older runs. The columns are Status, Name, Release, Prompt, Dataset (with its version and the split), Cases × trials, Pass rate, Failed and missing and Started. A run that is still running after 24 hours with no change reads Not finished.
| Filter | Choices |
|---|---|
| Status | Running, Completed, Failed, Cancelled |
| Release | A release label |
| Dataset | A dataset of the project |
| Split | Dev or Holdout |
Apply sets the filters. Clear filters removes them.
The page of one run
From top to bottom: the header, the numbers, the prompt line, the verdict, the checks and the cases. The header has the name, the status, the release, the prompt, the dataset with its version, the split, the trials, the times and the run id. Your key shows the external_id. If the dataset has cases that the run does not run, a line says how many have no expected answer, no input, or are in the other split.
The numbers
| Tile | What it shows |
|---|---|
| Pass rate | The mean over cases of the share of decided trials that passed. Each case counts once. The tile shows the half-width of a 95 percent interval, for example 62 % ± 6.0 %. The interval is the rate plus and minus 1.96 standard errors, cut to 0 and 100 percent |
| pass^1 to pass^k | For each case that has all k trials decided, the chance that j trials drawn at random from its k all pass. The tile shows the mean over those cases for j from 1 to k. pass^k is the share of cases that pass every trial |
| Flip cases | The cases that pass in some trials and fail in others, for example "16 of 50". The second number counts the cases that have every planned trial decided. A case with a trial that gave no decision is in neither number. It needs 3 trials. With fewer, the tile reads a dash and "Use at least 3 trials to see flips." |
| Failed and missing | Trials with no decision, and trials with no result. Planned trials, minus decided, minus failed, are missing |
| Cost, Latency p50, Latency p95, Traces, Tokens | Read from the trial traces of the run. A hosted run has none of these |
A number that the api did not give reads as a dash, never as 0. The line Declared prompt names the prompt of the run and says on how many trial traces Risicare saw it.
The table Checks has one row for each check: Kind, n, Failed, Missing, Rate or mean and Judge. A check that two judges scored shows a warning, because its rate mixes them. The table Cases has one row for each case with an Outcome (All pass, Flip, Never or Incomplete), one dot for each trial, and the passed trials over the decided trials. When you choose a baseline, it becomes Compare the cases: each row shows the baseline, the candidate and the change in points, and has the link Diff. Each dot opens its trial trace. The filter Outcome cuts the table.
The verdict
The verdict card starts with Baseline. The choices are No baseline, the 20 newest completed runs of the same dataset and split, and one entry for each release among them, for example Release 2.3.0, which is the newest completed run of that release. The choice is the baseline value in the address. With no baseline the card says "Pick a baseline to see if this run is better or worse."
How the verdict is made
Risicare pairs the cases that both runs decided and that have the same content. A change of a label is not a change of content. It tests the mean of the differences of their pass fractions. The noise band is plus and minus 1.96 standard errors. A difference inside the band is No change. Regressed needs a p-value below 0.05 and a lower pass rate. The card counts the cases compared and the cases left out.
| Word | The line under it |
|---|---|
| Regressed, Improved | "Pass rate 20.0 points lower than the baseline. Outside the noise (± 11.5 points, p = 0.0007)." |
| No change | "Inside the noise: ± 11.5 points. The change is not proven." |
| No verdict | With a number: "Pass rate 5.0 points lower than the baseline. This is not a verdict." Then the reasons, and "The numbers below are not a verdict." |
Only Regressed and Improved have a colour. The card also shows two lines:
- At the end, the note: "The interval is a normal approximation over cases. Two equal runs show a false regression, or a false improvement, about 1 time in 40, so a team that makes many comparisons will see some."
- What the gate does for this baseline, for example "The gate passes. risicare eval gate would exit with code 0." This line uses the baseline alone, with no other option.
No verdict
A No verdict always comes with a reason. The card gives one or more of these:
- Fewer than 20 cases can be compared between the two runs.
- More than 10 percent of the trials of this run failed or never arrived.
- More than 10 percent of the trials of the baseline failed or never arrived.
- The two runs use different pass rules, or the run has no pass rule yet.
- A required check was judged by a different judge in each run.
- The run is not finished.
- The gate asks for a holdout run, and this run measured dev cases.
Three notes do not block the verdict. The judge details of one run have expired. The two runs declare different prompt versions. The two runs have different numbers of trials.
The page refuses a baseline of another dataset or another split. It also refuses the run itself, and a release with no completed run on this dataset and split.
The holdout line
A run on dev cases says "Measured on cases you can tune on." A holdout run says "Holdout run. This holdout has been run 3 times (this version) and 5 times (this dataset)." The count includes runs that you deleted. The cases of a holdout run are served only while the run is open, so a new look needs a new run. See The holdout.
The case diff
A Regressed card has the link "Open the 15 cases that got worse". The count line above the cases has one link for each of got worse, got better and no change. With a baseline chosen, each case row has Diff. All of them open the case diff, which reads the cases one at a time. It shows the baseline answer beside the candidate answer, trial by trial. The page changes nothing, except Copy these cases.
- Show: Got worse, Got better, No change, Flip (3 trials or more), Never passes and All cases. After a Regressed verdict the page opens on Got worse.
- Class: Any class or one failure class. A line counts the match, for example "3 of the 15 cases that got worse are Fabrication."
- The list of cases: 50 at a time, with Load 50 more cases. A row shows the start of the case id, the first 60 characters of the input, and the change, for example "3/4 → 0/4".
- Copy these N cases: copies the cases that the filters show into a dataset. See Copy these cases.
Press ? to list the keys. They do nothing while the focus is in a list of choices or a window is open.
| Key | Action |
|---|---|
| J or K (also Down and Up in the case list) | Next or previous case |
] and [ | Next or previous candidate trial |
} and { | Next or previous baseline trial |
| D | Show or hide the word marks |
| O or B | Open the candidate or the baseline trace in a new tab |
| E | Show or hide the expected output |
Mark as cancelled and Delete run
The run page has Delete run, and, on a run that is still running and has not changed for 24 hours, Mark as cancelled. A member, an admin or an owner can use them. A viewer sees them disabled, with the reason "Your role cannot change runs."
- Mark as cancelled closes a run that is still running and has not changed for 24 hours. Risicare closes no run by itself.
- Delete run removes the run from every list. You cannot restore it from the dashboard.
Hosted comparison
A hosted comparison runs two saved versions of a prompt on a dataset through a model connection, and ends in the same verdict. It writes two runs. A hosted run makes no trace, so its page has no cost or latency tiles. See Compare on a Dataset.
Alert on a regression
On a Regressed card, a member or higher role sees the link "Alert me when a run regresses". It opens Alerts on a new rule of the metric Run verdict. The rule fires when a run closes as completed and its verdict against the previous release is Regressed. It takes the drop in points, Datasets (up to 20) and Minimum cases (20 to 10,000). See Cost Alerts for the other fields.