Datasets

A dataset is a named set of cases. A case holds an input, an expected answer and an optional failure class. A run takes its cases from one version of one dataset. A dataset name is unique in a project.

The Datasets page

The Datasets item in the side navigation opens the list. It shows 50 datasets on a page, with Next page and First page. Each row has the name, the tags, the description, the number of cases and the time of the last update. Filter by tag takes one exact tag.

A member, an admin or an owner can click Create dataset. The form has these fields:

A dataset page has the buttons Export, Edit dataset and Delete dataset. A viewer sees Export only. The page lists the newest cases, at most 1,000. Under the cases, Newest runs of this dataset lists the five newest runs, with the link All runs of this dataset.

Cases

The case table has the columns Case, Expected, Failure class, Release, Prompt, Source trace, Added and Version. A click on a row opens the case.

• Input: the text that your function gets.
• What the system answered (evidence, not graded): the answer that the trace gave. You cannot edit it. Clear the evidence removes it.
• Expected answer: an empty field is Pending. A filled field is Set. Use the evidence as the expected answer asks "Use this only if the answer was correct." before it fills the field.
• Failure class: a choice from the failure classes of the project.

A run runs only the cases that have an expected answer and an input that is not empty. The run page counts the others. A member or higher role saves a case with Save and removes it with Delete this case. A viewer reads the case and changes nothing.

Versions

A new dataset is version 0. Every change to a case makes one version: a case added, edited or removed. The name, the description and the tags make none. The link Version N in the header opens Versions. Each row has the number, the change, the author (You, A team member or An API key), the time and a link to the case. A click on an older version opens the dataset read only, with the line "You are reading version 3. It is read only."

A run pins one version. If you do not name a version, the run reads the newest version at the time it starts.

Add to dataset

Add to dataset is in the header of a trace and on each span. It needs the member role or higher. It is not on the preview panel of the trace list or on a session page. The labelling queue offers it too, as Make a case after you save a fail label.

The case keeps the first user message as its input. The answer of the trace is kept as evidence. The expected answer is pending, and you set it later. A trace with no user message keeps the input preview, the first 1,000 characters. With content capture off, the case holds the source ids only, and you type the input in the case editor. See Traces.

Copy these cases

On the case diff of a run, the button Copy these N cases copies the cases that the filters show, for the whole run. It needs the member role. A copy is a new case. It keeps the input, the expected output and a link to its source case.

• Copy into is A dataset of this project or a new dataset name. The dataset of the run is not offered.
• One copy takes at most 1,000 cases.
• Risicare keeps one trial for each case: the first trial that did not pass, else the first trial that passed. The answer of that trial is saved as evidence. This covers the first 50 cases. The rest keep their source trace link.
• A run on the holdout cannot be copied.
• The result reads, for example, "15 cases copied to my-dataset (version 12 → 27). 2 already there. 3 went to the holdout." A second copy does not repeat a case.

The holdout

A holdout is a share of cases that you cannot tune on. A key cannot list or export them. A key reads the holdout cases of a holdout run only while that run is open.

• Who sets it: an owner or an admin, in Create dataset or Edit dataset, in the field Holdout share (0 to 50). The value is a whole number from 0 to 50. The default is 0.
• It is final: the form asks once "Set the holdout share to 20 %? You cannot undo this." After you save a share above 0, the page shows "Holdout share: 20 %. The share is final." For a different share, make a new dataset.
• How a case is assigned: when a case is added, a hash of its id decides. A case below the share goes to the holdout, and every other case is Dev. A case never moves. Cases that exist keep their split. Only cases added later use the share. A copy into the dataset follows the same rule. A holdout case has the chip Holdout.
• What an API key gets: an export with a key and the pytest --collect-only list leave holdout cases out. The cases of a holdout run are served to the run helper while the run is open, and never again after it closes.
• What a person sees: every case. The export from the dashboard includes holdout cases.

A run on the holdout uses split="holdout". Its cases are served only while the run is open. Each holdout run counts as a look. See Runs and verdicts.

Export

Export opens Export cases. It has three choices:

The dialog counts first, for example "41 cases are ready. 3 have no expected answer and are left out. 1 has no input content and is left out." Download the file gives a file with one case on each line (.jsonl). The evidence is not in the file. With no case ready, the button reads No case is ready to export.

Next Steps