Scores and Labels

A score is a named value on a trace, on a span or on a session. Scores come from your code, from the checks of a run, from Add score on a trace page, and from people who label targets in the dashboard.

There is no page of scorers and no library of built-in scorers in the dashboard. There is no /scorers page. You write the scoring in your code, and Risicare stores and reads the scores.

Record a score from code

risicare.score() (npm: score()) is part of the SDK. The typed form needs Python 0.8.0 or npm 0.11.0 and later. An ingest key can send scores.

import risicare
 
risicare.init(api_key="rsk-...")
 
risicare.score(name="sql_valid", value=True, trace_id="4bf92f3577b34da6a3ce929d0e0e4736")

For a score from a judge or from an experiment, add kind="eval", the judge fields (judge_model, judge_version, judge_prompt_hash), trial, experiment_id and failure_class. The fields, the limits and the rules are in the Python reference, the JavaScript reference and the REST API reference. The checks of a run from code post one score for each check and trial.

Add a score in the dashboard

In the Scores block of a trace page, click Add score, or press S when no field, dialog or menu has focus. The form asks for a name (up to 128 characters), a value (a number, a label of up to 256 characters, or true or false), a kind (annotation is the default), a comment (up to 4,096 characters) and a target: this trace, a span, or this session.

The page of a score name

No page lists all score names. You open /scores/{name} from a score on a trace, or from the Scores card of a release page. The page has:

• The header: the value types and the kinds found for the name, and the pass rule with its source: "You chose it on this page.", "From rubric version 2." or "No rubric rule: the default."
• The pass rule controls: Pass at for a number (above, at least, below or at most, and a number), Pass labels for a label (1 to 20 labels, separated by commas), and Pass when for true or false. Apply puts the rule in the address of the page, so a link keeps it.
• The other controls: Window (All retained labels by default), Scope (Span, Trace or Session), Judge and Failure class.

The judge validation card

The card Judge agreement with people compares a judge with the labels of people on the same targets. A judge is whatever posts scores of kind eval for this name: your model or your own rule. Your code sends its identity with judge_model, judge_version and judge_prompt_hash. The card reads each identity on its own, so a new prompt hash is a new identity. A label is a score of kind annotation.

The card shows plain numbers. It has no colour scale and no verdict word. For the judge it gives:

• how often the judge passes a target that people passed, and how often it fails a target that people failed, each with an interval;
• the agreement, next to the agreement of a judge that gives one answer to every target;
• kappa with its interval, which removes the agreement that chance gives.

A figure shows only when the card has enough labels: 50 targets that people passed and 50 that people failed, each also scored by the judge. With fewer, the card shows the counts and no rate. It always shows the judge calls that failed, the labelled targets with no result, and the targets with labels that conflict, which it leaves out. A table by failure class splits the fail side, and a class with fewer than 50 labels reads "Not enough labels". A note says that pairs from one trace are not independent, so an interval can be a little narrow.

Targets and the rubric

Targets lists the targets in tabs: Judge passed, people failed, Judge failed, people passed, Conflicting labels, Judge call failed and No judge result. Label these opens the queue on the group of the tab. It shows on the disagreement tabs and on Conflicting labels.

The block Rubric holds the text that says what pass and fail mean. It shows the current version, who saved it and when, and the Earlier versions. A version is never edited or deleted. A member or higher role clicks Save as new version, writes the rubric text (up to 8,000 characters), and keeps or changes the pass rule. After a save, "Labels made before this time stay under the earlier version." A card that needs new labels under the current rubric says "3 targets need a label under rubric v1" and links Label them. A link such as Show the card for rubric v0 reads the card of the older version.

The labelling queue

Label items on the card opens /scores/{name}/queue. It lists 20 items a page. The queue shows targets that a judge scored. A viewer can read it, and a member can label.

The queue is blind by default. It hides the answer of the judge, and a reason that tells the direction of another label reads "Second opinion". The switch Show the judge before I label turns that off. Saving a label for an item also shows the judge for that item.

After you save a fail label on an item that has a trace, the queue offers Make a case. It reads the trace and opens the Add to dataset form, with your label as the note of the case. See Datasets.

The source menu has Best next, Contested, with your label, Judge and people disagree, Classes with too few labels and No label yet. An item that you labelled under an older rubric version comes back with the note "Label it again under v1."

Failure classes

A failure class names why an answer failed. The vocabulary of a project is the ten built-in classes, then the classes of the project. The built-in classes are a design aid, not a measure of how often each class happens. The dashboard lists the classes and has no control that adds one.

A class shows in these places:

• Add to dataset and the case editor, as Failure class, and the column Failure class of a dataset. See Datasets.
• The case diff, as the filter Class. The class of a case is its label, or else the class that a judge or a rule gave to a failing trial of the run. See Runs and verdicts.
• The score page, as the control Failure class and the class table of the card.
• The queue, as the class list after F.
• The Cost page, as the group Failure class. It stays off until you choose a score in the score picker, because it groups the numbers of that score. See Cost Tracking.
• Your code: score(failure_class="fabrication") (npm: failureClass) takes a key of 2 to 63 lowercase letters, digits and underscores. The api answers 422 for a key that is not in your vocabulary.

Find scored traces

In the filter bar of Traces, score.<name> matches the latest score of that name on a trace. Sessions takes it for session scores. An alert rule can use the metric Score mean. See Filtering and Search.

Next Steps