The release page

The page of one release answers one question: how does this release differ from the release before it? Open it with a click on a release in the Table or in the Trend, with the link View release on the Behaviour card of the Home, or with a chip on a trace.

The address holds the label as one segment, and the range. The range control at the top sets the window of the header numbers, of What changed and of Scores. The card Prompt versions in this release does not follow it: it shows the last 7 days of the traffic of the release. The runs and the cases show the whole release. A label of more than 200 characters has no page. Each card reads on its own. When a read fails, the card shows the error and the other cards stay.

The cards, top to bottom

Header

The title is the label. Under it is the range, for example Last 24 hours. The four numbers are Traces, Traces with errors, First seen and Last seen, all in the range. Open the traces of this release opens the trace list with the filter release: and the label, in the same range.

When no trace carries the release in the range, the card says so. Traces of an experiment run are not counted in release numbers. If the numbers cannot be read, the card says "Could not load the numbers of this release. The rest of the page is unaffected."

Prompt versions in this release

One row for each prompt version that served production calls in this release. The columns are Prompt, Version, Calls in this release, With errors, Median time per call, Cost per call, Label now, Evidence at promotion and First seen in this release.

• A stored version links to its page. A version that is not stored shows the name and the version only.
• The row Unknown hash holds calls whose hash is no stored version. A local edit or a code fallback made these calls. The row has no label and no evidence.
• A run or a case of this release can name a prompt version that no production call carries. The card shows it as a row with 0 calls.
• Under 30 calls, no median shows. A figure that is missing shows a dash and its reason. A cell that could not be read shows Not read and Retry.
• The card counts production calls only. Calls of hosted runs and of experiment runs are not counted.
• The card covers the last 7 days of the traffic of the release, not the range of the page. The cut does not apply to it.

See Prompts and Use a prompt in code.

Controls

• Compare with sets the reference. The default is The release before, with its name. You can choose any other release of the list. The choice goes in the address as reference. What changed and Scores follow it. The other cards do not compare, so they do not change. A release named previous cannot be chosen.
• Sensitivity shows the project setting as text, for example Sensitivity: normal, with the link Settings and the line Minimum traces: 30. You change the setting in Settings → General. See Behaviour and detections.
• Summary and Details switch the view. Details is the default. Your browser keeps the choice. Summary is read only. It has no selector, no sensitivity line and no cut bar.

The cut

The cut bar shows in Details. It narrows the traces that the page judges.

• The field Cut the traces of this release takes terms of the trace filter, for example agent:support-bot model:gpt-4o env:production. It refuses a release term, because the page names its release. It also refuses the terms prompt, prompt_version, prompt_hash and tool. Suggestions for values come from the traces of the release in the window.
• The select Counter has All counters and one entry for each counter, in groups. A counter cuts the detections list and chooses the counter of the hourly series.
• Clear the cut removes both.
• A line under the bar names the window. The range control at the top changes it.

The cut applies to the bands (both samples), the hourly series, the detections and the scores. The header, the prompt versions, the runs and the cases are not cut. In Summary, the filter terms of the cut still apply and a note says to choose Details to change them. The Counter choice does not apply.

What changed

The card has these parts in Details:

1. A sentence, for example "Release 2.4.0 against release 2.3.0 (the release before)." It says "chosen by you" for a pinned reference.
2. Notes: traces with no release that are left out, a sample cut at its cap, and tool counters limited to the 20 tools with the most calls.
3. The story: the status line, up to 5 lines (one for each counter outside its band, the largest ratio first, then "and N more") and the Traces list with the first 5 stored detections. The sentence "Your scores did not move. These behaviours did." shows when a behaviour is outside its band, at least one score interval of the release can be compared with its reference, and every such pair overlaps. Overlapping intervals do not prove that a score did not move.
4. Counters against the band: one row for each counter, in the sections Tools, Errors, Loops and exits, Steps and Guards.
5. Hourly series.
6. Counters: the table.
7. Detections: the list.

The status line

When a fixed rule matched, the line adds · 2 fixed rules matched. A fixed rule never changes the verdict, and the verdict never hides it.

The band chart

Each row has the counter name and the word Inside band, Below band or Above band. Outside the band, the row adds the ratio, for example "3.0 times the band". The chart has four marks: a shaded band, a tick for the reference mean, a dot for the release mean and a bar for the interval of the release mean. A bar outside the band has a hatch pattern. A line under the chart gives the numbers and the trace counts, and says how wide the band is and why.

A counter with no figures has no chart. The row gives the counts and the reason, for example "Below the minimum: 12 of 30 traces" or "Needs content capture". A fixed rule has no chart and no ratio. Its row says how many runs match.

The hourly series

The series shows the release mean for each hour against the band. It covers the counters outside their band (at most 5, behaviours first, cost and time after), or the counter you chose. A circle is an hour inside the band. A triangle that points up is above, and one that points down is below. A hollow tick is a gap: an hour with fewer than 5 traces has no mean. Press Tab, then the arrow keys, to read an hour. A table of the hours sits in a disclosure.

The counters table

The detections list

The list shows 50 rows at a time. Show more loads the next page. A chip on a trace links here, cut to its counter, with the line "Showing the detections of ..." and the link Show all detections. For the meaning of a detection, a class and a review, see Behaviour and detections.

The summary view

Summary shows one sentence with the release and its reference, the status line, up to 5 sentences and the first 3 traces. It has no table, no review button and no control.

Scores

The card reads the 5 scores with the most scores in the project (only scores with a number or a true or false value) and shows those that have a row on this release. A score outside those 5 can have rows on this release and not show here; the card then links to the pages of the other scores. The row of the release to compare with is read over its reference window, the 7 days that end at its last trace, not over the range of the page. Each score has two rows, one for this release and one for the release to compare with. The columns are Score, Release, Mean, Pass rate and Scored. A mean interval needs 30 scored results. A failed judge result is counted apart and never enters a rate. A result that is missing shows a dash, never 0.

When a name holds judge results and other kinds, the card reads the judge results only and says so. The cut applies. See Scores.

Runs

The card lists the runs from your code that name the release, newest first, at most 10. All runs of this release opens the Runs tab for the release. The card shows no verdict, because a verdict needs a baseline. Open a run to choose one. See Evaluations.

Cases

The card lists the cases made from a trace of this release, newest first, at most 20. The columns are Case, Failure class, Expected (Set or Pending), Dataset, Prompt, Added and Source. To make a case, use Add to dataset on a trace. See Traces.

Next Steps