Alerts

An alert rule checks one metric of a project over a window. When the condition is true, Risicare records a firing in History and sends it to the channels of the rule. You make rules and read firings on the Alerts page of the dashboard.

Open the Alerts page

Click Alerts in the side navigation. The bell icon in the top bar opens the same page. The bell is not an inbox: it shows no count and no list of firings. It is hidden on a narrow screen. The page has two tabs: Alert Rules and History. Click New rule on the first tab to make a rule.

A rule

The dialog New alert rule has these fields.

A line at the foot of the dialog reads the rule back, for example "Fires when Error rate above 12.5 % over the last 5 minutes for every trace." Click Create rule. To change a rule, click Edit. The same dialog opens, with Save changes.

Metrics

The list shows each metric with its unit, for example "Error rate (%)".

Type a percent as a percent: 5 means 5 %. A percent has at most 4 decimals, and any other threshold has at most 6.

Filter

The Filter has the same language as the filter bar of Traces. See Filtering and Search. An empty filter means every trace, except two kinds that a filter hides by default. Probe traces (root span control-channel-probe) are left out unless the filter has probes:show or a root: term. Traces that belong to an experiment run are left out unless the filter has an experiment term. Under the bar, the dialog counts the traces that the filter selects in the window, for example "Matches 12 traces over the last 5 minutes (now)." Press Enter in the bar to count again. A filter has at most 2,048 characters and 32 terms. A refused filter shows its message under the bar.

Fields of three metrics

Behaviour drift

The rule compares the traces of a release with the traces of an earlier release. A value above 1 x band means that at least one behaviour is outside its band. The condition is above or at or above. The threshold starts at 1, and the cooldown starts at 6 hours. See Behaviour and detections.

Score mean

The rule reads the mean of one score over the traces that its filter selects. A trace with no score of this name is left out. Risicare never counts it as 0.

Run verdict

The block When a run closes replaces the window and the filter. The rule checks each run when it closes. The threshold is Smallest drop (points), and it starts at 0. The line "Compared with the previous release" has no control. See Runs and Verdicts.

Preview a rule on past data

For Error rate, Tool error rate, LLM error rate, the three latencies, Trace volume, Cost, Total tokens and Score mean, the dialog shows a chart called The last 24 hours. It plots one point for each hour, and the threshold as a dashed line. It reads, for example: "With this rule, 3 of the last 24 hours were over the threshold. The alert checks a rolling window of 60 minutes, so this is an estimate."

• An hour of a rate with fewer than 30 traces is left out, and the dialog says how many hours.
• Show the numbers opens a table with Hour (UTC), Value, Traces and The rule. The last column reads Would fire, Would not fire or Fewer than 30 traces.
• Score mean needs a score name first: "Pick a score to see the last 24 hours."
• Behaviour drift has no preview.

Run verdict shows The last 10 completed runs in scope. Each run has its name, its stored verdict, the drop in points, the cases compared and what the rule would do: would fire, clear, below the minimum or a reason. The list applies no cooldown and sends nothing.

Manage rules

Each rule is one row. A row shows:

• the name;
• the badges Off and the severity;
• the number of firings;
• the rule as a sentence;
• the filter, or "Every trace";
• "No channel (History only)", or the number of channels;
• the cooldown;
• the number of checks in which the condition was true but the cooldown held back the firing;
• when the rule last fired.

The tab shows the first 50 rules.

Test shows the value now and the state of each delivery. It checks every 2 seconds for 20 seconds, then says "Still pending after 20 seconds. History shows the final state." A test is not a firing: it does not start the cooldown and it sends no alert.triggered webhook. You can test one rule at most 6 times in 60 seconds.

History

The tab History lists the firings, newest first, 50 at a time. Load More shows the next 50. A row has:

• The name of the rule, or "Deleted rule", and the severity. A test row has the chip Test.
• The value against the threshold, for example "Error rate: 3.4 % (threshold above 5 %)", then the window and the filter, or "every trace".
• One line for each channel: the name, then sent, pending or failed. A failed line shows the error text. A rule with no channel shows "No channel: recorded in History only."
• "Triggered" or, for a test, "Sent", and the time.

A Behaviour drift row adds the worst behaviour. A Score mean row reads "Value: score mean quality 0.583 (24 scored, 0 without a score). Condition: below 0.70." A Run verdict row reads "Regressed: 18.0 points lower (± 11.5 points, p = 0.0021)", names the two runs and the cases compared, and has the link Open run. A test of a channel from Settings shows as "Channel test".

Start a rule from a chart or a run

• From a chart: open the menu of a chart on a dashboard and click Create an alert. Under a row of a ranked-bar chart, click Alert on this group. The Alerts page opens with the dialog filled in. The metric comes from the measure of the chart. The window is 60 minutes for an hourly chart and 1,440 for a daily chart. The filter joins the filter of the dashboard, the filter of the chart and, for a group, the term of the group. An empty threshold gets a suggestion from the last 24 hours. A chart whose measure has no alert metric shows no item.
• From a run: a Regressed verdict card has the link Alert me when a run regresses. It opens a new Run verdict rule with the dataset of the run chosen. See Alert on a regression.

Watch a prompt label

For the metrics Error rate, Latency p95, Cost, Score mean and Run verdict, the dialog has the section Watch a prompt label. Choose a Prompt and a Label. The rule follows the version that the label holds. When the rule fires, Risicare proposes a rollback. It never moves a label by itself. See Rollback proposals.

Who can do what

A viewer sees the controls of a rule disabled, with the text "Only members, admins and owners can make or change alert rules. Your role here is viewer." See also the permission matrix.

Next Steps