Behaviour and detections

Risicare counts simple things in each trace: steps, tool calls, errors and repeats. It compares the counts of one release with the counts of the release before. When a count moves outside the band that the earlier release sets, Risicare reports it.

What is compared

• The release sample is the settled traces of the release that start in the window of the page. A trace settles when its newest span is at least 2 minutes old. A trace with no release, and a trace of an experiment run, are in no sample. Each side reads at most a fixed number of the newest traces. When it hits the cap, the page says so.
• The reference sample is, by default, the release before. That is the release with the greatest first seen time below the first seen time of this release. Only releases that have enough settled traces in their own reference window count. First seen is the earliest trace start of the release in the data that Risicare reads for this view: the data kept for the project, but at most the last 90 days. A release that began earlier is ordered by its first trace inside those 90 days. The reference window is the 7 days that end at the last trace start of the reference release.
• You can choose another reference with Compare with on the release page, or in a Behaviour drift rule.
• The first release of a project has no baseline. Its status is No baseline yet.
• The page computes the bands when you open it. Risicare keeps an answer for 60 seconds.

The trend uses another window. It measures each release over its own last 7 days, and the release page uses the range of the page.

What is measured

One trace is one run. A mean is the mean for each trace. A share is the share of traces that have the thing.

• The tool counters cover the 20 tools with the most calls, among the tools in at least 5 % of the traces of one sample.
• The counters that read order use the first 2,000 tool spans of a trace. A retry that a client library makes inside one tool span is not seen.
• Risicare sorts a tool error into a class by the first source that matches: the error code, the exception type, the HTTP status, then the error text (only when the span has content). All else is an unknown cause.
• The repeat counters need content capture. With it off, they have no value and read "Needs content capture". The exit counters show only when your spans carry an exit status.
• Cost per run and Time per run are guards. They show beside the behaviours, but they are not in the count of behaviours, they write no detection, and a Behaviour drift rule that names no counter leaves them out.

How a counter is judged

1. The band is the reference mean plus or minus a width h.
2. h is the largest of four numbers:
   • the half-width of the interval of the reference;
   • the difference between the means of the first and the second half of the reference sample;
   • a floor of 0.2 for a mean or 0.05 for a share;
   • a share of the reference mean that depends on the sensitivity.
3. Risicare puts the release values in time order and cuts them into two halves by count. The ratio of a half is 0 when the interval of the half holds the reference mean. Otherwise it is the distance from the reference mean to the nearest end of that interval, in widths h.
4. The ratio of the counter is the smaller of the two, when both are above 0 and both halves are on the same side of the reference mean. Otherwise it is 0.
5. A counter is outside its band when its ratio is above 1. A burst that fills one half cannot fire.

A mean uses the standard deviation of the sample for its interval. Below 30 values it uses a Student t factor in place of z, which makes the interval wider. A share uses a Wilson interval at z.

The release sample and the reference sample must each reach the minimum, counted in values that are not empty. Each half of the release sample must reach the larger of 5 and one third of the minimum. Below that, the counter reads Below the minimum and has no band, no ratio and no figure. A counter with no value on more than half of a side has no figure, and its row gives the reason.

A release can read "Inside its normal range" while some of its behaviours have no band. The status line, the chips, the tile and the Home card speak for the behaviours that were judged.

Fixed rules

Two rules hold in every release, the first one too. They have no band and no ratio. They count the runs that match.

A polling agent that repeats a call on purpose looks like a loop to the loop rule.

What a detection is

A detection is one stored row: one trace that was outside a band, or matched a fixed rule, for one counter. Guards write none.

• A background pass of Risicare stores the detections. The page only reads them. The bands are computed when you open the page, so a counter can be outside its band while the detections list is empty. A row is a claim made when it was written. The first row for a trace and a counter is kept.
• The pass stores at most 1,000 rows for each release, counter and key. The list then says "Showing 1,000 of 4,512" or "Only the newest 1,000 rows are stored". The list reads at most 5,000 stored rows of the release and then applies the cut. When more than 5,000 rows match, a note reads "More than 5,000 stored rows matched, so the list looked at the first 5,000."
• A row shows the trace and its span, the counter, the value of the trace against the band, and the mode: Against release 2.3.0 or Fixed rule.
• The class is a design aid, not a count of how often each failure happens. Risicare never guesses a class. No class means "Needs a person to read the run."
• Real change and Not a problem record your review. Risicare saves it as a score of kind annotation on the trace and, when the row has one, the span. The name of the score starts with behaviour-review.. The Scores card leaves these scores out. A role that cannot add scores gets a refusal.
• A detection stops showing when its own retention period, counted from the start of its trace, ends.

Chips on a trace

The header of a trace page shows one chip for each stored detection of the trace, at most 20. The text is the counter in words, then the class or No class. A trace with no stored detection has no chip. A chip opens the release page, cut to its counter, in the narrowest range that holds the start of the trace.

Where behaviour shows

• Home: the Behaviour card names the newest release of the range that has at least the minimum number of settled traces. It lists up to 3 behaviours outside their bands, the largest ratio first, and "and N more". It can also read "Release 2.4.0 is inside its bands." (no judged behaviour is outside its band) or "No baseline yet." The links are View release and Release trend. A failed read shows an error and Retry, never "inside its bands".
• Traces: when the filter names one release, the strip above the list gains a tile Behaviour, for example "4 of 41 outside bands". It links to the release page. A release with no verdict reads No baseline or Below minimum.
• Releases: the chips of the Trend and the Table. See Releases.
• The release page: all of it. See The release page.

Sensitivity

Sensitivity is a project setting in Settings → General, in the card Behaviour monitoring. The select has three choices. Save sensitivity applies the choice. It changes the width of the bands and the minimum number of traces, as the table above shows. An alert rule with the sensitivity Project setting reads it too.

Owners and admins can change it. Other roles read it as text. The release page shows it as Sensitivity: normal, read only, with a link to Settings.

The Behaviour drift alert

The metric Behaviour drift is on the Alerts page. Its value is the largest band ratio over the counters that the rule names, shown as 3.0 x band. A value above 1 means that at least one counter is outside its band.

The rule tests the current release. That is the release with the latest first seen time that has a settled trace in the window of the rule, or the release named by a release: term in the filter of the rule. When it cannot judge, the row has no value and says why, for example "no value (the release or its reference has too few traces for a band)". The fields of a rule that are not special to this metric are in Alerts.

Next Steps