Charts
Build a chart from a question or from a blank sentence. Pick a measure, an axis, a group and a filter, read the preview and the table, and start an alert from a chart.
A chart answers one question about your traces, your runs or your releases. The editor writes one sentence from your choices, for example "Show Error rate by model over the last 7 days, counted per trace, for all traffic." A member or higher adds and changes charts on a dashboard.
Add a chart
Open the picker
On a dashboard, click Add a chart. The picker is titled "Add a chart to (name)". New dashboard opens the same picker with the title "What do you want to know?".
Pick a question
Click a question. A question that asks for a dataset or a score has a select on its row. A question that needs data you do not have shows the reason and you cannot pick it. Build a chart from a blank sentence starts a chart of Traces on the Time axis.
Change the chart
The editor opens with the chart drawn. Change the fields, then click Add to the dashboard. When you change a saved chart, the button reads Save the chart.
| Group of the picker | Questions |
|---|---|
| Change from release to release | "How did cost change by release?" · "How did the time per run change by release?" · "Did the behaviour of the agent change by release?" · "Did the pass rate change by release?" · "How did the error rate change by release?" · "Did the mean of a score change by release?" |
| Change across runs of a dataset | "Did the pass rate of a dataset move across runs?" |
| Change over time | "How many traces failed each day?" · "How did cost change each day?" · "How slow were the traces each day?" · "Did the pass rate of a score move over time?" · "How many traces came in each day?" · "How many tokens did the agents use each day?" |
| Which one is the worst | "Which model fails most?" · "Which release costs most?" · "Which user costs most?" · "Which agent is slowest?" · "Which tool fails most?" · "Which error class is most common?" · "Which model costs most?" · "Which agent fails most?" · "Which model is slowest?" · "Which user sends the most traces?" |
A question that cannot be picked says why:
- "Needs a release on your traces. No trace in the last 90 days has a release." The six release questions need two releases that have a settled trace in the last 90 days.
- "Needs a dataset with two completed runs." The run question needs it. "Did the pass rate change by release?" needs a dataset with one completed run, although the picker words that case with the same message. The picker counts the completed runs of the split dev.
- "Needs a score with a pass rule. Set a rule on the Scores page." Two questions need it: "Did the pass rate of a score move over time?" and "Did the mean of a score change by release?" In this release the picker counts only a true or false score, which passes when it is true. A score with a number or a label does not unlock these two questions, also when you apply Pass at or Pass labels on its Scores page, because that rule stays in the address of that page. To chart a number score, build the chart from a blank sentence, pick Mean of a score or Pass rate of a score, and type the Pass rule. See Scores and Labels.
The editor
The left pane draws the chart on your data 400 ms after your last change. The table of numbers is always open under it. The right pane starts with the sentence, then the fields in the order of the sentence. A field shows only when the measure and the axis allow it. A dashboard holds 12 charts. At 12, the button is off and says so.
| Field | What it takes |
|---|---|
| Measure | One of the measures below. A line under the select gives its definition. The measure sets the unit, so no unit select exists. A rate shows two lines, Counted and Out of |
| Traces that count | Share of traces only. A filter of at most 16 terms, with no free word and no experiment term. The share is never above 100 % |
| Score, Pass rule | The score measures. A name from the list, or a name that you type. Pass rule is for Pass rate of a score, for example score.reward>=0.5 |
| Axis | Time, Release, Run of a dataset or No axis: one total for each group |
| Group by | Time and no axis only. See Group by |
| Filter | The trace filter. Example: env:production |
| Window | Time and no axis. The dashboard window or Last 24 hours, Last 7 days, Last 30 days, Last 90 days |
| Releases | Release axis. Newest 5, Newest 10 or Newest 20 |
| Dataset, Split, Runs | Pass rate (runs on a dataset). A dataset, the split dev or holdout, and on the run axis Newest 5, Newest 10 or Newest 20 |
| Grain | Time axis. Auto (per hour up to 2 days, per day above), Per hour, Per day or Per week |
| Shape | See Shapes |
| Title, Description | A title is required and has at most 120 characters. A description has at most 500 |
Press Ctrl or Cmd with Enter to save. Press Escape to leave. If you changed the chart, the dialog asks "Leave without saving? Your changes to this chart are lost." with Stay and Leave. If someone changed the dashboard while you worked, the editor says "Someone changed this dashboard while you edited this chart. Your chart was not saved." Click Reload, then add the chart again.
Measures
The select groups the measures under Counts, Rates, Cost, Time, Scores and Runs.
| Measure | Axes | What it reads |
|---|---|---|
| Traces | Time, Release, No axis | The number of traces of your agent. Runs of your code are not counted |
| Tokens | Time, No axis | The input and output tokens of the LLM calls |
| Behaviour | Release | How many behaviour counters of the release are outside their bands |
| Detections | Release | Traces of the release with stored evidence that a person can open |
| Error rate | Time, Release, No axis | Traces with a failed span ÷ all traces. A trace counts once |
| LLM error rate, Tool error rate | Time, No axis | Traces with a failed LLM call (or tool call) ÷ all traces |
| Failed LLM calls | Time, No axis | Failed LLM calls ÷ all LLM calls, counted per call |
| Share of traces | Time, Release, No axis | The traces that also match Traces that count ÷ all traces of the chart |
| Cost | Time, Release, No axis | The cost of the priced traces, in US dollars. Traces with no price are counted apart, never as $0 |
| Cost per run | Time, Release, No axis | The mean cost of one priced trace |
| Time per run | Time, Release | The mean wall-clock time of one trace |
| Latency p50, p95, p99 | Time, No axis | Half, 95 of 100 or 99 of 100 of the traces end in this time or less |
| Mean of a score | Time, Release, No axis | The mean of the newest value of the score on each trace |
| Pass rate of a score | Time, Release, No axis | Traces that pass the rule ÷ traces with a value of the score |
| Pass rate (runs on a dataset) | Run, Release | The mean share of passing trials over the cases of a completed run |
An axis that a measure cannot use is greyed in Axis, and a line says why, for example "Behaviour has no time read." If you pick a measure that does not fit the axis, the axis changes and a line says so. Latency p99 on No axis needs a group.
Group by
The choices are Model, Provider, Agent, Environment, User, Tag, Release, Failure class, Token type, Error class, Root name, Tool and Experiment run.
- A release or run axis takes no group. The editor says "A release or run axis takes no group."
- Time per run takes no group on the time axis. Cost and Cost per run take no Tool group, because a tool has no cost. The editor removes a group that cannot be read and says why.
- Token type is for the cost measures. Failure class is for the score measures on No axis, and it needs a score. Failed LLM calls takes Model, Provider and Agent only.
- A trace with two models counts in both rows. The same holds for providers, agents, tags, error classes and tools. Experiment run counts only traces of runs of your code.
- A ranked-bars chart shows up to 10 groups. The rest is one row, Other (3 groups). For a rate, a group with fewer than 30 traces shows no rate and ranks after the groups that have one.
Shapes
| Shape | When |
|---|---|
| Line | Time, release or run axis, with no group. On a release or run axis, a measure with a band reads Line with bands: Cost per run, Time per run and Pass rate (runs on a dataset). The band comes from the release or the run before |
| Small charts, one for each group | Time axis with a group. One chart for each of the top 6 groups, on one scale. The table has all groups |
| Ranked bars | No axis with a group, highest first |
| One figure | No axis with no group |
| Status strip | Behaviour on the release axis |
| Table only | Every axis |
Read a chart
A card shows the title, the sentence, a headline figure, the chart, the reasons under the chart, trace links, the line "Read at 14:02 UTC" and what the chart counts. Read again on the dashboard reads every chart again. A chart that fails shows its words and Retry, and the other cards stay.
- A value that is not known has a reason in words. The card never shows a bare dash or $0.00. A real zero is counted: "0.0 % · 0 of 461 traces".
- A rate needs 30 traces. With fewer, the card shows the counts and no rate: "3 of 12 traces had a failed span. Too few for a rate: fewer than 30 traces."
- Buckets are in UTC. Per week starts on Monday. A line stops at a bucket with no trace. The read refuses Per hour for a window longer than 14 days. For Cost and Cost per run, the editor turns Per hour off in that case. For other measures the option stays, and the card shows the error of the read. Per week is off for Cost or Tokens by a group.
- On a release axis, Cost per run, Time per run and Behaviour measure each release over its own last 7 days. Pass rate (runs on a dataset) shows the newest completed run of each release. The window does not apply to these. Other measures read the window, and a release with no trace there shows "Not in the window". A run axis shows completed runs only, oldest on the left.
These keys work on a chart. Tab moves to the next chart. Arrows move the point or the row, and Home and End go to the first and the last. Enter opens the first link of the point. T shows or hides the table. E opens the editor. On a dashboard, press ? outside a field to read this list.
The table and the CSV
Show the numbers as a table opens the table, and Hide the table closes it. It has a column for the measure and a column Traces with the links of each row. Download CSV under the table saves the rows on screen, for example Cost 7d.csv. Your browser makes the file. A missing value is an empty cell, and the reason is in the next column. A cell that starts with =, +, - or @ gets a leading apostrophe, so a spreadsheet does not run it as a formula.
Links to the trace list
Point at a bucket, a release or a bar to see its links, for example "Open the 25 traces" or "Open the 3 traces with a failed span". Each link states its count. It opens Traces with the window of the bucket and the filter of the dashboard, the filter of the chart, the term of the group and, for a link to failed traces, a term such as error:true. The row Other and a row with no value have no link and say why.
The menu of a chart
A person who can change the dashboard sees the menu Menu of (title). Its items are Edit, Duplicate, Move up, Move down, Make wide (or Make narrow), Copy to another dashboard…, Create an alert and Remove. Copy to another dashboard… copies the chart to a dashboard that you can change, or to New dashboard…. The filter of the source dashboard becomes part of the copy, and the dialog says so.
Create an alert and Alert on this group open the Alerts page with the rule form filled in. A member or higher sees them. Alert on this group is under each group row of a ranked-bars chart whose measure has an alert metric. It adds the term of the group to the filter. The window of the rule is 60 minutes for an hourly chart and 1,440 for a daily one. The form suggests a threshold from the last 24 hours.
| Measure | Alert metric |
|---|---|
| Traces | Trace volume |
| Error rate, LLM error rate, Tool error rate, Cost, Latency p50, p95, p99 | The metric of the same name |
| Tokens | Total tokens |
| Mean of a score | Score mean, for a chart with a score |
| Pass rate (runs on a dataset) | Run verdict, for a chart on the run axis with a dataset |
| Behaviour | Behaviour drift, for a chart on the release axis |
The traffic metrics need the time axis or No axis. These measures have no alert metric, so their menu has no such item:
- Failed LLM calls
- Share of traces
- Cost per run
- Time per run
- Pass rate of a score
- Detections