Evaluations
Run a dataset through your agent, read a verdict against a baseline, and stop a release in CI when it gets worse.
An evaluation run sends the cases of a dataset through your function and checks each answer. Risicare stores the run. It gives a pass rate, a verdict against a baseline, and an exit code that your CI can act on.
Runs start in your code. The dashboard reads them.
Your function and your checks run in your process. Risicare runs no model for them and holds no key of your model provider. The Runs page has no button that starts a run. The one exception is a hosted comparison of two saved prompt versions.
The loop
Make a dataset
A dataset is a set of cases. Make cases from traces with Add to dataset, or copy them from a run that got worse. See Datasets.
Run it from your code
evals.run (npm: runDataset) or the pytest plugin runs each case for a number of trials. Each trial is one trace. See Run from code.
Read the run
The run page shows the pass rate with its interval, how often a case passes every trial, and the cases that flip. See Runs and verdicts.
Compare with a baseline
Choose an earlier run or a release. The verdict is Regressed, Improved, No change or No verdict. A change inside the noise is not called a regression.
Gate the release in CI
risicare eval gate reads the verdict and exits with a code. See The gate.
Where each part is in the dashboard
| Part | Where | Page |
|---|---|---|
| Runs, verdicts, the case diff | Releases, then the Runs tab | Runs and verdicts |
| Datasets and cases | Datasets in the side navigation | Datasets |
| One score name, its pass rule, the labelling queue | The page /scores/{name} | Scores and labels |
| The trial traces of a run | Traces, with experiment:any in the filter | Run from code |
The side navigation has no Evaluations item. Releases covers both tabs, Releases and Runs. The old Evaluations address of a project redirects to Runs.
Which key a run needs
A run needs a key of the secret class. An ingest key sends spans and scores, and it has no evaluation or dataset scope. The api answers 403 to an ingest key, and the helper raises a RisicareEvalError that names the scope that is missing.
| Class | Can run a dataset | Why |
|---|---|---|
secret | Yes | It holds evaluations:read, evaluations:write and datasets:read |
ingest | No | It holds none of those three |
The helper uses these scopes: ingest:write, evaluations:read, evaluations:write, datasets:read and scores:write. Use a key of its own for CI, and rotate it if it leaks. Set it in the environment as RISICARE_API_KEY. See API Keys.
An API key can reach these routes for evaluations:
- Datasets:
GETandPOST /v1/datasets,GET /v1/datasets/{dataset_id}/exportandPOST /v1/datasets/{dataset_id}/items. - A run:
POST /v1/evaluations,GETandPATCH /v1/evaluations/{evaluation_id}, andGET /v1/evaluations/{evaluation_id}/inputs,/verdict,/casesand/results. - Writes:
POST /v1/evaluations/{evaluation_id}/resultsandPOST /v1/evaluations/{evaluation_id}/cases/copy. - Scores:
POST /v1/scores.
So a key can create a dataset, add cases, and copy the cases of a run into a dataset. Deleting runs, and changing or deleting a dataset, need a signed-in person in the dashboard. No REST route lists runs for a key. The MCP server lists runs and dataset versions, and reads a dataset with its cases, with a secret key. See the REST API reference.
Versions
The run helper and the gate are in Python 0.11.0 and npm 0.14.0 and later. The pytest plugin is in Python only. See the Changelog.