Run from Code

The run helper is from risicare import evals in Python and import { runDataset, defineCheck } from 'risicare/evals' in npm. Both are in Python 0.11.0 and npm 0.14.0 and later. The helper runs your function and your checks in your process. It runs no model and no judge of its own, and it holds no key of your model provider. It calls only the api host and the ingest host that init() or the environment names. It sends no telemetry about you.

A run needs a secret key. Call init(api_key=...), or set RISICARE_API_KEY. Tracing must be on, because each trial needs a trace. If you did not call init(), the helper does it. A sample_rate below 1.0 is refused.

Run a dataset

A repeat of an external_id with the same dataset, version, split and trials gives the first run back. The helper goes on with it if it is still running. If it is closed, or the key belongs to a run with other settings, the helper stops before it runs a trial.

A Case has id, input, expected, metadata, failure_class (npm: failureClass) and split. It does not hold the answer of the source trace.

Checks

A check is a named measure of one trial: evals.Check(name, fn, ...) in Python and defineCheck({ name, fn, ... }) in npm. fn(case, output) returns a boolean, a number, a string, or evals.CheckResult(value, comment, failed) (npm: { value, comment, failed }). It may be async. A Python fn that takes a third argument gets the trace id.

• A boolean check passes when its value is true. A categorical check never decides and cannot be required. A required numeric check needs a threshold.
• A trial passes when every required check passes. An optional check posts a score and decides nothing.
• A function that raises, or a required check that raises, gives a failed result. It is no pass and no fail. Risicare counts it and keeps it out of every rate. A failed optional check posts a score with judge_failed and no value, and the trial keeps its decision.
• The pass rule is the sorted required checks, for example all of: correct is true; tone >= 0.7. One run has one pass rule. Two runs compare only when their rules are the same. Python writes a number with repr, so write a whole threshold as an int (threshold=1) to get the same text as npm.
• A rule that you break is a ValueError (npm: TypeError) before any request.

What each trial records

A trial runs in an empty context, so it is a new trace even when you call the helper inside a span.

• The trace: the root span is risicare.eval.trial. Every span of the trace has risicare.experiment.id, the run id. With release, it also has risicare.release. The root span has risicare.experiment.case_id and risicare.experiment.trial. The helper writes no other attribute. It never prints or sends the case input or your output. The spans hold them under the content switch of the project.
• The scores: one score for each check and trial, with experiment_id, trial and the judge fields. See Scores and labels.
• The result: the case, the trial, the trace id, and pass, fail or error. The helper posts results in batches of 50, then closes the run as completed.

run.summary holds the fields of the run and its numbers, for example pass_rate, pass_all, flip and trials_failed. The helper computes none of them. score_post_failed (npm: scorePostFailed) counts check scores that could not be posted. run.results holds one record for each trial. run.id is the run id.

Bind a run to a prompt

Pass a prompt from get_prompt (npm: getPrompt) as prompt. The helper sends the name, the version and the hash. It runs each trial inside prompt.context() (npm: prompt.run()). The api binds the run to the stored version when the hash equals the content hash of that version. For any declared prompt, the run page says "Declared prompt support@4. Seen on X of Y trial traces."

If the hash differs, the api refuses with 422 prompt_hash_mismatch and no trial runs. The prompt changed after the pull, or the process runs a fallback or an old cached copy. Pull the prompt again. A hash that is not a SHA-256 gives prompt_hash_invalid and stops the run too.

The pair prompt=("support", "4") (npm: prompt: { name: 'support', version: '4' }) sends no hash. The run records the pair and binds to nothing. See Use a Prompt in Code.

The state file, cancel and failure

The helper writes risicare-run.json in the current directory, or at the path in RISICARE_EVAL_STATE. It holds run_id, api_endpoint, project_id and release, and no key. The gate reads it.

• Python: Ctrl+C closes the run as cancelled and raises again. Any other exception closes it as failed and raises again.
• npm: the signal option, or SIGINT, closes the run as cancelled. runDataset rejects with a RisicareEvalError with the code cancelled. Any other error closes the run as failed.
• In the dashboard: a run that is still running after 24 hours with no change gets Mark as cancelled.

A run closes once. A run holds at most 10,000 cases and 50,000 results. A post that meets a connection error or a 5xx is tried again three times, after 1, 2 and 4 seconds. A 4xx raises RisicareEvalError at once.

The pytest plugin (Python)

from risicare import evals
 
def test_case(risicare_case, risicare_eval):
    res = risicare_eval.run_case(
        risicare_case, my_agent, [evals.Check("correct", exact_match)]
    )
    assert res.pass_fraction >= 0.5     # or: res.assert_passed_all()

pytest --risicare-dataset support-regressions --risicare-trials 4 --risicare-release 2.4.1

The plugin is installed with the package. It does nothing unless --risicare-dataset or RISICARE_EVAL_DATASET is set.

The plugin creates the run when the session starts. A test with the argument risicare_case gets one call for each case, and the case id is the test id. The session fixture risicare_eval has run_case(case, function, checks), which runs every trial of one case. It returns a result with trials, pass_fraction (None when no trial was decided), passed_all, flipped, failed_trials and assert_passed_all(). Your test decides what to assert, and pytest decides its exit code.

• The plugin closes the run at the end as completed, also when tests fail. An interrupted session closes it as cancelled. It prints the line "Risicare run completed" with the run page, and it writes risicare-run.json.
• The run has no pass rule when it starts. The rule of the first checks that a test gives is used. Use the same required checks in every test.
• With --collect-only, the plugin lists the dev cases from the export and makes no run.
• pytest-xdist is not supported. Turn the plugin off with -p no:risicare.

npm has no plugin for vitest or jest. Make one run in beforeAll and assert on run.results.

Find the trial traces

The trace list hides trial traces. The note under the filter bar reads "Experiment traces are hidden. Add experiment:any to see them." Add experiment:any for all of them, or experiment: and a run id for one run. Risicare does not count them in release numbers. A trial trace shows the chip Experiment run, with the trial number, that opens the run page. See Filtering and Search.

Next Steps