The Gate

risicare eval gate reads the verdict of one run with one call and exits with a code. It has no statistics code and rewrites no code. The api computes the verdict, the state of each criterion and the exit code. The command prints the answer and exits with that code. It is in Python 0.11.0 and npm 0.14.0 and later.

Install and call

• Python: pip install risicare adds the console script risicare. Run risicare eval gate.
• npm: npm install risicare adds the command risicare. Run npx risicare eval gate.

The command needs a key that can read evaluations, which is a secret key. Set it as RISICARE_API_KEY. RISICARE_ENDPOINT and RISICARE_API_ENDPOINT name the hosts, if you do not use the defaults.

risicare eval gate [--run ID] [--baseline RUN_ID|release:LABEL] [--max-drop X]
                   [--min-pass-rate T] [--min-pass-all T] [--require-holdout]
                   [--strict] [--allow-no-verdict] [--json]

Options

A number outside 0 to 1 is a usage error. The command prints the lines Run, Baseline, Pass rate with its 95 percent interval, Pass^k, Cases, Trials, Noise band, Verdict, one line for each criterion, Holdout, Gate and Exit code. These are the numbers of the run page.

How a criterion is decided

Each criterion has a state, from the 95 percent interval of the number that it tests.

The gate blocks on proof. Only missed exits with 100. An unclear criterion prints a line and exits with 0, unless you set --strict. A baseline compare is missed only when the run is provably worse than the baseline by more than --max-drop.

A no-verdict never passes, unless you set --allow-no-verdict. Under --strict, an unclear criterion stays at 101 even with that option. A verdict can fail to exist for the reasons on the run page: fewer than 20 cases compared, more than 10 percent of the trials failed or missing, different pass rules, a run that is not closed, and the rest.

With none of --baseline, --min-pass-rate, --min-pass-all and --require-holdout, the api passes a closed run and answers 101 with a reason otherwise, for example that the run is not finished.

Exit codes

The exit code of the process is always the code in the answer. The printed lines and the --json object give the same number. The verdict card of a run shows the code that the gate would give for its baseline alone, for example "The gate passes. risicare eval gate would exit with code 0."

A CI example

The run step writes risicare-run.json. Run the gate in the same directory, or pass --run with the run id. Keep the key in the secret store of your CI and read it from the environment.

name: eval
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    env:
      RISICARE_API_KEY: ${{ secrets.RISICARE_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install "risicare>=0.11.0"
      - name: Run the dataset
        run: python evals/run_support.py   # calls evals.run(..., release="2.4.1")
      - name: Gate
        run: risicare eval gate --baseline release:2.3.0 --min-pass-rate 0.6

The step fails on any code other than 0. To treat the codes in different ways, read the code in a shell step:

set +e
risicare eval gate --baseline release:2.3.0 --max-drop 0.02
code=$?
case "$code" in
  0)   echo "gate passed" ;;
  100) echo "a criterion is missed with proof"; exit 1 ;;
  101) echo "no verdict"; exit 1 ;;
  *)   echo "the gate could not decide (code $code)"; exit 1 ;;
esac

Next Steps