Scorers

Risicare provides two ways to score your traces:

1. Built-in scorers — 13 pre-configured LLM-based evaluators that run server-side when you trigger an evaluation
2. Custom scores — Use risicare.score() to record any metric from your own code

Custom Scores with risicare.score()

The simplest way to add scores to your traces. No extra packages needed — it's built into the SDK you already have.

import risicare
 
risicare.init(api_key="rsk-your-api-key")
 
# Score a trace with any custom metric
risicare.score(
    trace_id="4bf92f3577b34da6a3ce929d0e0e4736",
    name="sql_valid",
    value=1.0,
    comment="Query executed without errors"
)

JavaScript / TypeScript:

import { init, score } from 'risicare';
 
init({ apiKey: 'rsk-your-api-key' });
 
score('4bf92f3577b34da6a3ce929d0e0e4736', 'sql_valid', 1.0, {
    comment: 'Query executed without errors',
});

Scoring Inside a Trace

import risicare
 
@risicare.trace
def my_pipeline(query):
    result = llm.invoke(query)
 
    # Score this trace based on custom logic
    trace_id = risicare.get_current_trace_id()
    if trace_id:
        is_valid = validate_output(result)
        risicare.score(
            trace_id=trace_id,
            name="output_valid",
            value=1.0 if is_valid else 0.0
        )
 
    return result

Parameters

score() reports a failed request: an error, a timeout or a status of 300 or more adds 1 to failed_scores (Python) or failedScores (JavaScript), and the next flush() returns False. The first failure logs a WARNING with the score name and the reason; failures in the next 10 seconds are counted, and the next WARNING gives their number.

Scoring via REST API

You can also create scores via HTTP. The trace_id must be 32 lowercase hex characters:

curl -X POST "https://api.risicare.ai/v1/scores" \
  -H "Authorization: Bearer rsk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
    "name": "accuracy",
    "score": 0.95,
    "comment": "Response matched expected output",
    "source": "api"
  }'

Built-in Scorers

An illustration of the Evaluations page as designed. It is not a record of production results: no evaluation has run in this release.

As designed, when you create an evaluation, you specify which scorers to run using the criteria field. The Risicare server runs these scorers — you don't need to install any extra packages.

Triggering Built-in Scorers

Scorers requiring only the output text

These need nothing beyond the model output already on the trace:

tool_correctness declares no required fields at all.

Scorers requiring extra fields

These read fields that standard trace data does not carry. Supply them in the evaluation payload or the scorer will not have its inputs:

Next Steps