Scorers

Built-in and custom scoring for LLM evaluation.

Risicare provides two ways to score your traces:

Built-in scorers — 13 pre-configured LLM-based evaluators that run server-side when you trigger an evaluation
Custom scores — Use risicare.score() to record any metric from your own code

Custom Scores with `risicare.score()`

The simplest way to add scores to your traces. No extra packages needed — it's built into the SDK you already have.

import risicare
 
risicare.init(api_key="rsk-your-api-key")
 
# Score a trace with any custom metric
risicare.score(
    trace_id="trace-abc123",
    name="sql_valid",
    value=1.0,
    comment="Query executed without errors"
)

JavaScript / TypeScript:

import { init, score } from 'risicare';
 
init({ apiKey: 'rsk-your-api-key' });
 
score('trace-abc123', 'sql_valid', 1.0, {
    comment: 'Query executed without errors',
});

Scoring Inside a Trace

import risicare
 
@risicare.trace
def my_pipeline(query):
    result = llm.invoke(query)
 
    # Score this trace based on custom logic
    trace_id = risicare.get_current_trace_id()
    if trace_id:
        is_valid = validate_output(result)
        risicare.score(
            trace_id=trace_id,
            name="output_valid",
            value=1.0 if is_valid else 0.0
        )
 
    return result

Parameters

Parameter	Type	Required	Default	Description
`trace_id`	str	Yes	—	The trace to score
`name`	str	Yes	—	Score name (e.g., "accuracy", "user_satisfaction")
`value`	float	Yes	—	Score value
`span_id`	str	No	`null`	Specific span within the trace
`comment`	str	No	`null`	Human-readable explanation

Scoring via REST API

You can also create scores via HTTP:

curl -X POST "https://app.risicare.ai/api/v1/scores" \
  -H "Authorization: Bearer rsk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "trace_id": "trace-abc123",
    "name": "accuracy",
    "score": 0.95,
    "comment": "Response matched expected output",
    "source": "api"
  }'

Built-in Scorers

Risicare Evaluations dashboard showing 20 runs, 13 available scorers across RAG/Safety/Agent/General categories, and completed evaluation results

When you create an evaluation via the API or dashboard, you specify which scorers to run using the criteria field. The Risicare server runs these scorers automatically — you don't need to install any extra packages.

Triggering Built-in Scorers

curl -X POST "https://app.risicare.ai/api/v1/evaluations" \
  -H "Authorization: Bearer rsk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Quality check",
    "evaluation_type": "llm_judge",
    "trace_ids": ["trace-abc123"],
    "criteria": ["faithfulness", "toxicity"]
  }'

Or from the dashboard: Evaluations → New Evaluation, select traces, and choose scorers.

Server-side execution

Built-in scorers run on the Risicare server using LLM-as-judge. You don't need to install any additional packages or provide your own LLM API key for built-in scorers. Evaluations are queued (HTTP 202) and processed asynchronously by a worker.

No built-in scorer has produced a result yet

All 13 scorers below are implemented and registered as active built-ins. None of them has ever produced a score: on a full prod-parity corpus the evaluations, scorer_runs, scorer_results and evaluation_results tables are all empty. Treat the built-in scorers as available but unverified — the descriptions below state what each scorer is written to measure, not a measured accuracy, and no scorer's output has been checked against a reference.

The custom-score path above (risicare.score() / POST /api/v1/scores) is separate and does write rows.

Scorers requiring only the output text

These need nothing beyond the model output already on the trace:

Scorer	Category	What it is written to evaluate	Score direction
`toxicity`	Safety	Is the content toxic, harmful, or offensive?	Lower is better
`bias`	Safety	Does the output show demographic or cultural bias?	Lower is better
`pii_leakage`	Safety	Does the output leak personal identifiable information?	Lower is better
`factuality`	General	Are factual claims in the output accurate?	Higher is better
`g_eval`	General	Configurable framework; grading criteria come from scorer config	Higher is better
`tool_correctness`	Agent	Were the right tools used with correct parameters?	Higher is better

tool_correctness declares no required fields at all.

Scorers requiring extra fields

These read fields that standard trace data does not carry. Supply them in the evaluation payload or the scorer will not have its inputs:

Scorer	Category	Required fields	What it is written to evaluate
`faithfulness`	RAG	`answer`, `contexts`	Is the answer grounded in the provided context?
`hallucination`	RAG	`answer`, `contexts`	Does the answer contain fabricated claims?
`answer_relevancy`	RAG	`question`, `answer`	Does the answer address the question?
`context_precision`	RAG	`question`, `contexts`	Is the retrieved context relevant?
`context_recall`	RAG	`contexts`, `ground_truth`	Compares retrieval against a reference answer
`task_completion`	Agent	`task_description`, `output_text`	Did the agent complete the requested task?
`goal_accuracy`	Agent	`goal`, `output_text`	Did the agent achieve a specific goal?