Skip to main content

Import Cases

Add cases to a dataset from a CSV or JSONL file, or from pasted text. The file is checked first, and nothing is written while one row has an error.

Use Import cases when your cases are already in a file: a CSV from a spreadsheet, a JSONL file from another tool, or the file that Export made with the format Risicare. For one case from a trace, use Add to dataset.

An import is all or nothing. The dialog checks the whole file first. While one row has an error, nothing is written. An import that adds cases makes one new version of the dataset, for all of them.

Open the dialog

  1. Open a dataset from Datasets in the side navigation.
  2. Click Import cases in the header of the dataset, next to Export.

A member, an admin or an owner can import. A viewer sees the button turned off, with the reason beside it. A past version of a dataset is read only and has no Import cases. Open the latest version.

The top of the dialog says the limits and the room of the dataset, for example "One import takes at most 1,000 rows and 8 MiB of UTF-8 text (CSV or JSONL). This dataset holds 9,500 cases of 10,000, so an import can add at most 500 more." See Limits.

The file

Select Choose a file and pick a .csv or .jsonl file. Or select Paste the text, paste the text of the file, and click Use this text. The text must be UTF-8. A byte order mark is removed. A line end can be LF or CRLF.

The dialog finds the Format from the file name and the first line. You can change it. There are three formats.

FormatWhat the file holds
CSVA header row, then one case on each row
JSONLOne JSON object on each line
Risicare exportThe JSONL file that Export makes with the format Risicare

CSV

The first row is the header. Each next row is one case. A cell can hold a comma, a quote written twice and a line break, when the cell is in quotes (RFC 4180). A file has at most 64 columns, and no column name is empty or used twice. Every cell is stored as text, as it is: a cell with =SUM(A1) stays that text.

question,expected_answer,external_id
What is the capital of France?,Paris,geo-001
What is 2 + 2?,4,math-001

JSONL

Each line is one JSON object. Only a line feed ends a line. A line of spaces, tabs and carriage returns only is blank, and it is skipped. A value can be any JSON value: text, a number or an object. An external id must be text or a whole number.

{"input": "What is the capital of France?", "expected_output": "Paris", "external_id": "geo-001"}
{"input": "What is 2 + 2?", "expected_output": "4", "external_id": "math-001"}

A .json file that holds one array, from [ to ], is not read. Write each case on its own line.

Risicare export

You do not write this file. It is the file that Export makes with the format Risicare. No mapping is needed. Each line keeps its input, its expected answer, its failure class, its metadata and its source trace, and its evidence when the line has it. The split, the dataset id, the dataset version and the creation time of a line are not used: Risicare picks the split of each new case again. See The holdout.

{"case_id": "00000000-0000-4000-8000-000000000001", "input": "What is 2 + 2?", "expected_status": "set", "expected_output": "4", "failure_class": null, "source": {"trace_id": null, "span_id": null}, "metadata": {}}

A line has the keys of an export only. A line with another key is an error. A line needs case_id (a UUID), input and expected_status (set or pending).

The columns

For CSV and JSONL, the dialog shows the columns of the first rows with their first value. Choose a Use for each column:

UseWhat it does
InputThe input of the case. Choose one column, or several: then the input of each case is an object with one key for each column. You must choose at least one
Expected answerThe expected answer of the case. One column. A row with an empty cell (CSV), or with no value or a null value (JSONL), makes a Pending case
External idThe key of a case in your file, at most 200 characters. One column. A row with an empty value is an error. See What is left out
MetadataThe value is kept with the case under the name of the column. The names risicare and failure_class cannot be used
Not importedThe column is left out

The dialog proposes a use from common column names. For example, question, prompt and query are proposed as the input, and expected_output, expected_answer and ground_truth as the expected answer. A column that no name matches is Not imported until you choose a use. The dialog does not propose a column named answer or output as the expected answer, because it often holds what a system answered. Choose it yourself when it holds the expected answer.

A case with no expected answer is added as Pending. A run runs only the cases that have an expected answer, so set the answer in the case editor first. See Cases.

Check, then import

Choose the file and the columns

Choose a file or paste the text. Check the Format and the use of each column.

Click Check the file

The server reads the whole file and writes nothing. The result says how many rows are valid, for example "All 212 rows are valid." It says how many cases the import will add, "as one new version". It lists the rows that will be left out, and the new cases that have no expected answer.

Click Import N cases

The button is on only after a check with no error of the same file, format and columns. If you change one of them, check again. The write carries a key that the dialog makes for each checked file, so a double click or a retry never imports twice.

Read the result

The dialog says, for example, "Version 7: 212 cases added from cases.csv." The page shows the new cases and the new latest version.

If a row has an error, the check lists the first 50 errors in the words of the server, for example Row 7: the input column "question" is empty. When there are more, the dialog says "Showing the first 50 of 312 errors." Nothing is imported while any row has an error. Fix the rows in the file, then choose it again. A row is counted as in your tool: in a CSV file the header is row 1 and the first case is row 2, and in a JSONL file the first line is line 1.

If the answer to the import is lost (the connection fails or the server is slow), the import may or may not be written. Close the dialog and open Version N, next to the case count. An import shows in that list as "imported N cases". Trying again is safe: the dialog sends the same key, and the server never writes the same import twice.

What is left out, and why

A row is left out, not refused, when the dataset already holds that case. So a second import of the same file adds nothing, and a retry does not make a copy. The key of a case is:

  • The External id column, when you chose one.
  • The input and the expected answer, when you chose no external id column. A row with the same input and the same expected answer is the same case. A row with the same input and another expected answer is a new case.
  • The case id or the source trace, for a Risicare export. So the export of a dataset, imported into the same dataset, adds nothing.

A row is also left out when an earlier row of the same file is the same case. The first row wins.

An import never changes a case that exists. To change a case, edit it in the case editor. The check tells you how many rows will be left out, for example "40 rows will be left out: the dataset already holds a case with the same external id."

Versions

An import makes one new version, for all the cases it adds. The Versions list shows it as "imported 180 cases from cases.csv", with no link to one case. A run pins one version, so a run that started before the import keeps its cases.

An import that adds no case makes no version. The dialog says "No case was added, so no version was made." See Versions.

The new cases follow the holdout rule of the dataset, as any new case does. When some of them went to the holdout, the dialog says how many.

Limits

LimitValue
Rows in one import1,000 (CSV rows, or JSONL lines). Blank lines do not count
Text of the file8 MiB of UTF-8 text in one import
The whole request10 MiB. The escapes of JSON count: each quote, backslash, tab and line break of the file takes 2 bytes when the file is sent. If the file is over the limit, split it into two imports
Cases in a dataset10,000 after the import, so that the dataset can always be exported again
Columns of a CSV file64
Input columns, and metadata columnsAt most 64 each
Name of a columnAt most 256 characters
External idAt most 200 characters
One input, expected answer or evidenceAt most 1,000,000 characters of JSON
Errors that the check listsThe first 50. The count is the true count
Requests to the import route on api.risicare.ai100 in 5 minutes for each client IP address. A dry run and an import each count as one request. A request over the limit gets 429. Callers that share one address share the limit

Call the api

The dashboard sends the import to POST /v1/datasets/{dataset_id}/imports. An API key can send the same request to https://api.risicare.ai. The route needs the scope datasets:write, which a member and above has. A secret key has the scope too. An ingest key does not have it. On the API host, only POST is available on this path. The body is JSON:

FieldMeaning
formatcsv, jsonl, risicare or cases
contentThe text of the file, at most 8 MiB. Required for csv, jsonl and risicare. Not allowed for cases
casesFor format cases only: an array of JSON objects, at most 1,000, read with mapping
mappinginput (a column, or a list of columns), expected_output, external_id and metadata (a key for each column). Not allowed for risicare. Without it, the input is the column input, and the expected answer is the column expected_output when the file has it
file_nameThe name of the file, for the Versions list. It is not read
idempotency_key1 to 200 visible ASCII characters
dry_runtrue does every check and count and writes nothing

Send a dry run first. A dry run answers 200 and does not read or keep the key. Then send the import with a key that is new for each file. The shell variables below hold a key with the scope datasets:write and the id of the dataset. Do not put a real key in a script that you share.

export RISICARE_API_KEY="<a key with the scope datasets:write>"
export DATASET_ID="<dataset id>"
export IMPORT_KEY="$(uuidgen)"   # a new key for each file
 
jq -n --rawfile content cases.csv '{
  format: "csv",
  content: $content,
  mapping: {input: "question", expected_output: "expected_answer", external_id: "external_id"},
  file_name: "cases.csv"
}' > import.json
 
# 1. Dry run. Nothing is written. The answer is 200.
jq '.dry_run = true' import.json \
  | curl -sS -X POST "https://api.risicare.ai/v1/datasets/$DATASET_ID/imports" \
      -H "Authorization: Bearer $RISICARE_API_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @-
 
# 2. Import. The answer is 201.
jq --arg key "$IMPORT_KEY" '.idempotency_key = $key' import.json \
  | curl -sS -X POST "https://api.risicare.ai/v1/datasets/$DATASET_ID/imports" \
      -H "Authorization: Bearer $RISICARE_API_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @-

A dry run and an import each count as one request toward the limit in Limits. If the answer is 429, wait, then send the import again with the same idempotency_key. The api never writes the same import twice.

The answer holds the counts of the import:

{
  "import_id": null,
  "dry_run": true,
  "replayed": false,
  "version": null,
  "rows": 212,
  "added": 200,
  "skipped_existing": 10,
  "skipped_repeat": 2,
  "pending": 0,
  "added_to_holdout": null,
  "invalid_rows": 0,
  "error_count": 0,
  "errors": []
}
  • version is the one version that the import made. It is null for a dry run and for an import that adds no case.
  • A repeat of an idempotency_key in the same dataset answers the first answer with "replayed": true (200) and writes nothing. The same key with another request is a 409 idempotency_key_reused.
  • One bad row is a 422 import_rows_invalid. The first 50 errors are in details.errors, each with row, field, code and message. A problem of the whole file is a 422 with details.reason, for example import_missing_column or import_too_many_rows. A dry run answers 200 with the errors in errors.
  • A dataset that would hold more than 10,000 cases is a 422 dataset_too_large. A dataset of another project is a 404.

What it does not do yet

  • A .json file that holds one array is not read. Write one object on each line. For code, the api format cases takes an array in the request body.
  • It does not update a case that exists. A row for a case that the dataset already holds is left out.
  • No SDK or CLI command imports cases. Use the dashboard, or call the api as shown in Call the api.
  • It reads UTF-8 CSV and JSONL only. A spreadsheet file and UTF-16 text are refused with a message. A tab-separated file gets a message too. Save the file as CSV UTF-8.
  • It cannot import into a past version. A past version is read only.

Next Steps