Skip to main content

Production & Failure Modes

What the SDK does inside your process, what it costs you when our backend is down, and what you lose.

This page describes what the Risicare SDK does inside your production process: what it costs your request path, what happens when our backend is unreachable, and exactly which spans you lose. Every number here was measured against the published packages — Python risicare 0.2.0 (with risicare-core 0.1.6) and JavaScript risicare 0.5.0 — not against internal builds. The published versions are now Python risicare 0.11.0 (with risicare-core 0.1.8) and JavaScript risicare 0.14.0, and not every figure on this page has been re-measured against them. A table or a measurement that names a version is a measurement of that version. The rules for delivery in the text are those of Python 0.11.0 and npm 0.14.0 (see the changelog).

If you are evaluating whether to put this library in a hot path, read the first two sections. They are the ones that decide it.

What the SDK does not do to your application

These are the guarantees. They hold whether our backend is healthy, degraded, or completely unreachable.

Tracing never raises into your code. An export failure is logged, never thrown. No span operation propagates an exception into your call frame. If our gateway returns 500 for an hour, your application does not see an error from us.

Your thread does not block on export. Spans are handed to an in-process queue and written by a background worker. Span creation cost is unchanged by backend health — measured over 20,000 spans on each arm:

Backend statep50p95p99
Healthy0.0141 ms0.0162 ms0.0236 ms
Returning 500 to every request0.0119 ms0.0140 ms0.0163 ms

Tens of microseconds, and the dead-backend arm is not slower. (In this measurement of 0.2.0 it is marginally faster, because once the queue is full, dropping the incoming span is cheaper than enqueuing one. Python 0.8.0 and npm 0.11.0 drop the oldest queued span instead, and that was not re-measured.) Treat these as an order of magnitude on your own hardware, not a guarantee of a specific figure.

Memory is bounded. The queue is capped at max_queue_size, default 10,000 spans. When it is full, the SDK drops the oldest queued span and keeps the new one. (Through Python 0.7.0 and npm 0.10.0, it dropped the new span.) The queue does not grow without limit during an outage.

A clean shutdown does not lose what it holds. shutdown() drains the queue within a budget of 10,000 ms by default (shutdown_timeout_s; npm: shutdownTimeoutS), reserving the last 100 ms to close the exporter. Through Python 0.7.0 and npm 0.10.0, the default was 5,000 ms. Anything still queued when that budget expires is counted and announced with a warning. A batch that waits for a retry gets one more attempt at once, and a failed attempt in the drain is not retried. docker stop waits 10 s by default, so set shutdown_timeout_s lower when your application needs time for its own cleanup.

What happens at exit

On a normal exit and on SIGTERM, both SDKs flush the queue by themselves. In Python an uncaught exception also flushes. A process that ends with os._exit() (Python), process.exit() (JavaScript) or SIGKILL loses what is in the queue, with no message, and so does an uncaught exception in JavaScript. Call risicare.shutdown() (Python) or await risicare.shutdown() (JS) before such an exit.

flush() returning true means delivered, also in a signal handler and after shutdown(). It is true only when the gateway acknowledged every span and score that the SDK accepted since the previous flush(). A loss, such as a span that ended after the SDK's own drain (shutdown_residue), makes one flush() false and raises a counter that names it (dropped_spans_by_reason, rejected_spans, failed_scores). With no API key (and, in Python, no exporter of your own), before init() and with tracing off, flush() is false. In Python, a span that you start after shutdown() has returned, and a call to a traced provider in your handler after the SDK's drain, make no span, and nothing counts them, so a true does not show that they were traced (JavaScript counts such a span as shutdown_residue). In JavaScript, register your own SIGTERM or SIGINT listener before init() if it makes spans that must be delivered. In Python 0.5.1 and earlier, and in npm 0.8.0 and earlier, a true in a signal handler or after shutdown() could mean that nothing was delivered.

What you lose when our backend is unreachable

This is the part that costs you data, and it is the reason this page exists.

The HTTP exporter keeps a batch through an outage for up to retry_horizon_s (npm: retryHorizonS). The default is 300 s, counted from the first attempt of the batch. It retries HTTP 408, 429, 500, 502, 503 and 504, a connection error and a timeout. A batch that the gateway does not take within the horizon is dropped (retry_exhausted). A gateway outage that is longer than the horizon loses spans. (Through Python 0.7.0 and npm 0.10.0, the exporter gave up after a few seconds.)

The exporter has a circuit breaker. After 5 consecutive failed export calls it stops attempting exports entirely for a 60-second cooldown. During that cooldown no request is made — even if the backend has already recovered.

In both SDKs, a batch refused for capacity — HTTP 429 or 503, a healthy gateway shedding load — does not count toward the breaker. The SDK retries it, honouring Retry-After when the response carries one; if the batch is still refused when the horizon ends, it is dropped (retry_exhausted), and exports continue.

Count failed calls, not seconds

An export call is one batch, not one span and not one HTTP request. A call can make many HTTP attempts. The wait before a retry is full-jitter exponential backoff (base 1 s, cap 30 s). A Retry-After header (seconds or an HTTP date) replaces the wait when it is more than 0 s. The horizon is the cap on that wait. get_metrics() (npm: getMetrics()) counts the attempts in attempts, and oldest_pending_age_s (npm: oldestPendingAgeS) shows how long the oldest pending span has waited.

The exporter sends every other answer once: 400, 401, 403, 404, 422 and the rest, a redirect, a 2xx that is not a Risicare acknowledgement, and an acknowledgement of 0 and 0. It drops such a batch as transport_refused. On 413 it splits the batch in two, one time. A half that gets 413 again is dropped (too_large).

A call counts toward the breaker when it ends refused, or when it used up its retries, except when its last answer was 429 or 503. With the default horizon, a dead gateway opens the breaker only after five batches in a row were retried for 300 s each. A gateway that answers 401 or 403 fails each call at once, with one request. Where the opening lands in wall-clock time is not fixed: it depends on your span rate and on how your endpoint fails. This is why the threshold is documented in calls rather than seconds — any second-based figure would be specific to one harness.

Measured on Python 0.2.0

Driving the published HttpExporter of Python 0.2.0 against a loopback sink that returned 500 to everything, then flipping that sink healthy. (That release made 3 HTTP attempts for each call. From Python 0.8.0 and npm 0.11.0 a call retries until the horizon, so these request counts differ.)

EventObserved
Export calls before the breaker opened5
HTTP requests that reached the sink15 (3 per call)
Requests reaching the sink once open0
Sink made healthy att = 0
First request to reach the healthy sinkt = 60.1 s

The sink was healthy and idle for the whole minute. The SDK did not probe it once. That was the cost of the breaker in 0.2.0: a brief outage billed you for the outage plus up to a further 60 seconds. With the default horizon of 0.8.0 and later, the breaker opens only after five batches in a row were each retried for 300 s. Then it still holds for up to a further 60 seconds.

A failed probe restarts the clock

When the cooldown expires the exporter allows one half-open export call. That call follows the retry rules above. (In npm 0.10.0 and earlier, it made one attempt with no retry.) If that call succeeds, the failure counter resets and normal export resumes. If it fails, the cooldown is re-armed for another full 60 seconds. A backend that is flapping rather than cleanly down can hold the circuit open indefinitely.

Which spans survive

When the queue is full, the SDK drops the oldest queued span (queue_full) and keeps the new one. This holds since Python 0.8.0 and npm 0.11.0. Through Python 0.7.0 and npm 0.10.0, the SDK dropped the incoming span and kept the oldest spans.

Measured on Python 0.2.0, which dropped the incoming span: creating 20,000 spans against a dead backend, then restoring it, delivered exactly 10,500 distinct spans, contiguously numbered from the first span created — 10,000 from the queue plus one in-flight batch of 500. Spans 10,501 onward were dropped. With that rule, a truncated trace showed its beginning, not its end. With the rule of 0.8.0 and later, the spans that the queue loses are the oldest ones.

Dropped spans are counted by reason (queue_full, transport_refused, retry_exhausted, too_large, quota_exceeded, shutdown_residue, unacknowledged). Read the counts with risicare.get_metrics() in Python (dropped_spans_by_reason) or getMetrics() in JavaScript (droppedSpansByReason).

Defaults that govern this

SettingPython defaultJS default
Circuit breaker threshold5 consecutive failed export calls5
Circuit breaker cooldown60 s60 s (6e4 ms)
Retry horizon (retry_horizon_s; npm: retryHorizonS)300 s300 s
Re-queue limit before discard, for a failure that the exporter does not name (for example a custom exporter)33
batch_size500500
batch_timeout_ms1,0001,000
max_queue_size10,00010,000
Export request timeout5 s for each attempt. The horizon is checked between attempts, so the last attempt can end up to one attempt timeout after it5 s for each attempt
Shutdown budget (shutdown_timeout_s; npm: shutdownTimeoutS)10,000 ms10,000 ms

The OTLP exporter (Python) also opens after 5 failed calls and stays open for 60 seconds. After the cooldown it lets one call through; if that call fails, it opens again at once. It counts a 429 or a 503 as a failure. It keeps its own retry rules: up to max_retries attempts for each call (default 3), and no retry horizon.

Multi-worker servers and fork()

If you run gunicorn, uvicorn with workers, or celery with the default prefork pool, your process forks after startup. This matters, because a forked child inherits the parent's exporter state — including its open sockets and its circuit-breaker counters.

The Python SDK registers an os.register_at_fork hook that resets that state in the child. The hook first shipped in 0.1.14.

VersionPost-fork hook
0.2.0Yes
0.1.14Yes
0.1.12 and earlierNo

There is no 0.1.13 release. If you are on 0.1.12 or earlier, upgrade — this is the single highest-value upgrade for anyone running a forking server.

What the hook actually does, measured by dirtying the parent's exporter state and then inspecting it in the child:

State in the child0.2.00.1.12
Inherited HTTP client handledropped, rebuilt lazily by the childinherited from the parent
consecutive_failures (parent set to 4)reset to 04
Circuit-open deadline (parent set)reset to 0.0inherited

On 0.1.12 the child shares the parent's socket pool — which scrambles HTTP/2 stream state for both sides — and starts life one failure away from an open circuit that it never earned.

# Which version are you actually running?
python -c "import risicare; print(risicare.__version__)"

On 0.1.14+ no post_fork hook of your own is required: initialise once, before the fork, and the child repairs itself.

Process pools: in a pool worker, call risicare.flush() at the end of each task. Since Python 0.5.0 a forked child does not keep the SIGTERM handler of the parent, so a pool no longer hangs in pool.terminate(). pool.terminate() and the end of a with Pool() block end the workers with SIGTERM, and a worker that did not flush loses the spans it made since its last export (every 1 s by default). The main process keeps its exit handler and its SIGTERM flush.

Do not validate fork behaviour on macOS

On macOS a forked child that performs any network I/O crashes before it can send anything. We reproduced this with and without the Risicare SDK loaded: a control script that never imports risicare — parent starts a thread, child makes one HTTP request — dies with objc[…]: +[NSNumber initialize] may have been in progress in another thread when fork() was called. … Crashing instead.

This is a macOS Objective-C fork-safety property, not a Risicare defect. Validate multi-worker behaviour on Linux, which is what you deploy on.

Silent failure modes

These are the ones that cost people days, because nothing crashes.

No API key — nothing is sent

With no API key (also a key of only whitespace), Python 0.6.0 or later and npm 0.9.0 or later send nothing to Risicare. Tracing is off: is_enabled() / isEnabled() and flush() are false, nothing is counted as exported, and init() logs one WARNING that says so. The WARNING does not name an endpoint, because no request is made. It is a single line at startup, so it is visible at Python's default logging level, but only if you read the process log. RISICARE_TRACING=true and enable() do not change it. (In Python 0.5.1 and earlier, and in npm 0.8.0 and earlier with RISICARE_TRACING=true, spans were created and then discarded, and the WARNING named the endpoint.)

import risicare
risicare.init(api_key="rsk-...")     # or set RISICARE_API_KEY

If you see no data in the dashboard, check this first.

debug=True writes unredacted content to stdout

In the Python SDK, debug=True attaches a console exporter unconditionally — including when an API key is set. That exporter writes every span attribute to stdout with no redaction. Values are truncated at 80 characters, which is truncation, not masking: anything shorter than 80 characters is printed in full.

We measured this A/B at the file-descriptor level, with the process's stdout redirected to a file, using reserved test values:

Value on a span attributedebug=Truedebug=False
SSN-shaped (987-65-4320)printed in fullabsent
Visa test PAN (4111111111111111)printed in fullabsent
Email (user@example.invalid)printed in fullabsent
Documentation IP (192.0.2.10)printed in fullabsent

Both arms delivered their span to the sink, so the debug=False column is a real absence, not a run that did nothing.

debug=True adds span payloads, not diagnostics

debug=True sets a config flag. The WARNING for a failed export, which names the HTTP status, is logged with or without it. Measured with Python 0.6.0 against a backend returning 500, over three spans:

WARNING naming HTTP 500Lines for each attemptUnredacted span dumps
Default100
debug=True103 spans, each printed once for each export round (12 dumps)
Logger at DEBUG1yes0

So debug=True gives you the payloads you did not ask for and no more diagnostics. To see each attempt, use the logger instead.

import logging
logging.getLogger("risicare").setLevel(logging.DEBUG)

The JavaScript SDK differs here: debug: true prints no span, with or without a key (it writes the SDK's own debug lines). In npm 0.8.0 and earlier, with no key and tracing forced on, it printed each span and counted it as exported.

Content is captured by default, and the endpoint defaults to production

trace_content defaults to true in the published SDKs (Python risicare 0.11.0, npm risicare 0.14.0), so prompt and completion content is captured unless you turn it off. The default has been on since Python 0.7.0 and npm 0.10.0. It was off from Python 0.2.2 to 0.6.3 and from npm 0.5.2 to 0.9.2. In both SDKs, spans go to https://ingest.risicare.ai by default, and scores, prompt pulls, evaluation runs and fix reads go to https://api.risicare.ai.

To keep content in your process:

import risicare
 
risicare.init(api_key="rsk-...", trace_content=False)
# or: export RISICARE_TRACE_CONTENT=false

The project setting "Capture Content, this project" (Settings > General) is on by default. When you turn it off, the gateway replaces prompt, completion and tool content with <risicare:content-omitted> before it stores the span, and it also masks the exception message and the status message. The change takes effect within two minutes. It does not remove content that is already stored. This switch acts after the text has left your process, so use trace_content=False when the text must not leave it.

trace_content gates content-named attributes

With trace_content off, the published SDK replaces the value of every content-named attribute with <risicare:content-omitted> before export — the provider prompt and completion keys, and any key ending in .content or _content — whichever code set it. An attribute your own code sets under any other key is still exported. (In 0.2.0, which the measurements on this page used, only the provider and framework integrations honoured the setting.)

If you need content filtered regardless of where it came from, use mask.

The text of an error is not covered by trace_content. The exception message, the stack trace, the span status message and the error.message attribute are exported also when trace_content is off. Only the project setting masks them, at the gateway. If a provider's error message repeats part of a prompt, that text is in those fields. mask receives them as statusMessage, exceptions.{i}.message, exceptions.{i}.stacktrace and error.message.

mask — the only pre-export redaction hook

mask(key, value) -> value runs on every content-bearing field before anything leaves your process. It is the only such hook in either SDK. It receives the attribute key, so you can redact content without erasing structural values like model names.

import re
import risicare
 
SENSITIVE = re.compile(r"(ssn|card|email|authorization|api[_-]?key)", re.I)
 
def mask(key: str, value):
    if SENSITIVE.search(key):
        return "[REDACTED]"
    return value
 
risicare.init(api_key="rsk-...", mask=mask)

Verified against a loopback sink, reading the request body:

On the wireWith maskWithout mask
SSN / PAN / email valuesabsentpresent
[REDACTED] markerpresentabsent
gen_ai.request.model (gpt-4o)presentpresent

Structural values survive; content is replaced. Note the right-hand column: with no mask and default settings, that content goes out.

The mask runs on the export worker, not your request thread — but a slow mask stalls the drain. If it raises, that one field becomes a marker string and the span is still exported.

Environment variables

risicare.init() and the import hooks read sixteen environment variables: init() reads fifteen, and the import hooks read RISICARE_STRICT_INSTRUMENTATION. Explicit arguments to init() take precedence over the variables that init() reads.

VariableDefault
RISICARE_API_KEYunset. With no key (also a key of only whitespace), the SDK sends nothing, tracing is off, is_enabled() / isEnabled() and flush() are false, and init() logs one WARNING (Python 0.6.0 or later, npm 0.9.0 or later). In Python, an exporter that you name yourself still delivers
RISICARE_ENDPOINThttps://ingest.risicare.ai
RISICARE_API_ENDPOINThttps://api.risicare.ai. If this variable is unset and RISICARE_ENDPOINT (or endpoint) names a host other than the default ingest host, scores, prompt pulls, evaluation runs and fix reads go to that host
RISICARE_ENVIRONMENTunset. init() and the Tier 0 path (RISICARE_TRACING and no init() call) then use development
RISICARE_SERVICE_NAMEunset. No service name is sent
RISICARE_SERVICE_VERSIONunset
RISICARE_PROJECT_IDunset — deprecated, ignored by the gateway; removed in v1.0
RISICARE_TRACINGunset
RISICARE_TRACE_CONTENTtrue (content is captured)
RISICARE_SAMPLE_RATE1.0 (an unparseable value falls back to 1.0)
RISICARE_DEBUGfalse
RISICARE_OTLP_ENDPOINTunset
RISICARE_OTLP_HEADERSunset
RISICARE_OTEL_BRIDGEfalse
RISICARE_FIX_RUNTIMEfalse
RISICARE_STRICT_INSTRUMENTATIONunset

RISICARE_FIX_* is not read by init()

The names RISICARE_FIX_ENABLED, RISICARE_FIX_CACHE, RISICARE_FIX_CACHE_TTL, RISICARE_FIX_AUTO_REFRESH, RISICARE_FIX_REFRESH_INTERVAL, RISICARE_FIX_DRY_RUN, RISICARE_AB_TESTING and RISICARE_TRACK_EFFECTIVENESS exist in the package, but they are only read when a FixRuntimeConfig is built from the environment. risicare.init() does not do that, so setting them has no effect on the normal path.

The one switch that init() reads is fix_runtime (or RISICARE_FIX_RUNTIME). It is off by default.

The JavaScript SDK reads sixteen variables of its own, including four with no Python equivalent: RISICARE_BATCH_SIZE, RISICARE_BATCH_TIMEOUT_MS, RISICARE_MAX_QUEUE_SIZE and RISICARE_COMPRESS. Both SDKs also read RISICARE_PROMPT_CACHE_TTL_S and RISICARE_PROMPT_TIMEOUT_S for prompts, and RISICARE_EVAL_STATE for evaluation runs.

Outbound requests you should expect

By default the SDK makes two kinds of outbound request:

  • POST /v1/spans to https://ingest.risicare.ai, from the exporter.
  • POST /v1/scores to https://api.risicare.ai, each time your code calls score().

These calls also send requests to https://api.risicare.ai:

  • get_prompt (npm: getPrompt) sends GET /v1/prompts/resolve.
  • evals.run (npm: runDataset), the pytest plugin and risicare eval gate send requests to the dataset and evaluation routes.

The fix runtime is off by default since Python 0.4.0 and JavaScript 0.7.0. init() does not start it, and nothing polls /v1/fixes/active. If you turn it on with fix_runtime=True (or RISICARE_FIX_RUNTIME=true), the SDK sends a GET to /v1/fixes/active on the API host at startup, and again every 60 seconds by default. During the beta the Risicare service gives no fixes to customer projects, so keep the fix runtime off.

Verifying delivery yourself

Two traps if you write your own check:

The SDK's success signal is the gateway's acknowledgement, and it means "queued". The exporter reads the response body. A response is a success only if it is a 2xx that carries accepted and rejected counts. A redirect is a failure, and the SDK does not follow it. A 2xx without those counts is a failure. If the gateway rejects some of the spans in a batch, the SDK logs a WARNING, counts the accepted spans in exported_spans and the others in rejected_spans, and the next flush() returns False. The acknowledgement means that the gateway put the spans in its queue. It does not mean that the spans are stored. To confirm that a trace is stored, open it in the dashboard.

In JavaScript, traceAct and traceThink return a wrapper you must call. Building the wrapper and never invoking it emits no span — and every check then reads a legitimate zero for the wrong reason.

import * as risicare from 'risicare';
 
risicare.init({ apiKey: 'rsk-...' });
 
// Wrong: builds a wrapper, never runs it, emits nothing.
risicare.traceAct('never-invoked', async () => 'body never runs');
 
// Right: call what traceAct returns.
const wrapped = risicare.traceAct('act-invoked', async (x) => x * 2);
await wrapped(21);
 
await risicare.shutdown();

Both forms above were executed against a loopback sink: only act-invoked arrived. never-invoked produced no span at all.