Production & Failure Modes

This page describes what the Risicare SDK does inside your production process: what it costs your request path, what happens when our backend is unreachable, and exactly which spans you lose. Every number here was measured against the published packages — Python risicare 0.2.0 (with risicare-core 0.1.6) and JavaScript risicare 0.5.0 — not against internal builds.

If you are evaluating whether to put this library in a hot path, read the first two sections. They are the ones that decide it.

What the SDK does not do to your application

These are the guarantees. They hold whether our backend is healthy, degraded, or completely unreachable.

Tracing never raises into your code. An export failure is logged, never thrown. No span operation propagates an exception into your call frame. If our gateway returns 500 for an hour, your application does not see an error from us.

Your thread does not block on export. Spans are handed to an in-process queue and written by a background worker. Span creation cost is unchanged by backend health — measured over 20,000 spans on each arm:

Tens of microseconds, and the dead-backend arm is not slower. (It is marginally faster, because once the queue is full, dropping a span is cheaper than enqueuing one.) Treat these as an order of magnitude on your own hardware, not a guarantee of a specific figure.

Memory is bounded. The queue is capped at max_queue_size, default 10,000 spans. When it is full, further spans are dropped rather than buffered. The queue does not grow without limit during an outage.

A clean shutdown does not lose what it holds. shutdown() drains the queue within a budget of 5,000 ms by default, reserving the last 100 ms to close the exporter. Anything still queued when that budget expires is counted and announced with a warning — it is never discarded silently.

What you lose when our backend is unreachable

This is the part that costs you data, and it is the reason this page exists.

The exporter has a circuit breaker. After 5 consecutive failed export calls it stops attempting exports entirely for a 60-second cooldown. During that cooldown no request is made — even if the backend has already recovered.

Count failed calls, not seconds

An export call is one batch, not one span and not one HTTP request. Each export call makes up to 3 HTTP attempts internally, with 0.1 s and 0.2 s backoff between them, before it counts as a single consecutive failure.

So the breaker opens after 5 failed calls — which is up to 15 HTTP requests. Where that lands in wall-clock time is not fixed: it depends on your span rate and on how quickly your endpoint fails. An endpoint that refuses connections instantly burns those 5 calls in well under a second; one that times out slowly takes far longer. This is why the threshold is documented in calls rather than seconds — any second-based figure would be specific to one harness.

Measured

Driving the published HttpExporter against a loopback sink that returned 500 to everything, then flipping that sink healthy:

The sink was healthy and idle for the whole minute. The SDK did not probe it once. That is the cost the breaker imposes: a brief outage bills you for the outage plus up to a further 60 seconds.

Which spans survive

When the queue is full the incoming span is dropped. The oldest spans already queued are kept.

Measured by creating 20,000 spans against a dead backend, then restoring it: exactly 10,500 distinct spans were delivered, contiguously numbered from the first span created — 10,000 from the queue plus one in-flight batch of 500. Spans 10,501 onward were dropped. If you are debugging an incident from a truncated trace, you are looking at its beginning, not its end.

Dropped spans are counted by reason (queue_full, transport_refused, shutdown_residue) and surfaced through the processor's metrics.

Defaults that govern this

The OTLP exporter applies the same 5-failure / 60-second policy as the default HTTP exporter.

Multi-worker servers and fork()

If you run gunicorn, uvicorn with workers, or celery with the default prefork pool, your process forks after startup. This matters, because a forked child inherits the parent's exporter state — including its open sockets and its circuit-breaker counters.

The Python SDK registers an os.register_at_fork hook that resets that state in the child. The hook first shipped in 0.1.14.

There is no 0.1.13 release. If you are on 0.1.12 or earlier, upgrade — this is the single highest-value upgrade for anyone running a forking server.

What the hook actually does, measured by dirtying the parent's exporter state and then inspecting it in the child:

On 0.1.12 the child shares the parent's socket pool — which scrambles HTTP/2 stream state for both sides — and starts life one failure away from an open circuit that it never earned.

# Which version are you actually running?
python -c "import risicare; print(risicare.__version__)"

On 0.1.14+ no post_fork hook of your own is required: initialise once, before the fork, and the child repairs itself.

Silent failure modes

These are the ones that cost people days, because nothing crashes.

No API key — spans are created and dropped

Without an API key no exporter is configured. Spans are still created, and then discarded. The SDK emits a WARNING naming the endpoint when this happens, so it is visible at Python's default logging level — but only if you actually read the process log, and it is a single line at startup.

import risicare
risicare.init(api_key="rsk-...")     # or set RISICARE_API_KEY

If you see no data in the dashboard, check this first.

debug=True writes unredacted content to stdout

In the Python SDK, debug=True attaches a console exporter unconditionally — including when an API key is set. That exporter writes every span attribute to stdout with no redaction. Values are truncated at 80 characters, which is truncation, not masking: anything shorter than 80 characters is printed in full.

We measured this A/B at the file-descriptor level, with the process's stdout redirected to a file, using reserved test values:

Both arms delivered their span to the sink, so the debug=False column is a real absence, not a run that did nothing.

import logging
logging.getLogger("risicare").setLevel(logging.DEBUG)

The JavaScript SDK differs here: it attaches its console exporter only when debug is set and no API key is present. A configured JS production process does not hit this.

Content is captured by default, and the endpoint defaults to production

trace_content defaults to true in both SDKs, and the endpoint defaults to https://app.risicare.ai in both. Nothing needs to be switched on for prompt and completion content to leave your process.

To turn the automatic capture off:

import risicare
 
risicare.init(api_key="rsk-...", trace_content=False)
# or: export RISICARE_TRACE_CONTENT=false

mask — the only pre-export redaction hook

mask(key, value) -> value runs on every content-bearing field before anything leaves your process. It is the only such hook in either SDK. It receives the attribute key, so you can redact content without erasing structural values like model names.

import re
import risicare
 
SENSITIVE = re.compile(r"(ssn|card|email|authorization|api[_-]?key)", re.I)
 
def mask(key: str, value):
    if SENSITIVE.search(key):
        return "[REDACTED]"
    return value
 
risicare.init(api_key="rsk-...", mask=mask)

Verified against a loopback sink, reading the request body:

Structural values survive; content is replaced. Note the right-hand column: with no mask and default settings, that content goes out.

The mask runs on the export worker, not your request thread — but a slow mask stalls the drain. If it raises, that one field becomes a marker string and the span is still exported.

Environment variables

Fourteen environment variables are honoured on the risicare.init() path in Python. Explicit arguments to init() take precedence over all of them.

The JavaScript SDK reads fourteen variables of its own, including four with no Python equivalent: RISICARE_BATCH_SIZE, RISICARE_BATCH_TIMEOUT_MS, RISICARE_MAX_QUEUE_SIZE and RISICARE_COMPRESS.

Outbound requests you should expect

When an API key is set, init() starts a fix runtime alongside the exporter. It issues a GET to /api/v1/fixes/active on your configured endpoint at startup. If your egress rules only allow the span-ingest path, allow this one too or expect a warning in your logs.

Verifying delivery yourself

Two traps if you write your own check:

The SDK's success signal is the HTTP status, nothing more. The exporter treats any response below 300 as success and never parses the response body. So "the SDK did not warn" means "the endpoint returned a 2xx" — it is not evidence that every span in that batch was stored. If you need to prove end-to-end delivery, count what arrives at the far end rather than trusting the absence of a warning.

In JavaScript, traceAct and traceThink return a wrapper you must call. Building the wrapper and never invoking it emits no span — and every check then reads a legitimate zero for the wrong reason.

import * as risicare from 'risicare';
 
risicare.init({ apiKey: 'rsk-...' });
 
// Wrong: builds a wrapper, never runs it, emits nothing.
risicare.traceAct('never-invoked', async () => 'body never runs');
 
// Right: call what traceAct returns.
const wrapped = risicare.traceAct('act-invoked', async (x) => x * 2);
await wrapped(21);
 
await risicare.shutdown();

Both forms above were executed against a loopback sink: only act-invoked arrived. never-invoked produced no span at all.

Related