Heal

Risicare detects and classifies errors automatically today. Diagnosing root causes and generating fix recommendations are built and held — see the callout above.

Overview

The healing pipeline follows the DoVer methodology (Diagnosis via Observation of Verification):

1. Generate Hypotheses - Create testable hypotheses about fixes
2. Validate Statistically (built, never exercised) - Test fixes with A/B testing
3. Deploy Safely (built, never exercised) - Canary release with automatic rollback

Fix Types

Risicare can generate 7 types of fixes:

Fix Configuration

Fixes are JSON configurations, not code:

{
  "fix_id": "fix-abc123",
  "fix_type": "retry",
  "config": {
    "max_retries": 3,
    "initial_delay_ms": 1000,
    "exponential_base": 2.0,
    "max_delay_ms": 30000,
    "jitter": true,
    "retry_on": ["TimeoutError"]
  }
}

Hypothesis Testing

Before deployment, fixes are validated through hypothesis testing:

Generate Hypotheses

Diagnosis: TOOL.EXECUTION.TIMEOUT on weather_api

Hypothesis 1: Adding retry with backoff will reduce timeout errors
  Prior probability: 0.75 (based on similar patterns)

Hypothesis 2: Increasing timeout to 60s will reduce errors
  Prior probability: 0.60

Hypothesis 3: Adding fallback to cached data will maintain uptime
  Prior probability: 0.55

Statistical Validation

Each hypothesis is tested with:

• Sample size calculation for statistical power (0.8)
• Two-proportion z-test for significance (p < 0.05)
• Bayesian updates to posterior probability
• O'Brien-Fleming boundaries for early stopping

Test Results:
  Baseline error rate: 12.3%
  Treatment error rate: 2.1%
  Effect size (Cohen's h): 0.38
  P-value: 0.0023 ✓

  Decision: Hypothesis VALIDATED

Deployment Pipeline

Fix Created
     ↓
┌─────────────────┐
│ Canary (5%)     │  Minimum 100 samples
│                 │  Monitor error rate
└─────────────────┘
     ↓ (if passing)
┌─────────────────┐
│ Ramp (25%)      │  Statistical A/B test
│                 │  O'Brien-Fleming boundaries
└─────────────────┘
     ↓ (if winning)
┌─────────────────┐
│ Ramp (50%)      │  Continue testing
│                 │
└─────────────────┘
     ↓ (if winning)
┌─────────────────┐
│ Graduate (100%) │  Hold for 24 hours
│                 │  Mark as graduated
└─────────────────┘

Automatic Rollback

Fixes are automatically rolled back if:

• Error rate increases >10% vs baseline
• P99 latency exceeds 2x baseline
• Manual rollback triggered

Rollback latency design target: under 500ms — never measured. Automatic rollback runs in the deployment worker, which is gated off with the rest of fix deployment, so it does not run today.

Fix Runtime

The SDK includes a fix runtime. It is off by default since Python 0.4.0 and JavaScript 0.7.0. As designed, when it is on it:

1. Loads fixes from the API on startup
2. Caches locally with periodic refresh
3. Routes requests based on A/B assignment
4. Applies fixes at LLM call time

import risicare
 
# init() does not start the fix runtime: fix_runtime defaults to False since 0.4.0.
risicare.init(api_key="rsk-...")
 
# To start it, pass fix_runtime=True (or set RISICARE_FIX_RUNTIME=true).
# During the beta the route that the runtime reads is not reachable with an
# API key, so the runtime loads no fix. Keep it off.

Knowledge Base (planned)

A knowledge base (stage 6) is designed to store successful fixes so similar errors can reuse a known remedy instead of regenerating one:

• Error patterns and fix templates keyed by error code
• Cross-customer learning (federated, no raw data)

This stage is not connected — a knowledge store exists in the codebase but nothing uses it, there is no fix-template or pattern store in production, and fix generation (held for the beta) uses templates and LLM fallback only (see Diagnose → Pipeline).

Next Steps