Skip to main content
GitHub

Overview

How Risicare's self-healing pipeline works.

Nothing deploys on its own — and stage 6 does not exist

Stages 1-3 (detect → diagnose → generate fix recommendation) ship today and run automatically. Stages 4-5 (deploy → validate) are built and running, but nothing reaches them by itself: a fix leaves draft only when someone calls POST /api/v1/fixes/{fix_id}/promote, and no one ever has — the deployments table has never held a row. So the statistical validation and progressive rollout below are real code that has never run on real traffic.

Stage 6 (learn) is genuinely not built — there is no learned pattern or fix-template store.

Risicare automatically detects errors, diagnoses root causes, and generates fix recommendations.

Pipeline

Self-healing pipeline: Error → Diagnosis → Fix Generation → Canary Deploy → A/B Testing → Graduate

Diagnosis → Hypothesis Generation → Validation → Deployment → Learning
             ↓                        ↓            ↓           ↓
         Generate fix ideas     Test each one   Canary → A/B  Store pattern

How It Works

1. Receive Diagnosis

When an error is diagnosed, the healing pipeline receives:

  • Error code (e.g., TOOL.EXECUTION.TIMEOUT)
  • Root cause analysis
  • Context from the error trace
  • Similar past errors (if any)

2. Generate Hypotheses

Create testable hypotheses about what might fix the issue:

Diagnosis: TOOL.EXECUTION.TIMEOUT on weather_api

Hypothesis 1: Retry with backoff (0.75 prior)
Hypothesis 2: Increase timeout (0.60 prior)
Hypothesis 3: Add fallback (0.55 prior)

3. Validate Statistically

Each hypothesis is tested:

  1. A/B Test: Split traffic between baseline and fix
  2. Measure: Error rate, latency, cost
  3. Analyze: Statistical significance (p < 0.05)
  4. Decide: Accept, reject, or continue testing

4. Deploy Safely

Not yet connected

Automatic fix deployment is not wired to the runtime yet. The auto_fix_enabled project setting exists but is inert today — no fix is promoted out of draft, so nothing deploys regardless of the toggle. Generated fixes are visible in the dashboard for manual review only.

When connected, validated fixes will be deployed progressively:

Canary (5%) → Ramp (25%) → Ramp (50%) → Graduate (100%)

With automatic rollback if:

  • Error rate increases >10%
  • Latency exceeds 2x baseline
  • Manual intervention

5. Learn (planned — not implemented)

A future learning stage is designed to turn successful fixes into reusable knowledge (error-pattern → fix-template mappings, shared across customers in federated form). This stage is not built today — there is no pattern or fix-template store in production.

Fix Types

Risicare generates 7 types of fixes:

TypeWhat It Does
PromptModify system prompt
ParameterAdjust LLM settings
ToolFix tool configuration
RetryAdd retry with backoff
FallbackUse alternative strategy
GuardAdd validation
RoutingChange agent delegation

No Code Injection

Declarative Fixes

Fixes are JSON configurations, not code. Risicare never injects code into your system. Both the Python and JavaScript SDKs ship a Fix Runtime that interprets these at runtime, and both start it during init() — there is no separate opt-in. Each implements the same five appliers (prompt, parameter, retry, fallback, guard); neither implements tool or routing.

The active-fix list both runtimes fetch is empty until a fix is promoted, so in practice no fix is applied today. That is because nobody has called POST /api/v1/fixes/{fix_id}/promote — not because the path is missing. Promote a fix and GET /api/v1/fixes/active will serve it to both runtimes on their next refresh.

Example fix:

{
  "fix_id": "fix-abc123",
  "fix_type": "retry",
  "config": {
    "max_retries": 3,
    "initial_delay_ms": 1000,
    "exponential_base": 2.0,
    "max_delay_ms": 30000,
    "jitter": true,
    "retry_on": []
  }
}

Confidence Levels

Fixes have confidence scores:

ConfidenceMeaning
> 0.8High - strong recommendation
0.6 - 0.8Medium - review suggested
< 0.6Low - suggest only

Confidence drives recommendations, not deployment

Today, confidence scores rank fix recommendations for human review. They do not trigger any automatic deployment (stages 4-6 are not connected).

Dashboard

View healing activity:

  • Active Fixes: Deployed fixes (planned)
  • Testing: Fixes in A/B testing
  • Candidates: Suggested but not deployed
  • Graduated: Successfully deployed
  • Rolled Back: Failed fixes

Metrics

MetricDescription
Fix Rate% of errors with deployed fixes
Success Rate% of fixes that graduate
MTTRMean time to remediation
Error Reduction% error reduction from fixes

Next Steps