Self-Healing Execution

Four layers, each independent. A failure at one layer is caught by the next.

graph TB L1[Layer 1 · Provider Failover<br/>Switch LLM providers on failure] --> L2 L2[Layer 2 · Stream Error Recovery<br/>Classify + retry with backoff] --> L3 L3[Layer 3 · Agent Safety Guard<br/>Hard caps on events, tools, time, loops] --> L4 L4[Layer 4 · Context Compaction<br/>Prevent context overflow]

Layer 1 — Provider failover

ProviderFailoverService classifies every provider error and picks the next candidate from the configured chain.

Reason	Cooldown	Triggers
`auth`	10 min	HTTP 401/403, “invalid API key”
`rate_limit`	1 min	HTTP 429, “quota exceeded”
`billing`	30 min	HTTP 402, billing errors
`timeout`	30 s	HTTP 408, `ETIMEDOUT`, network timeout
`overloaded`	2 min	HTTP 502/503/529, “overloaded”
`model_not_found`	1 hr	HTTP 404
`format`	—	Parse errors (don’t failover; the model itself is fine)
`unknown`	30 s	Everything else

Classification walks the error cause chain up to 5 levels deep. Probe recovery attempts fire 30 s before cooldown expires.

Failover chain: configured via codebuddy.failover.providers (empty = auto-detect from configured keys). When the primary trips, the next non-cooldown provider takes over. Response carries isFallback: true so callers know a switch happened.

Layer 2 — Stream error recovery

ErrorRecoveryService classifies individual stream errors:

Class	Behavior	Patterns
Transient	Retry with exponential backoff	`timeout`, `ECONNRESET`, `429`, `502`, `503`, `overloaded`, `socket hang up`
Permanent	Fail immediately	`loop detected`, `safety limit`, `authentication`, `invalid api key`, `quota exceeded`

Retry: max 2 attempts per stream, base delay 1500 ms, exponential (1.5s → 3s → 6s). Each retry includes a nudge message to the agent explaining the failure so it can adjust.

Safety-guard errors are NEVER retried — that would circumvent Layer 3’s guarantees.

Layer 3 — Agent safety guard

Hard caps. Not overridable by the agent.

Guardrail	Limit	Setting
Max stream events	15000	`codebuddy.agent.maxEventCount`
Max tool calls	2000	`codebuddy.agent.maxToolInvocations`
Wall-clock timeout	60 min	`codebuddy.agent.maxDurationMinutes`
`edit_file` invocations	75	`codebuddy.agent.limits.editFile`
`delete_file` invocations	30	`codebuddy.agent.limits.deleteFile`
`run_command`	100	`codebuddy.agent.limits.runCommand`
`run_terminal_command`	500	`codebuddy.agent.limits.runTerminalCommand`
`web_search`	60	`codebuddy.agent.limits.webSearch`

Loop detection:

Tool loops — same tool called repetitively without progress → force stop after N in a row.
File edit loops — same file edited more than codebuddy.agent.fileEditLoopThreshold (default 20) → force stop with a “file loop” reason.

Stop messages are human-readable and include the counters:

Forced stop: reached maximum of 2000 tool invocations.
Events: 6247 · Tool calls: 2000 · Elapsed: 34m 23s.
Please review the work completed so far.

The LangChainTool layer also enforces a per-tool sliding-window rate limit (600/min global, 200/min per-tool) via ToolRateLimiter. That’s belt-and-suspenders with the safety guard’s total-count limits.

Layer 4 — Context compaction

ContextWindowCompactionService keeps message history within the active model’s context window.

Tiers, escalating:

Tier	Name	Approach	LLM used?
0	None	No action	—
1	Tool strip	Strip tool results > 200 chars from older messages	No
2	Multi-chunk	Summarize message batches with overlapping windows	Yes
3	Partial	Summarize the oldest half	Yes
4	Plain fallback	Plain-text description when LLM summarization itself fails	No

Thresholds:

Warning at 80% of the window — log + prepare.
Auto-compact at 90% — run Tier 1 immediately.
Higher tiers kick in as needed until usage is back under threshold.

Protected: most recent 4 messages are never summarized. A minimum of 6 messages must exist before any summarization runs.

Known context windows:

Model	Window
Claude Sonnet 4/5	200 K
Claude Opus 4	200 K
GPT-4o	128 K
GPT-4	8 K
Gemini 1.5 Pro	2.1 M
DeepSeek Chat	64 K
Qwen Plus	131 K

Unknown-model resolution (priority): user codebuddy.contextWindow setting → exact model match → family fallback → 16 K default. The family fallback maps an unlisted dated variant to its family’s window by prefix (claude-* → 200 K, gemini-* → 1.05 M, etc.), so a new model id like claude-sonnet-4-6 resolves to 200 K instead of collapsing to the 16 K default and triggering catastrophic compaction (the 2026-08-09 incident root cause).

Cancellation & crash recovery

Beyond the four automatic layers, two mechanisms keep a bad turn from taking the session with it:

Cooperative cancellation — a Stop from the chat UI signals a per-conversation AbortController; the loop checks between steps and unwinds cleanly. Work already applied stays applied and reviewable.
Graceful degradation instead of crashes — a policy-denied write (e.g. a restricted profile blocking a path) degrades to a message and returns rather than failing the whole turn or triggering failover; a mid-turn vendor recompile error (code-interpreter path) degrades to a note instead of aborting.
Failed-turn persistence — if a turn crashes mid-stream, it’s written back with a visible failedTurn marker so it shows on reload instead of vanishing silently. Per-session checkpoint isolation (thread_id == sessionId) keeps one conversation’s crash from bleeding into another.

Layers in a single scenario

sequenceDiagram participant Agent participant L4 as L4 Compaction participant L3 as L3 Safety Guard participant L2 as L2 Recovery participant L1 as L1 Failover participant LLM Note over Agent, LLM: Rate-limited then recovered Agent->>LLM: request LLM-->>L2: HTTP 429 L2->>L2: transient → retry L2->>LLM: retry after 1.5s LLM-->>L2: HTTP 429 L2->>LLM: retry after 3s LLM-->>L2: HTTP 429 L2->>L1: give up → failover L1->>LLM: switch to backup provider LLM-->>L1: success L1-->>Agent: continue (isFallback: true) Note over Agent, LLM: Hit tool cap Agent->>L3: tool call #2001 L3-->>Agent: stop — max 2000 Note over Agent, LLM: Context near cap Agent->>L4: at 85% L4->>L4: Tier 1 strip → 88% L4->>L4: Tier 3 summarize → 62% L4-->>Agent: continue

Architecture — where the safety guard sits in the request flow
Providers — the failover chain configuration
Security — permission profiles + catastrophic-command denies