Self-Healing Execution

Four layers, each independent. A failure at one layer is caught by the next.

graph TB L1[Layer 1 · Provider Failover<br/>Switch LLM providers on failure] --> L2 L2[Layer 2 · Stream Error Recovery<br/>Classify + retry with backoff] --> L3 L3[Layer 3 · Agent Safety Guard<br/>Hard caps on events, tools, time, loops] --> L4 L4[Layer 4 · Context Compaction<br/>Prevent context overflow]

Layer 1 — Provider failover

ProviderFailoverService classifies every provider error and picks the next candidate from the configured chain.

Reason	Cooldown	Triggers
auth	10 min	HTTP 401/403, “invalid API key”
rate_limit	1 min	HTTP 429, “quota exceeded”
billing	30 min	HTTP 402, billing errors
timeout	30 s	HTTP 408, ETIMEDOUT, network timeout
overloaded	2 min	HTTP 502/503/529, “overloaded”
model_not_found	1 hr	HTTP 404
format	—	Parse errors (don’t failover; the model itself is fine)
unknown	30 s	Everything else

Classification walks the error cause chain up to 5 levels deep. Probe recovery attempts fire 30 s before cooldown expires.

Failover chain: configured via codebuddy.failover.providers (empty = auto-detect from configured keys). When the primary trips, the next non-cooldown provider takes over. Response carries isFallback: true so callers know a switch happened.

Layer 2 — Stream error recovery

ErrorRecoveryService classifies individual stream errors:

Class	Behavior	Patterns
Transient	Retry with exponential backoff	timeout, ECONNRESET, 429, 502, 503, overloaded, socket hang up
Permanent	Fail immediately	loop detected, safety limit, authentication, invalid api key, quota exceeded

Retry: max 2 attempts per stream, base delay 1500 ms, exponential (1.5s → 3s → 6s). Each retry includes a nudge message to the agent explaining the failure so it can adjust.

Safety-guard errors are NEVER retried — that would circumvent Layer 3’s guarantees.

Layer 3 — Agent safety guard

Hard caps. Not overridable by the agent.

Guardrail	Limit	Setting
Max stream events	15000	codebuddy.agent.maxEventCount
Max tool calls	2000	codebuddy.agent.maxToolInvocations
Wall-clock timeout	60 min	codebuddy.agent.maxDurationMinutes
edit_file invocations	75	codebuddy.agent.limits.editFile
delete_file invocations	30	codebuddy.agent.limits.deleteFile
run_command	100	codebuddy.agent.limits.runCommand
run_terminal_command	500	codebuddy.agent.limits.runTerminalCommand
web_search	60	codebuddy.agent.limits.webSearch

Loop detection:

• Tool loops — same tool called repetitively without progress → force stop after N in a row.
• File edit loops — same file edited more than codebuddy.agent.fileEditLoopThreshold (default 20) → force stop with a “file loop” reason.

Stop messages are human-readable and include the counters:

Forced stop: reached maximum of 2000 tool invocations.
Events: 6247 · Tool calls: 2000 · Elapsed: 34m 23s.
Please review the work completed so far.

The LangChainTool layer also enforces a per-tool sliding-window rate limit (600/min global, 200/min per-tool) via ToolRateLimiter. That’s belt-and-suspenders with the safety guard’s total-count limits.

Layer 4 — Context compaction

ContextWindowCompactionService keeps message history within the active model’s context window.

Tiers, escalating:

Tier	Name	Approach	LLM used?
0	None	No action	—
1	Tool strip	Strip tool results > 200 chars from older messages	No
2	Multi-chunk	Summarize message batches with overlapping windows	Yes
3	Partial	Summarize the oldest half	Yes
4	Plain fallback	Plain-text description when LLM summarization itself fails	No

Thresholds:

• Warning at 80% of the window — log + prepare.
• Auto-compact at 90% — run Tier 1 immediately.
• Higher tiers kick in as needed until usage is back under threshold.

Protected: most recent 4 messages are never summarized. A minimum of 6 messages must exist before any summarization runs.

Known context windows:

Model	Window
Claude Sonnet 4/5	200 K
Claude Opus 4	200 K
GPT-4o	128 K
GPT-4	8 K
Gemini 1.5 Pro	2.1 M
DeepSeek Chat	64 K
Qwen Plus	131 K

Unknown models fall back to 8 K conservatively.

Layers in a single scenario

sequenceDiagram participant Agent participant L4 as L4 Compaction participant L3 as L3 Safety Guard participant L2 as L2 Recovery participant L1 as L1 Failover participant LLM Note over Agent, LLM: Rate-limited then recovered Agent->>LLM: request LLM-->>L2: HTTP 429 L2->>L2: transient → retry L2->>LLM: retry after 1.5s LLM-->>L2: HTTP 429 L2->>LLM: retry after 3s LLM-->>L2: HTTP 429 L2->>L1: give up → failover L1->>LLM: switch to backup provider LLM-->>L1: success L1-->>Agent: continue (isFallback: true) Note over Agent, LLM: Hit tool cap Agent->>L3: tool call #401 L3-->>Agent: stop — max 400 Note over Agent, LLM: Context near cap Agent->>L4: at 85% L4->>L4: Tier 1 strip → 88% L4->>L4: Tier 3 summarize → 62% L4-->>Agent: continue

Related

• Architecture — where the safety guard sits in the request flow
• Providers — the failover chain configuration
• Security — permission profiles + catastrophic-command denies