Caching

Two distinct caches. Users search “caching” and land here, so this page distinguishes them.

1. Prompt caching (provider-side)

For Anthropic models, CodeBuddy uses Anthropic prompt caching to avoid paying full input-token cost on repeat context. When the same system prompt + early messages are seen twice, the second call reads cached tokens at a fraction of the write price.

• Cache hit rate on turn 2+ of a conversation: 99.3–99.98% measured 2026-07-10.
• Applied via anthropicPromptCachingMiddleware in the deepagents pipeline; nothing to configure.
• Costs tracked per bucket (input / cache_read / cache_creation / output) in CostTrackingService — see Cost tracking.

Turn 1 in each session writes the cache; every following turn hits the cache prefix. Bailout only if the system prompt changes mid-conversation (rare — the prompt is built once at agent construction).

Other providers don’t currently participate in provider-side caching.

2. In-process multi-tier cache (EnhancedCacheManager)

Local cache the extension uses to skip redundant vector-DB work, LLM responses, and file-metadata computation. Independent from provider-side caching.

Tiers

Tier	Key	Value	Default TTL	Purpose
Embedding	File content hash	number[]	1 hr	Skip re-embedding unchanged files
Search	Query hash	Search results	1 hr	Avoid duplicate vector-DB lookups
Metadata	File path	File metadata	1 hr	Cache stats, language detection
Response	Prompt hash	string	1 hr	Cache LLM responses for identical prompts

Each tier is its own Map<string, CacheEntry<T>> — no shared eviction pressure.

Entry shape

interface CacheEntry<T> {
  data: T;
  timestamp: number;      // Created-at
  accessCount: number;    // For LFU
  lastAccessed: number;   // For LRU
  ttl: number;            // ms
  size: number;           // Estimated bytes
}

Eviction

Three policies, one per instance:

Policy	Behavior	Best for
LRU	Drop least recently accessed. Default.	General use
LFU	Drop least frequently accessed.	Hot-path repeat queries
TTL	Drop oldest by creation timestamp.	Time-sensitive data

Configuration

interface CacheConfig {
  maxSize: number;                 // entries per tier (default 10 000)
  defaultTtl: number;              // ms (default 1 hr)
  maxMemoryMB: number;             // memory ceiling (default 100 MB)
  cleanupInterval: number;         // sweep timer (default 5 min)
  evictionPolicy: "LRU" | "LFU" | "TTL";  // default LRU
}

Sweep timer

Every 5 min:

1. TTL pass — remove entries where now > timestamp + ttl.
2. Memory pass — if total > maxMemoryMB, evict by policy until under.

Stats

getStats() returns hitCount, missCount, size, maxSize, memoryUsage, avgAccessTime, evictionCount. Reported to the Performance Profiler when connected.

Named instances

Multiple caches with different configs coexist:

const vectorCache = new EnhancedCacheManager(
  { maxSize: 10000, maxMemoryMB: 100, evictionPolicy: "LRU" },
  profiler,
  "vector-db",
);

const responseCache = new EnhancedCacheManager(
  { maxSize: 1000, maxMemoryMB: 20, evictionPolicy: "TTL" },
  profiler,
  "responses",
);

Each logs under a prefixed name (EnhancedCacheManager:vector-db).

Consumers

Consumer	Tiers used	Purpose
VectorDBService	Embedding + Search	Skip re-embedding, cache search results
ContextRetriever	Search + Metadata	Cache semantic search results
EnhancedPromptBuilder	Response	Cache identical prompt responses
CodebaseAnalysisWorker	Metadata	Cache file language detection

Memory estimation

Entry size is estimated by JSON-serializing the value and measuring bytes — an upper-bound approximation. For embeddings (number[]), size is roughly dimensions × 8 bytes (64-bit floats) plus array overhead.

Disposal

Implements vscode.Disposable. On dispose: cleanup timer cleared, tier maps cleared, stats reset. Prevents leaks on extension reload.

Related

• Semantic search — the biggest consumer of the vector caches
• Cost tracking — how cache-read/creation tokens get priced separately