Local Models

The Local provider uses the OpenAI-compatible API protocol. Anything that speaks that API works — Ollama, Docker Model Runner, LM Studio, vLLM, LocalAI, KoboldCpp, text-generation-webui with --api, or any other OpenAI-shape server.

What connects where

Mode	Class	Underlying wire
Ask	LocalLLM	OpenAI Node SDK pointed at local.baseUrl
Inline completion	LocalLLM	Same
Agent	ChatOpenAI	LangChain wrapper — same endpoint, full tool-calling

Runtimes

Runtime	Default port	Base URL
Ollama (native)	11434	http://localhost:11434/v1
Ollama (Docker)	11434	Same (started via bundled docker-compose.yml)
Docker Model Runner	12434	http://localhost:12434/engines/llama.cpp/v1
LM Studio	1234	http://localhost:1234/v1
vLLM	8000	http://localhost:8000/v1
Any OpenAI-compatible	varies	Set local.baseUrl accordingly

Port fallback: if Docker Model Runner on 12434 is unreachable, CodeBuddy silently falls back to Ollama on 11434 and updates config.

Quick starts

Ollama (recommended)

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5-coder
# starts automatically on :11434

Set generativeAi.option = "Local" — the default local.baseUrl already points at Ollama.

Docker Compose (bundled)

The extension ships a docker-compose.yml for containerized Ollama with persistent storage:

1. Settings → Models → Start Server
2. Pick a model from the list
3. Use to activate

Compose config: 32 GB memory cap, ollama_data volume. NVIDIA GPU support via uncommenting the deploy block.

Docker Model Runner

Docker Desktop 4.37+ ships a built-in llama.cpp runner:

1. Settings → Models → Enable Docker Model Runner
2. Pull via the Docker model registry
3. Models exposed as ai/<name> at port 12434

Recommended models

Model	Size	Best for
Qwen 2.5 Coder 7B	~4.7 GB	Default coding model — recommended
Qwen 2.5 Coder 3B	~2.0 GB	Faster / lighter coding
DeepSeek Coder	~6.7 GB	Strong completion benchmarks
CodeLlama 7B	~3.8 GB	Meta’s code-focused model
Llama 3.2 3B	~2.0 GB	Efficient general-purpose

Any model available in your runtime works — these are just the ones surfaced in the UI.

Settings

Setting	Default	Purpose
local.model	"qwen2.5-coder"	Model name (must match what’s pulled)
local.baseUrl	"http://localhost:11434/v1"	API endpoint
local.apiKey	"not-needed"	Placeholder — most local servers don’t need it
generativeAi.option	current provider	Set to "Local" to activate

Inline completion

Local is the default provider for ghost-text inline completion. Fast, private, no cloud calls.

Setting	Default	Purpose
codebuddy.completion.provider	"Local"	Completion provider
codebuddy.completion.model	"qwen2.5-coder"	Completion model
codebuddy.completion.debounceMs	300	Trigger delay
codebuddy.completion.maxTokens	128	Max tokens per completion
codebuddy.completion.triggerMode	"automatic"	automatic or manual
codebuddy.completion.multiLine	true	Multi-line completions

Strategies attempted, in order:

1. Fill-in-the-Middle API (for models with FIM support)
2. Chat API fallback (for chat-only models)

Local embeddings

Local models can back the vector store:

{ "codebuddy.vectorDb.embeddingModel": "local" }

Calls /embeddings on the local server. Default embedding model: text-embedding-v1. If your model doesn’t expose /embeddings, fall back to "gemini" (default) or "openai".

Agent-mode performance note

Agent mode runs many sequential LLM calls per task (reason → tool select → execute → reason). Local models — especially 3B–7B — are noticeably slower than cloud. Larger models (13B+) or GPU acceleration close the gap.

The system prompt is tuned to prevent local models from hallucinating tool calls when they’re outside an agent context.

Model management UI

Settings → Models page:

• Runtime status indicators (Ollama / Docker Model Runner up-or-down)
• Per-model Pull / Use / Delete buttons
• Pull progress indicators
• Active model highlighted
• Docker Compose controls to start / stop containerized Ollama

The sidebar model-selector pill shows the active model and polls runtime status every 30 s.

Related

• Providers — full provider abstraction
• Configuration — setting keys for all providers
• Inline completion — ghost-text detail