Self-host an LLM stack in 2026: Ollama, Open WebUI, llama.cpp

A working self-hosted LLM stack in 2026 is three layers: runtime → chat UI → optional tools. Start with the first two; skip agents until you can chat reliably offline.

Minimal path (most people)

1. Install Ollama and pull one small and one mid-size model (e.g. a 7–8B and a 14–32B if VRAM allows).
2. Put Open WebUI (or a similar local UI) in front for multi-chat, model switching, and basic RAG.
3. Confirm: new chat → answer → stop generation → switch model. Only then add documents or tools.

If you need raw speed on Linux or custom quantizations, run llama.cpp / a compatible server instead of Ollama — same mental model, more knobs.

Hardware reality check

“Self-host” does not mean “frontier quality for free.” It means control and cost predictability.

When to add the next layer

• RAG: after you feel pain pasting the same docs into prompts.
• Coding agents (Continue, Aider, Cline): after the model can follow project conventions in plain chat.
• Gateways (LiteLLM, OpenRouter as hybrid): when you mix local + paid APIs behind one client.

See Open-source / Self-host and Free-only.

Attribution / honesty

Model cards and licenses differ. Treat every weights download like a dependency: check license, training data claims, and evals that match your tasks.

This guide is operational, not a vendor endorsement.