Ollama
Local LLM runtime with a one-command install and an OpenAI-compatible server on localhost. For laptop development, edge deployment and air-gapped environments.
Overview
Ollama wraps llama.cpp, an MLX backend for Apple Silicon and its own engine behind a single command line and a local HTTP server. Pull a model, run it, and it answers on localhost:11434 in the OpenAI Chat Completions format. That is the whole product and it explains the reach: around 52 million monthly downloads and more than 40,000 community integrations, which is more distribution than every other local runtime combined.
For an engineering lead the interesting deployment is not the laptop, it is the edge. A production line that cannot send process data off-site, a clinic network with residency rules, a site with no reliable link to a cloud region. Those need inference on hardware inside the building, and Ollama turns that into something any engineer on the team can install and maintain rather than a GPU serving stack that needs a specialist to keep alive. The same property makes it the cheapest way to run model evaluations: pull six candidates, compare them on your own prompts, spend an afternoon and zero API budget.
What it is not is a serving platform, and it should never be presented internally as one. Four parallel requests by default, no continuous batching, no paged KV cache, so the moment real traffic arrives throughput flattens while latency compounds. The migration path is vLLM or a hosted endpoint, and Ollama Cloud at $20 a month makes that transition softer by keeping the HTTP surface identical. Just note that the Max tier stopped accepting new signups on capacity constraints, which tells you something about how much headroom the hosted side actually has.
Key Features
- ✓ One-command model pulls from a public registry, then a local HTTP server on port 11434 speaking the OpenAI Chat Completions format
- ✓ A native MLX engine for Apple Silicon alongside llama.cpp, so a 64GB M-series machine serves 30B-class models with no GPU rig involved
- ✓ Structured outputs with JSON schema validation and tool calling that matches the OpenAI surface
- ✓ Vision model support across the Qwen-VL and Llama vision families
- ✓ Ollama Cloud exposes the identical HTTP surface as the local runtime, so a laptop prototype redeploys against hosted GPUs without touching client code
- ✓ Version 0.33.0 caches resolved model metadata between requests, cutting time to first token from about 995ms to 524ms
Where it holds
- • Nothing gets a model running faster. Install to first token in under five minutes on a machine you already own
- • On the local path data never leaves the hardware, which is the entire argument in a regulated plant, a hospital network or anywhere procurement has said no to an API
- • Identical API local and hosted, which makes the prototype-to-pilot step unusually cheap
Where it breaks
- • The 4096 token default context is a trap. Go past it and Ollama truncates from the front silently: no warning, no error, just quietly worse answers. Set OLLAMA_CONTEXT_LENGTH before you do anything else
- • Concurrency is the hard ceiling. Four parallel requests by default, and 2026 benchmarks on Blackwell hardware put vLLM at roughly 16x the throughput, 8,033 against 484 tokens per second, with 6x faster time to first token
- • Quantized defaults mean the model you are evaluating is not the model that scored on the benchmark you read, and the gap is rarely documented in whatever comparison you are running
- • The $100/month Max tier has new signups paused on capacity, so the cloud side is not something to plan a roadmap around
My Take
Start with the flaw: the default context window is 4096 tokens, and when you exceed it Ollama drops the front of the prompt without telling you. Teams have shipped agents that degraded for weeks before anyone traced the cause back to a config default. Past that it is the fastest route from nothing to a running local model, v0.33.1 landed 26 August 2026 with time to first token roughly halved, and the runtime costs nothing on hardware you already own. Just do not put it behind more than about five concurrent users, because vLLM does 16x the throughput at that point and the gap only widens from there.
Quick Info
- Pricing:
- freemium
- Openness:
- Open source
- Licence:
- MIT
- Starting at:
- The local runtime is free and open source with no usage cap on your own hardware. Ollama Cloud adds hosted models: Free covers 1 concurrent cloud model at light usage, Pro is $20/month or $200/year for 3 concurrent models plus private model uploads, Max is $100/month for 10 concurrent models but new signups are currently paused on capacity. Team is $25 per seat per month with a 5 seat minimum and adds zero data retention, US and Europe model residency and priority support. Enterprise is custom.
- Added:
- Aug 2026
- Updated:
- Aug 2026
Use Cases
Judge it on your own work
The notes above say where Ollama holds and where it breaks. The fastest check is your own workload.
Visit website ↗Alternatives to Ollama
llama.cpp
open-sourceC/C++ LLM inference engine behind the GGUF format, running quantised models on CPU, CUDA, Metal, Vulkan, ROCm and Hexagon NPUs with no Python at runtime.
LMDeploy
open-sourceApache 2.0 LLM serving and quantization toolkit from Shanghai AI Laboratory's InternLM group. Builds for NVIDIA, Huawei Ascend, Cambricon and AMD ROCm from a single codebase.
vLLM
open-sourceOpen-source inference server for self-hosted LLMs. PagedAttention and continuous batching for high-concurrency serving across Nvidia, AMD, Intel and CPU backends.