llama.cpp
C/C++ LLM inference engine behind the GGUF format, running quantised models on CPU, CUDA, Metal, Vulkan, ROCm and Hexagon NPUs with no Python at runtime.
Overview
Georgi Gerganov started this as a weekend port of LLaMA inference to plain C++ in March 2023. It is now the default way to run a language model on hardware you actually own. The engine sits on ggml, a compact tensor library with its own memory model, and distributes models as GGUF: a single file holding weights, tokenizer and metadata, which removes the entire Python dependency graph from the deployment target.
Quantisation is where the practical value lives. The k-quant family from Q4_K_M through Q6_K is the usual working range, trading a few points of quality for a three to four times memory cut, and the IQ importance-matrix quants push below 3 bits when memory is the binding constraint. Backend coverage is unusually broad for a community project: CUDA, HIP for AMD, Metal for Apple silicon, Vulkan as the vendor-neutral fallback, SYCL for Intel and a Hexagon HTP path for Qualcomm NPUs. Crossing 100,000 GitHub stars in March 2026 is a vanity number, but the merge volume behind it is not.
Pick it for edge devices, laptops, single-user assistants and anything air-gapped. Do not pick it as a multi-tenant serving layer. And treat published quantisation benchmarks with some suspicion, since perplexity holds up better than instruction-following under aggressive quantisation, and small MoE models in particular can fall apart at Q4 in ways a perplexity table will never show you.
Key Features
- ✓ GGUF single-file format carrying weights, tokenizer and metadata together, with k-quants (Q4_K_M, Q5_K_M, Q6_K), Q8_0 and IQ importance-matrix quants below 3 bits
- ✓ Backends for CPU (AVX2, AVX-512, Arm NEON and SVE), CUDA, HIP/ROCm, Metal, Vulkan, SYCL and Qualcomm Hexagon HTP
- ✓ llama-server with an OpenAI-compatible HTTP API, per-slot context sizing and continuous batching
- ✓ Partial GPU offload by layer count, so a model larger than available VRAM still runs at a speed cost rather than failing
- ✓ GBNF grammar-constrained sampling for forcing valid JSON out of small models that would otherwise wander
- ✓ Builds for Android, Raspberry Pi, Jetson and desktop from one tree with no Python dependency on the target
Where it holds
- • A static binary with no dependency graph. Cross-compiles for Arm SBCs and Android without dragging a Python stack onto the device
- • New model architectures land within days. More than 700 contributors merged over 3,800 pull requests in 2025
- • GGUF is now the de facto distribution format for quantised weights, so community quants appear almost immediately after a release
- • No telemetry and an MIT licence, which clears procurement for on-premise and air-gapped deployments
Where it breaks
- • Throughput under concurrent load sits well behind vLLM and TensorRT-LLM. This is a single-stream engine that grew batching afterwards, not a serving system
- • No LTS branches. Build tags roll several times a day, so an unpinned build will drift under you between images
- • Quantisation quality varies a lot by architecture. Mixture-of-experts and newer attention variants degrade more at Q4 than perplexity tables suggest
- • Backend parity is uneven. Vulkan and SYCL trail CUDA and Metal on new operator coverage by weeks or months
My Take
Nothing else gets a 7B model onto a Jetson or an Android handset with this little ceremony. One binary, one GGUF file, done. Where it loses is concurrency: put twenty simultaneous users on llama-server and vLLM will beat it comfortably on throughput, because paged attention and continuous batching were designed in there from the start rather than retrofitted. Also worth knowing that releases are build tags, not versions, and the tree was around b10668 in late August 2026, so pin one or your container image will move without you noticing.
Quick Info
- Pricing:
- open-source
- Openness:
- Open source
- Licence:
- MIT
- Starting at:
- Free, MIT licensed, no hosted service and no commercial tier. Releases are sequential build tags rather than semantic versions (the tree was in the b10600s on 28 August 2026), published sometimes several times a day, so pin a build number in CI instead of tracking a branch.
- Added:
- Aug 2026
- Updated:
- Aug 2026
Use Cases
Judge it on your own work
The notes above say where llama.cpp holds and where it breaks. The fastest check is your own workload.
Visit website ↗Alternatives to llama.cpp
Ollama
freemiumLocal LLM runtime with a one-command install and an OpenAI-compatible server on localhost. For laptop development, edge deployment and air-gapped environments.
vLLM
open-sourceOpen-source inference server for self-hosted LLMs. PagedAttention and continuous batching for high-concurrency serving across Nvidia, AMD, Intel and CPU backends.
LMDeploy
open-sourceApache 2.0 LLM serving and quantization toolkit from Shanghai AI Laboratory's InternLM group. Builds for NVIDIA, Huawei Ascend, Cambricon and AMD ROCm from a single codebase.
MNN
open-sourceAlibaba's on-device inference engine (阿里 MNN), Apache 2.0. Runs Qwen3, Qwen3-VL and DeepSeek distills on phones across CPU, OpenCL, Metal, Vulkan, CUDA and Qualcomm Hexagon.