vLLM
Open-source inference server for self-hosted LLMs. PagedAttention and continuous batching for high-concurrency serving across Nvidia, AMD, Intel and CPU backends.
Overview
vLLM is an open-source inference server that started at UC Berkeley's Sky Computing Lab and now sits under the PyTorch Foundation on an Apache-2.0 licence. The core idea is PagedAttention, which manages KV cache memory the way an operating system manages RAM: in pages, with no requirement that one sequence occupy a contiguous block. Add continuous batching, where a new request joins the running batch instead of queueing behind it, and you get an engine whose throughput rises with load rather than collapsing under it.
The teams it fits are the ones that have already decided to run their own GPUs. That decision usually comes from one of three places: token spend that has outgrown an API bill, data that cannot legally or contractually leave a building, or a fine-tuned model nobody will host for you. Manufacturing, healthcare and anything defence-adjacent hit the second reason constantly. Because the server speaks the OpenAI format, application code written against a hosted API mostly points at it unchanged, which makes the migration a networking task rather than a rewrite.
Do not adopt it if nobody on the team is going to own the infrastructure. vLLM hands you a process that serves tokens well and nothing else: no scheduler, no scale-to-zero, no observability, no story for a node that dies mid-request. A small team with bursty traffic will spend more engineering hours than it saves in token cost, and is better off on a managed endpoint until volume justifies the switch. The honest test is whether you have someone who will be paged at 3am when a GPU falls out of the pool.
Key Features
- ✓ PagedAttention plus continuous batching, which is why throughput climbs with concurrency instead of flattening out at five simultaneous users
- ✓ OpenAI-compatible HTTP server, with an Anthropic Messages endpoint alongside it, so client code moves over with a base URL change
- ✓ Over 200 model architectures covered, including mixture-of-expert and multimodal families, usually within days of a weights drop
- ✓ Quantization across FP8, INT8, INT4, GPTQ and AWQ, plus FP8 KV cache to stretch context per GPU
- ✓ Tensor, pipeline and data parallelism for multi-node serving of large models
- ✓ Backends beyond Nvidia: AMD, Intel GPU, x86/ARM CPU, with plugins for Google TPU, Intel Gaudi, Huawei Ascend and Apple Silicon
Where it holds
- • No compile step. A new model is serving in minutes, where TensorRT-LLM can take one to two weeks of engine-build and tuning work before it returns a single token
- • Hardware portability is real and it matters commercially. If you are negotiating GPU supply, running an engine that works on AMD and Intel gives you a credible second source
- • Release cadence is fast: 0.28.0 landed 26 August 2026, and new architectures normally appear days after the weights do
- • Apache-2.0 under PyTorch Foundation governance, with over 2,000 contributors. No licence rug-pull risk
Where it breaks
- • It does not win on peak throughput. SGLang is roughly 29% faster on prefix-heavy traffic thanks to RadixAttention, and TensorRT-LLM takes 15 to 30% more out of an H100
- • The defaults leave performance on the table. Getting max-num-seqs, KV cache fraction and chunked prefill right needs a real benchmarking pass against your own traffic shape
- • Version churn is high and config flags move between minor releases, so pin your version and read the changelog before upgrading
- • You get a serving process, not a platform. Autoscaling, model warm-up, health checks and GPU failure handling are all yours to build
My Take
Most teams pick a serving engine on peak benchmark numbers, which is the wrong instinct. What actually matters is which one survives six model swaps and a GPU vendor change. Version 0.28.0 shipped 26 August 2026 covering over 200 model architectures across Nvidia, AMD, Intel and CPU backends with no compile step in the way. It gives up real performance for that flexibility: SGLang runs about 29% faster on prefix-heavy traffic and TensorRT-LLM squeezes 15 to 30% more throughput from an H100, though TensorRT charges you one to two weeks of setup and total vendor lock-in for the privilege.
Quick Info
- Pricing:
- open-source
- Openness:
- Open source
- Licence:
- Apache 2.0
- Starting at:
- Free, Apache-2.0. You pay only for GPUs. Rented H100 capacity runs roughly $2 to $4 per GPU-hour depending on provider, against $0.05 to $3.00 per million tokens on hosted APIs, so the crossover point for most workloads sits somewhere past $8k to $10k of monthly token spend. Commercial support is sold separately through vendor distributions such as Red Hat AI Inference Server.
- Added:
- Aug 2026
- Updated:
- Aug 2026
Use Cases
Judge it on your own work
The notes above say where vLLM holds and where it breaks. The fastest check is your own workload.
Visit website ↗vLLM compared head to head
Alternatives to vLLM
LMDeploy
open-sourceApache 2.0 LLM serving and quantization toolkit from Shanghai AI Laboratory's InternLM group. Builds for NVIDIA, Huawei Ascend, Cambricon and AMD ROCm from a single codebase.
llama.cpp
open-sourceC/C++ LLM inference engine behind the GGUF format, running quantised models on CPU, CUDA, Metal, Vulkan, ROCm and Hexagon NPUs with no Python at runtime.
Ollama
freemiumLocal LLM runtime with a one-command install and an OpenAI-compatible server on localhost. For laptop development, edge deployment and air-gapped environments.
Groq
freemiumInference API running open-weights models on custom LPU silicon. Sub-second responses and $0.05 per million input tokens, for latency-bound voice and agent workloads.