LMDeploy

Shanghai AI Laboratory's InternLM group maintains LMDeploy as the compression and serving toolkit for its own models, and it has long since outgrown that origin. Two engines ship in the same package. TurboMind is the C++ and CUDA implementation built for raw throughput, carrying hand-written kernels including SM90 native BF16 and FP8 GEMM paths. The PyTorch engine is pure Python and exists so that supporting a new architecture does not require kernel work first. Quantization covers AWQ weight-only 4-bit, K/V cache quantization, FP8 for MoE layers and MXFP4 on recent NVIDIA silicon.

The reason it matters outside the InternLM ecosystem is hardware. Set LMDEPLOY_TARGET_DEVICE at build time and the same codebase targets NVIDIA CUDA, Huawei Ascend, Cambricon or AMD ROCm. For any organisation in China operating under domestic-chip procurement rules, that one environment variable is the difference between a working deployment and a rewrite, and no Western serving stack offers an equivalent. The licence is plain Apache 2.0 with no field-of-use restriction and no user cap, so it can go into a commercial product without a legal review cycle.

Judge the performance claims yourself. The README states up to 1.8 times the request throughput of vLLM and 2.4 times FP16 speed at 4-bit, both from the project's own benchmarks, and that gap narrows or reverses depending on model shape and batch profile. Community size is the real trade-off: 8,030 stars against a vastly larger vLLM ecosystem means fewer integrations and thinner third-party answers when something breaks. v0.16.0 arrived 19 August 2026 with GLM-5.2 support, TurboMind ViT paths for InternVL and Qwen-VL plus the SM90 kernels, following v0.15.0 on 31 July with DeepSeek V4 support. English documentation sits on readthedocs and is adequate rather than generous.