MNN
Alibaba's on-device inference engine (阿里 MNN), Apache 2.0. Runs Qwen3, Qwen3-VL and DeepSeek distills on phones across CPU, OpenCL, Metal, Vulkan, CUDA and Qualcomm Hexagon.
Overview
Alibaba open sourced MNN in 2019 after building it to serve inference inside Taobao, Tmall and Alipay, and that provenance shows in the code. The ARM kernels are tuned for handsets people actually own rather than for benchmark rigs. The licence is Apache 2.0, with no user threshold and no field-of-use restriction, which means it goes into a commercial product without a legal conversation.
The 2026 releases have been squarely aimed at on-device language models. Version 3.5.0 in April added Vulkan LLM inference, TurboQuant TQ3 and TQ4 KV cache quantization, multi-turn prompt caching and RISC-V vector support. Version 3.6.1 on 22 July added a Qualcomm Hexagon backend and reported Qwen3-0.6B at W4 symmetric block64 reaching 2667 tokens per second of prefill on a Snapdragon 8 Elite, close to eight times the CPU figure, alongside large Metal and OpenCL gains. Converted weights for Qwen3, Qwen3-VL, Qwen3-Coder-30B-A3B and Qwen3.5 live under the taobao-mnn organisation on Hugging Face as well as ModelScope, so pulling them from outside China needs no special account.
Two things to plan for. The documentation split is real: build instructions and the C++ and Java APIs exist in English, while conversion, quantization and backend tuning are Chinese, as is most issue discussion. And backend maturity varies, which the project states openly by grading each hardware and precision combination from S down to C in its own README. There is a HiAI path for Kirin NPUs on phones but nothing for Ascend Atlas modules, so hardware built on Huawei's datacentre and edge silicon needs a different runtime.
Key Features
- ✓ Apache 2.0 with no user cap and no field-of-use clause, so it ships inside a commercial app without a separate agreement
- ✓ Backends for ARM CPU, x86 AVX512, OpenCL, Vulkan, Metal, CUDA, CoreML, NNAPI, Huawei HiAI and Qualcomm QNN with Hexagon DSP
- ✓ Version 3.6.1 (22 July 2026) added the Hexagon backend: Qwen3-0.6B at W4 symmetric block64 reports 2667 tok/s prefill on Snapdragon 8 Elite, about 7.9x the CPU path
- ✓ Version 3.5.0 (April 2026) added Vulkan LLM inference, TurboQuant TQ3 and TQ4 KV cache quantization, multi-turn prompt caching, a pipelined sampler and a RISC-V vector path
- ✓ Converted weights for Qwen3, Qwen3-VL, Qwen3-Coder-30B-A3B and Qwen3.5 published under the taobao-mnn organisation on both Hugging Face and ModelScope
- ✓ On-device voice stack with async Token2Wav pipeline, acoustic echo cancellation and barge-in, plus Android and iOS chat apps as reference integrations
Where it holds
- • Weights are mirrored on Hugging Face, not ModelScope only, so no mainland phone number or account is needed to pull them
- • NPU coverage is genuinely wide for one engine: QNN and Hexagon, NNAPI, CoreML and Huawei HiAI, which no Western on-device runtime matches
- • Release notes publish per-backend performance deltas version to version, so you can tell whether an upgrade helps your target
- • Permissive licensing plus a CMake build that drops into an existing Gradle or Xcode project without vendor SDKs
Where it breaks
- • Documentation is Chinese-first. English pages cover building and the basic APIs, while conversion, quantization and backend tuning material is Chinese only
- • Model conversion is its own skill. Anything outside the published zoo means working through MNNConvert and operator coverage gaps
- • Backend quality is uneven and the README grades each combination S through C, with NNAPI near the bottom, so per-device benchmarking is unavoidable
- • Kirin NPUs are reachable through HiAI but there is no Ascend backend for Atlas modules. For Ascend you are on MindSpore Lite or CANN
My Take
Few inference engines let you target a Snapdragon Hexagon DSP, a Kirin NPU and an Apple Neural Engine from one graph, and this one does it under a plain Apache 2.0 licence with no user cap attached. The 3.6.1 release in July 2026 reported 2667 tokens per second of prefill for Qwen3-0.6B on a Snapdragon 8 Elite, roughly 7.9 times the CPU path, which is the number that decides whether an on-device assistant feels alive or sluggish. Documentation is the tax you pay: English covers the basics, everything past that is Chinese. Budget real time for MNNConvert if your model is not already sitting in the taobao-mnn zoo.
Quick Info
- Pricing:
- open-source
- Openness:
- Open source
- Licence:
- Apache 2.0
- Starting at:
- Free and Apache 2.0. No paid tier, no registration, no telemetry. Costs are engineering time: model conversion with MNNConvert, per-device benchmarking and quantization tuning. Converted weights are free to download from Hugging Face and ModelScope.
- Added:
- Aug 2026
- Updated:
- Aug 2026
Use Cases
Judge it on your own work
The notes above say where MNN holds and where it breaks. The fastest check is your own workload.
Visit website ↗Alternatives to MNN
ncnn
open-sourceTencent's zero-dependency C++ inference library (腾讯 ncnn), BSD 3-Clause. Runs inside WeChat and QQ, with prebuilt libraries for HarmonyOS, WebAssembly, RISC-V and LoongArch.
llama.cpp
open-sourceC/C++ LLM inference engine behind the GGUF format, running quantised models on CPU, CUDA, Metal, Vulkan, ROCm and Hexagon NPUs with no Python at runtime.
ONNX Runtime
open-sourceMicrosoft's cross-platform inference engine for ONNX models, with execution providers for CUDA, TensorRT, OpenVINO, CoreML, NNAPI, QNN, DirectML and XNNPACK.
Ollama
freemiumLocal LLM runtime with a one-command install and an OpenAI-compatible server on localhost. For laptop development, edge deployment and air-gapped environments.