ONNX Runtime
Microsoft's cross-platform inference engine for ONNX models, with execution providers for CUDA, TensorRT, OpenVINO, CoreML, NNAPI, QNN, DirectML and XNNPACK.
Overview
Microsoft open-sourced this in 2018 as the reference engine for the ONNX interchange format, and it has become the piece of infrastructure most edge teams end up standing on whether they planned to or not. The design is a small core plus a pluggable execution provider layer. You export a model once, then choose the backend at session creation: CUDA or TensorRT on NVIDIA, OpenVINO on Intel CPU, GPU and NPU, DirectML on Windows, CoreML on Apple silicon, NNAPI on Android, QNN on Qualcomm Hexagon, MIGraphX on AMD, XNNPACK for plain Arm and x86 CPU.
Quantisation runs through the ONNX QDQ representation, where QuantizeLinear and DequantizeLinear pairs sit in the graph and the execution provider decides how much of that it can fold into native INT8 or INT4 kernels. Static post-training quantisation with a calibration set is the usual path; INT4 weight-only is available for transformer blocks. ONNX Runtime GenAI sits on top for language model work, managing the decode loop, KV cache and beam search.
The honest limits are worth stating plainly. The abstraction leaks at operator level and coverage differs sharply between providers, so a graph that runs entirely on the accelerator under TensorRT may partition into a dozen CPU islands under NNAPI. Export from PyTorch remains the reliable source of pain. And for microcontroller-class hardware you want LiteRT for Microcontrollers or ExecuTorch instead; this is not a Cortex-M runtime.
Key Features
- ✓ One model file across many backends: CUDA, TensorRT, OpenVINO, DirectML, Android NNAPI, Qualcomm QNN, AMD MIGraphX, Vitis AI and XNNPACK all carry production status
- ✓ Post-training quantisation tooling for INT8 and INT4 weight-only, using QDQ nodes baked into the graph so the execution provider can absorb them into native kernels
- ✓ Bindings for C, C++, Python, C#, Java, JavaScript, Rust and Objective-C off the same core runtime
- ✓ ONNX Runtime Web with WebGPU and WASM backends for browser-side inference
- ✓ ONNX Runtime GenAI handling the LLM decode loop, KV cache and sampling so you are not writing that in C++ yourself
- ✓ Graph optimisation, operator fusion and compile-only sessions for offline engine builds in CI
Where it holds
- • The practical hedge against silicon churn. Swap the execution provider, keep the model and the application code
- • MIT licence with no runtime fee, so it clears legal review for shipped product without a conversation
- • Widest NPU coverage of any single runtime, with Qualcomm QNN and AMD Vitis AI at production status rather than preview
- • Predictable release train. 1.29.0 landed 12 August 2026 and older branches get patch servicing
Where it breaks
- • Behaviour diverges per execution provider. An op that runs on CUDA can silently fall back to CPU under CoreML or NNAPI and quietly destroy your latency budget
- • Getting a model out of PyTorch through the exporter is usually the longer half of the job. Unsupported ops and dynamic-shape breakage eat days
- • Preview providers (Rockchip NPU, Arm NN, TVM) are community-maintained and lag the core badly
- • Binary footprint is too heavy for MCU-class parts. This targets Linux, Android, Windows, macOS and the browser, not Cortex-M
My Take
Portability here is real rather than aspirational. The same .onnx file runs on CUDA, TensorRT, OpenVINO, NNAPI and Qualcomm QNN, all production-status providers, which matters the day a client swaps from an RK3588 board to a Jetson halfway through a programme. The trap is silent CPU fallback: an unsupported op does not error, it just partitions the graph and halves your frame rate, so profile per node before you trust any published number. Version 1.29.0 shipped 12 August 2026.
Quick Info
- Pricing:
- open-source
- Openness:
- Open source
- Licence:
- MIT
- Starting at:
- Free under the MIT licence. No paid tier, no account, no runtime royalty. Real costs are the vendor SDKs some execution providers pull in: CUDA and cuDNN for the CUDA EP, the Qualcomm AI Engine Direct SDK for QNN, the OpenVINO runtime for Intel. Version 1.29.0 published 12 August 2026 on a roughly quarterly cadence with patch branches in between.
- Added:
- Aug 2026
- Updated:
- Aug 2026
Use Cases
Judge it on your own work
The notes above say where ONNX Runtime holds and where it breaks. The fastest check is your own workload.
Visit website ↗Alternatives to ONNX Runtime
ncnn
open-sourceTencent's zero-dependency C++ inference library (腾讯 ncnn), BSD 3-Clause. Runs inside WeChat and QQ, with prebuilt libraries for HarmonyOS, WebAssembly, RISC-V and LoongArch.
MNN
open-sourceAlibaba's on-device inference engine (阿里 MNN), Apache 2.0. Runs Qwen3, Qwen3-VL and DeepSeek distills on phones across CPU, OpenCL, Metal, Vulkan, CUDA and Qualcomm Hexagon.
llama.cpp
open-sourceC/C++ LLM inference engine behind the GGUF format, running quantised models on CPU, CUDA, Metal, Vulkan, ROCm and Hexagon NPUs with no Python at runtime.
NVIDIA Jetson and TensorRT
paidNVIDIA's edge inference stack: Jetson modules from Orin Nano to Thor T5000, with TensorRT compiling models into hardware-specific FP4, FP8, INT8 and INT4 engines.