NVIDIA Jetson and TensorRT
NVIDIA's edge inference stack: Jetson modules from Orin Nano to Thor T5000, with TensorRT compiling models into hardware-specific FP4, FP8, INT8 and INT4 engines.
Overview
Two things that get bought together in practice, because you rarely take one without the other. Jetson is the module family: Orin Nano and NX at the cheap end, AGX Orin in the middle, and since 2025 the Thor line built on Blackwell. The T5000 pairs that GPU with a 14-core Arm Neoverse-V3AE CPU and 128 GB of LPDDR5X, quoted at up to 2070 FP4 teraflops inside 130 W, which is a different class of machine from what edge meant three years ago.
TensorRT is the compiler half. You hand it an ONNX graph and a target, it autotunes kernels against that exact GPU, fuses layers, selects precisions and emits a serialised engine. Quantisation runs through TensorRT Model Optimizer, a free Apache-licensed library covering FP8, FP4, INT8, INT4 and AWQ, with explicit Q/DQ nodes in the graph rather than the old calibrator flow. JetPack 7.2.1 pins the versions together (Jetson Linux 39.2.1, CUDA 13.2.1, TensorRT 10.16.2) so you are not resolving CUDA conflicts by hand on a device with no display.
The downside is that none of it travels. An engine file is bound to its architecture and TensorRT version, plugins are written against NVIDIA's C++ API, and the whole thing assumes CUDA underneath. Teams wanting a hedge normally keep an ONNX Runtime path alongside and accept the performance loss on non-NVIDIA silicon. Pricing also moved against small projects this year, with module and dev kit increases of up to 101% announced in July 2026.
Key Features
- ✓ Module range from Orin Nano Super at 67 TOPS up to Thor T5000: Blackwell GPU, 14-core Arm Neoverse-V3AE, 128 GB LPDDR5X, up to 2070 FP4 teraflops inside a 130 W envelope
- ✓ TensorRT compiles an ONNX graph into a serialised engine tuned to one specific GPU, with kernel autotuning, layer fusion and per-layer precision selection
- ✓ Precisions down to FP4 on Blackwell, plus FP8, INT8 and INT4 weight-only through TensorRT Model Optimizer with AWQ and SmoothQuant
- ✓ JetPack 7.2.1 ships Jetson Linux 39.2.1, CUDA 13.2.1 and TensorRT 10.16.2 as a version-matched set
- ✓ DeepStream for multi-camera video analytics and Holoscan for low-latency sensor pipelines on the same modules
- ✓ TensorRT-LLM under Apache 2.0 for transformer serving, sitting alongside the proprietary core SDK
Where it holds
- • Still the fastest route from a trained model to real throughput inside a fixed power budget, and by a wide margin at the top end
- • One software stack from a $399 dev kit to a datacentre GPU, so prototyping and production share tooling and skills
- • Ecosystem depth around Shenzhen is unmatched. Camera drivers, carrier boards and ODM support are mature in a way no competing edge platform can claim
Where it breaks
- • Engines do not travel. A plan built for Orin will not load on Thor, or across a TensorRT minor version, so a mixed fleet needs one build pipeline per target SKU
- • The 11.x line removed IInt8Calibrator, setDynamicRange, weakly-typed networks and IPluginV2 outright, so old calibration scripts and custom plugins need rewriting rather than upgrading
- • Complete lock-in. Nothing in the engine format or the plugin API transfers to another vendor, and the fallback is a slower ONNX Runtime path
- • The July 2026 price rises hit the low end hardest. A 60% jump on the Orin Nano Super changes the arithmetic on low-volume and hobby products
My Take
Performance is not the argument here, nobody seriously disputes it. Cost of ownership is. Engines compile per GPU architecture, per TensorRT version and effectively per driver, so a fleet mixing Orin Nano, Orin NX and Thor needs three build pipelines and a versioned artefact store, which is real infrastructure nobody budgets for. Then the 11.x line deleted the INT8 calibrator API and the whole implicit-calibration path, pushing everyone onto explicit Q/DQ graphs; 11.2 reached GA on 4 August 2026. Still the right pick for robotics and machine vision at the edge, but price in the migration tax.
Quick Info
- Pricing:
- paid
- Openness:
- Proprietary
- Starting at:
- TensorRT, JetPack and TensorRT Model Optimizer are free downloads under NVIDIA's SDK licence; the cost is silicon, and it moved sharply this year. NVIDIA raised Jetson module and dev kit prices by up to 101% in July 2026: the Orin Nano Super dev kit went from $249 to $399, and the AGX Thor dev kit launched at $3,499 but is listed at $5,499 on NVIDIA's own marketplace as of August 2026. TensorRT-LLM and Model Optimizer are separately Apache 2.0 on GitHub.
- Added:
- Aug 2026
- Updated:
- Aug 2026
Use Cases
Judge it on your own work
The notes above say where NVIDIA Jetson and TensorRT holds and where it breaks. The fastest check is your own workload.
Visit website ↗Alternatives to NVIDIA Jetson and TensorRT
ONNX Runtime
open-sourceMicrosoft's cross-platform inference engine for ONNX models, with execution providers for CUDA, TensorRT, OpenVINO, CoreML, NNAPI, QNN, DirectML and XNNPACK.
ncnn
open-sourceTencent's zero-dependency C++ inference library (腾讯 ncnn), BSD 3-Clause. Runs inside WeChat and QQ, with prebuilt libraries for HarmonyOS, WebAssembly, RISC-V and LoongArch.
MNN
open-sourceAlibaba's on-device inference engine (阿里 MNN), Apache 2.0. Runs Qwen3, Qwen3-VL and DeepSeek distills on phones across CPU, OpenCL, Metal, Vulkan, CUDA and Qualcomm Hexagon.
Edge Impulse
freemiumEdge AI MLOps platform for MCU and NPU targets, covering sensor data capture, DSP blocks, training and C++ export with the EON Compiler. Qualcomm-owned since 2025.