ONNX Runtime
Open Neural Network Exchange runtime — a cross-platform inference engine. Supports many hardware backends. Used for cross-provider portability of models and for on-device deployment where llama.cpp does not fit.
Related terms
- vLLM
A high-throughput open-source LLM inference and serving stack developed at UC Berkeley. Introduced PagedAttention (KV cache paging); the reference implementation for efficient open-weights model serving. Widely used in enterprise self-hosted deployments.
- Quantization
Reducing the numeric precision of model weights (from FP16/BF16 to INT8, INT4, or lower) to shrink memory and speed inference, usually with a small quality loss. Common formats: GGUF, AWQ, GPTQ. Enables running large open-weights models on consumer hardware.
Related on this site
Where this fits
ONNX Runtime is part of the Data & Infrastructure vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.