TensorRT-LLM
NVIDIA's inference stack for LLMs, optimized for NVIDIA GPUs. Includes quantization, speculative decoding, and multi-GPU tensor parallelism. Fastest option for open-weights inference on NVIDIA hardware; less flexible on other accelerators.
Related terms
- vLLM
A high-throughput open-source LLM inference and serving stack developed at UC Berkeley. Introduced PagedAttention (KV cache paging); the reference implementation for efficient open-weights model serving. Widely used in enterprise self-hosted deployments.
- Speculative Decoding
An inference-time optimization that uses a fast draft model to propose tokens which the target model then verifies in parallel, reducing latency without changing output quality. Ships in vLLM, TensorRT-LLM, and most frontier provider stacks.
- Quantization
Reducing the numeric precision of model weights (from FP16/BF16 to INT8, INT4, or lower) to shrink memory and speed inference, usually with a small quality loss. Common formats: GGUF, AWQ, GPTQ. Enables running large open-weights models on consumer hardware.
Related on this site
Where this fits
TensorRT-LLM is part of the Data & Infrastructure vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.