TensorRT-LLM is an open-source inference library from NVIDIA that compiles and optimises LLMs specifically for NVIDIA GPUs, exposing a Python API that wraps CUDA kernels, fused attention operators, and graph-level compiler passes from the TensorRT framework. It supports Quantization (INT8, FP8, INT4 weight-only), tensor and pipeline parallelism across multiple GPUs, and techniques like in-flight batching and speculative decoding, all aimed at maximising tokens-per-second throughput while minimising latency per request during Inference.

Think of it like a master machinist taking a standard engine blueprint and re-machining every component to the tightest possible tolerances for a specific race track. The engine still runs the same race, but every friction loss has been eliminated for that exact track surface.

import tensorrt_llm
from tensorrt_llm.builder import BuildConfig
# Convert a Llama-3 checkpoint to a TRT-LLM engine
build_config = BuildConfig(max_batch_size=32, max_input_len=2048,
                            max_output_len=512, strongly_typed=True)
engine = tensorrt_llm.build(model_dir="./llama3-8b", build_config=build_config)
engine.save("./trt_engine")

The compiled engine is GPU-architecture-specific but runs significantly faster than a vanilla PyTorch checkpoint on the same hardware.

TensorRT-LLM integrates naturally with NVIDIA Triton Inference Server for production-grade serving and is the backend underpinning NVIDIA's NIM microservices. Its optimisation pipeline overlaps conceptually with Knowledge Distillation (making models smaller) and Quantization (reducing numerical precision), both of which TRT-LLM can apply jointly. Understanding Parameter counts and Transformer architecture shapes is a prerequisite for tuning build configurations effectively.

You will encounter TensorRT-LLM when deploying open-weight models on NVIDIA data-centre hardware, benchmarking inference throughput, or reading NVIDIA's official performance numbers for models like Llama or Mixtral.