The Triton backend for TensorRT-LLM. You can learn more about Triton backends in the backend repo. The goal of TensorRT-LLM Backend is to let you serve TensorRT-LLM models with Triton Inference Server. The inflight_batcher_llm directory contains the C++ implementation of the backend supporting inflight batching, paged attention and more.
Note
Please note that the Triton backend source code and test have been moved
to TensorRT-LLM under the
triton_backend directory.
Where can I ask general questions about Triton and Triton backends? Be sure to read all the information below as well as the general Triton documentation available in the main server repo. If you don't find your answer there you can ask questions on the issues page.
- TensorRT-LLM Backend
- Table of Contents
- Getting Started
- Building from Source
- Supported Models
- Model Config
- Model Deployment
- Launch Triton server within Slurm based clusters
- Triton Metrics
- Benchmarking
- Testing the TensorRT-LLM Backend
Serve any HuggingFace model directly — no engine compilation required.
docker run --rm -it --net host --shm-size=2g --ulimit memlock=-1 --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
nvcr.io/nvidia/tritonserver:25.12-trtllm-python-py3 bashReplace 25.12 with the latest tag from NGC.
git clone https://github.com/NVIDIA/TensorRT-LLM.gitEdit TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml
and set model: to any HuggingFace model ID or local path, for example:
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0All keys in model.yaml map directly to LLM() constructor arguments.
This is where you configure KV cache, quantization, parallelism, and more.
For gated models (e.g. Llama), set your token first: export HF_TOKEN=hf_...
Important
Run from the directory where you ran git clone (the parent of TensorRT-LLM/),
not from inside the TensorRT-LLM/ folder. Running from inside it causes
ModuleNotFoundError: No module named 'tensorrt_llm.bindings'.
python3 TensorRT-LLM/triton_backend/scripts/launch_triton_server.py \
--model_repo=TensorRT-LLM/triton_backend/all_models/llmapi/Once the server is up, send a request:
curl -X POST localhost:8000/v2/models/tensorrt_llm/generate \
-d '{"text_input": "The future of AI is", "sampling_param_max_tokens": 50}' | jqSend a second request with "stop": true and the same request_id you used in the original request:
Start a long-running request and note its request_id:
curl -X POST localhost:8000/v2/models/tensorrt_llm/generate \
-H "triton-request-id: my-req-1" \
-d '{"text_input": "Write a very long essay about the history of AI", "sampling_param_max_tokens": 500}' &Cancel it:
curl -X POST localhost:8000/v2/models/tensorrt_llm/generate \
-H "triton-request-id: my-req-1" \
-d '{"text_input": "", "stop": true}'The server stops generation immediately and returns a cancellation response.
For multi-GPU, multi-node, and advanced options see docs/llmapi.md.
Please refer to the build.md for more details on how to build the Triton TRT-LLM container from source.
Only a few examples are listed here. For all the supported models, please refer to the support matrix.
-
LLaMa
-
Gemma
-
Mistral
-
Multi-modal
-
Encoder-Decoder
Please refer to the model config for more details on the model configuration.
TensorRT-LLM backend relies on MPI to coordinate the execution of a model across multiple GPUs and nodes. Currently, there are two different modes supported to run a model across multiple GPUs, Leader Mode and Orchestrator Mode.
Note: This is different from the model multi-instance support from Triton Server which allows multiple instances of a model to be run on the same or different GPUs. For more information on Triton Server multi-instance support, please refer to the Triton model config documentation.
In leader mode, TensorRT-LLM backend spawns one Triton Server process for every
GPU. The process with rank 0 is the leader process. Other Triton Server processes,
do not return from the TRITONBACKEND_ModelInstanceInitialize call to avoid
port collision and allowing the other processes to receive requests.
The overview of this mode is described in the diagram below:
This mode is friendly with slurm deployments since it doesn't use MPI_Comm_spawn.
In orchestrator mode, the TensorRT-LLM backend spawns a single Triton Server
process that acts as an orchestrator and spawns one Triton Server process for
every GPU that each model requires. This mode is mainly used when serving
multiple models with TensorRT-LLM backend. In this mode, the MPI world size
must be one as TRT-LLM backend will automatically create new workers as needed.
The overview of this mode is described in the diagram below:
Since this mode uses MPI_Comm_spawn, it might not work properly with slurm deployments. Additionally, this currently only works for single node deployments.
Please refer to Running Multiple Instances of the LLaMa Model for more information on running multiple instances of LLaMa model in different configurations.
Check out the Multi-Node Generative AI w/ Triton Server and TensorRT-LLM tutorial for Triton Server and TensorRT-LLM multi-node deployment.
Tensor Parallelism, Pipeline Parallelism and Expert parallelism are supported in TensorRT-LLM.
With the LLM API, parallelism is configured directly in model.yaml; the keys
map to the LLM() constructor arguments.
Some examples are shown below:
- Serve LLaMA v3 70B using 4-way tensor parallelism and 2-way pipeline parallelism.
model: meta-llama/Meta-Llama-3-70B
tensor_parallel_size: 4
pipeline_parallel_size: 2- Serve Mixtral 8x22B with tensor parallelism and expert parallelism, using the
Mixture of Experts (MoE) parallelism keys
moe_expert_parallel_sizeandmoe_tensor_parallel_size.
model: mistralai/Mixtral-8x22B-v0.1
tensor_parallel_size: 8
moe_expert_parallel_size: 4
moe_tensor_parallel_size: 2See the LLM API reference to learn more about how TensorRT-LLM supports expert parallelism in Mixture of Experts (MoE).
See the MIG tutorial for more details on how to run TRT-LLM models and Triton with MIG.
The scheduler policy helps the batch manager adjust how requests are scheduled
for execution. There are two scheduler policies supported in TensorRT-LLM,
MAX_UTILIZATION and GUARANTEED_NO_EVICT. You can specify the scheduler
policy via the batch_scheduler_policy parameter in the
model config of tensorrt_llm model.
See the
KV Cache
section for more details on how TensorRT-LLM supports KV cache. Also, check out
the KV Cache Reuse
documentation to learn more about how to enable KV cache reuse. KV cache options
are configured through the LLM API
in model.yaml.
TensorRT-LLM supports various decoding modes, including top-k, top-p, top-k top-p, beam search Medusa, ReDrafter, Lookahead and Eagle. See the Sampling Parameters section to learn more about top-k, top-p, top-k top-p and beam search decoding. Please refer to the speculative decoding documentation for more details on Medusa, ReDrafter, Lookahead and Eagle.
Parameters for decoding modes can be found in the model config of tensorrt_llm model.
See the Speculative Decoding documentation to learn more about how TensorRT-LLM supports speculative decoding to improve the performance. The parameters for speculative decoding can be found in the model config of tensorrt_llm_bls model.
For more details on how to use chunked context, please refer to the Chunked Context section. Parameters for chunked context can be found in the model config of tensorrt_llm model.
Check out the Quantization Guide to learn more about how to install the quantization toolkit and quantize TensorRT-LLM models. Also, check out the blog post Speed up inference with SOTA quantization techniques in TRT-LLM to learn more about how to speed up inference with quantization.
Refer to lora.md for more details on how to use LoRa with TensorRT-LLM and Triton.
tensorrt_llm_triton.sub
#!/bin/bash
#SBATCH -o logs/tensorrt_llm.out
#SBATCH -e logs/tensorrt_llm.error
#SBATCH -J <REPLACE WITH YOUR JOB's NAME>
#SBATCH -A <REPLACE WITH YOUR ACCOUNT's NAME>
#SBATCH -p <REPLACE WITH YOUR PARTITION's NAME>
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=00:30:00
sudo nvidia-smi -lgc 1410,1410
srun --mpi=pmix \
--container-image triton_trt_llm \
--container-workdir /tensorrtllm_backend \
--output logs/tensorrt_llm_%t.out \
bash /tensorrtllm_backend/tensorrt_llm_triton.shtensorrt_llm_triton.sh
TRITONSERVER="/opt/tritonserver/bin/tritonserver"
MODEL_REPO="/triton_model_repo"
${TRITONSERVER} --model-repository=${MODEL_REPO} --disable-auto-complete-config --backend-config=python,shm-region-prefix-name=prefix${SLURM_PROCID}_If srun initializes the mpi environment, you can use the following command to launch the Triton server:
srun --mpi pmix launch_triton_server.py --oversubscribesbatch tensorrt_llm_triton.subYou might have to contact your cluster's administrator to help you customize the above script.
Starting with the 23.11 release of Triton, users can now obtain TRT LLM Batch Manager statistics by querying the Triton metrics endpoint. This can be accomplished by launching a Triton server in any of the ways described above (ensuring the build code / container is 23.11 or later) and querying the server. Upon receiving a successful response, you can query the metrics endpoint by entering the following:
curl localhost:8002/metricsBatch manager statistics are reported by the metrics endpoint in fields that
are prefixed with nv_trt_llm_. Your output for these fields should look
similar to the following (assuming your model is an inflight batcher model):
# HELP nv_trt_llm_request_metrics TRT LLM request metrics
# TYPE nv_trt_llm_request_metrics gauge
nv_trt_llm_request_metrics{model="tensorrt_llm",request_type="waiting",version="1"} 1
nv_trt_llm_request_metrics{model="tensorrt_llm",request_type="context",version="1"} 1
nv_trt_llm_request_metrics{model="tensorrt_llm",request_type="scheduled",version="1"} 1
nv_trt_llm_request_metrics{model="tensorrt_llm",request_type="max",version="1"} 512
nv_trt_llm_request_metrics{model="tensorrt_llm",request_type="active",version="1"} 0
# HELP nv_trt_llm_runtime_memory_metrics TRT LLM runtime memory metrics
# TYPE nv_trt_llm_runtime_memory_metrics gauge
nv_trt_llm_runtime_memory_metrics{memory_type="pinned",model="tensorrt_llm",version="1"} 0
nv_trt_llm_runtime_memory_metrics{memory_type="gpu",model="tensorrt_llm",version="1"} 1610236
nv_trt_llm_runtime_memory_metrics{memory_type="cpu",model="tensorrt_llm",version="1"} 0
# HELP nv_trt_llm_kv_cache_block_metrics TRT LLM KV cache block metrics
# TYPE nv_trt_llm_kv_cache_block_metrics gauge
nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type="fraction",model="tensorrt_llm",version="1"} 0.4875
nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type="tokens_per",model="tensorrt_llm",version="1"} 64
nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type="used",model="tensorrt_llm",version="1"} 1
nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type="free",model="tensorrt_llm",version="1"} 6239
nv_trt_llm_kv_cache_block_metrics{kv_cache_block_type="max",model="tensorrt_llm",version="1"} 6239
# HELP nv_trt_llm_inflight_batcher_metrics TRT LLM inflight_batcher-specific metrics
# TYPE nv_trt_llm_inflight_batcher_metrics gauge
nv_trt_llm_inflight_batcher_metrics{inflight_batcher_specific_metric="micro_batch_id",model="tensorrt_llm",version="1"} 0
nv_trt_llm_inflight_batcher_metrics{inflight_batcher_specific_metric="generation_requests",model="tensorrt_llm",version="1"} 0
nv_trt_llm_inflight_batcher_metrics{inflight_batcher_specific_metric="total_context_tokens",model="tensorrt_llm",version="1"} 0
# HELP nv_trt_llm_general_metrics General TRT LLM metrics
# TYPE nv_trt_llm_general_metrics gauge
nv_trt_llm_general_metrics{general_type="iteration_counter",model="tensorrt_llm",version="1"} 0
nv_trt_llm_general_metrics{general_type="timestamp",model="tensorrt_llm",version="1"} 1700074049
# HELP nv_trt_llm_disaggregated_serving_metrics TRT LLM disaggregated serving metrics
# TYPE nv_trt_llm_disaggregated_serving_metrics counter
nv_trt_llm_disaggregated_serving_metrics{disaggregated_serving_type="kv_cache_transfer_ms",model="tensorrt_llm",version="1"} 0
nv_trt_llm_disaggregated_serving_metrics{disaggregated_serving_type="request_count",model="tensorrt_llm",version="1"} 0If, instead, you launched a V1 model, your output will look similar to the output above except the inflight batcher related fields will be replaced with something similar to the following:
# HELP nv_trt_llm_v1_metrics TRT LLM v1-specific metrics
# TYPE nv_trt_llm_v1_metrics gauge
nv_trt_llm_v1_metrics{model="tensorrt_llm",v1_specific_metric="total_generation_tokens",version="1"} 20
nv_trt_llm_v1_metrics{model="tensorrt_llm",v1_specific_metric="empty_generation_slots",version="1"} 0
nv_trt_llm_v1_metrics{model="tensorrt_llm",v1_specific_metric="total_context_tokens",version="1"} 5Please note that versions of Triton prior to the 23.12 release do not support base Triton metrics. As such, the following fields will report 0:
# HELP nv_inference_request_success Number of successful inference requests, all batch sizes
# TYPE nv_inference_request_success counter
nv_inference_request_success{model="tensorrt_llm",version="1"} 0
# HELP nv_inference_request_failure Number of failed inference requests, all batch sizes
# TYPE nv_inference_request_failure counter
nv_inference_request_failure{model="tensorrt_llm",version="1"} 0
# HELP nv_inference_count Number of inferences performed (does not include cached requests)
# TYPE nv_inference_count counter
nv_inference_count{model="tensorrt_llm",version="1"} 0
# HELP nv_inference_exec_count Number of model executions performed (does not include cached requests)
# TYPE nv_inference_exec_count counter
nv_inference_exec_count{model="tensorrt_llm",version="1"} 0
# HELP nv_inference_request_duration_us Cumulative inference request duration in microseconds (includes cached requests)
# TYPE nv_inference_request_duration_us counter
nv_inference_request_duration_us{model="tensorrt_llm",version="1"} 0
# HELP nv_inference_queue_duration_us Cumulative inference queuing duration in microseconds (includes cached requests)
# TYPE nv_inference_queue_duration_us counter
nv_inference_queue_duration_us{model="tensorrt_llm",version="1"} 0
# HELP nv_inference_compute_input_duration_us Cumulative compute input duration in microseconds (does not include cached requests)
# TYPE nv_inference_compute_input_duration_us counter
nv_inference_compute_input_duration_us{model="tensorrt_llm",version="1"} 0
# HELP nv_inference_compute_infer_duration_us Cumulative compute inference duration in microseconds (does not include cached requests)
# TYPE nv_inference_compute_infer_duration_us counter
nv_inference_compute_infer_duration_us{model="tensorrt_llm",version="1"} 0
# HELP nv_inference_compute_output_duration_us Cumulative inference compute output duration in microseconds (does not include cached requests)
# TYPE nv_inference_compute_output_duration_us counter
nv_inference_compute_output_duration_us{model="tensorrt_llm",version="1"} 0
# HELP nv_inference_pending_request_count Instantaneous number of pending requests awaiting execution per-model.
# TYPE nv_inference_pending_request_count gauge
nv_inference_pending_request_count{model="tensorrt_llm",version="1"} 0Check out GenAI-Perf tool for benchmarking TensorRT-LLM models.
You can also use the
benchmark_core_model script
to benchmark the core model tensosrrt_llm. The script sends requests directly
to deployed tensorrt_llm model. The benchmark core model latency indicates the
inference latency of TensorRT-LLM, not including the pre/post-processing latency
which is usually handled by a third-party library such as HuggingFace.
benchmark_core_model can generate traffic from 2 sources. 1 - dataset (json file containing prompts and optional responses) 2 - token normal distribution (user specified input, output seqlen)
By default, exponential distrution is used to control arrival rate of requests. It can be changed to constant arrival time.
cd tools/inflight_batcher_llmExample: Run dataset with 10 req/sec requested rate with provided tokenizer.
python3 benchmark_core_model.py -i grpc --request_rate 10 dataset --dataset <dataset path> --tokenizer_dir <> --num_requests 5000Example: Generate I/O seqlen tokens with input normal distribution with mean_seqlen=128, stdev=10. Output normal distribution with mean_seqlen=20, stdev=2. Set stdev=0 to get constant seqlens.
python3 benchmark_core_model.py -i grpc --request_rate 10 token_norm_dist --input_mean 128 --input_stdev 5 --output_mean 20 --output_stdev 2 --num_requests 5000Expected outputs
[INFO] Warm up for benchmarking.
[INFO] Start benchmarking on 5000 prompts.
[INFO] Total Latency: 26585.349 ms
[INFO] Total request latencies: 11569672.000999955 ms
+----------------------------+----------+
| Stat | Value |
+----------------------------+----------+
| Requests/Sec | 188.09 |
| OP tokens/sec | 3857.66 |
| Avg. latency (ms) | 2313.93 |
| P99 latency (ms) | 3624.95 |
| P90 latency (ms) | 3127.75 |
| Avg. IP tokens per request | 128.53 |
| Avg. OP tokens per request | 20.51 |
| Total latency (ms) | 26582.72 |
| Total requests | 5000.00 |
+----------------------------+----------+
Please note that the expected outputs in that document are only for reference, specific performance numbers depend on the GPU you're using.
Please follow the guide in tensorrt_llm/triton_backend/ci/README.md to see how to run
the testing for TensorRT-LLM backend.