Add TensorRT-LLM support for inference backend#121

Open

chungen04 wants to merge 28 commits into

lightseekorg:mainfrom

chungen04:chungen/trtllm

Commits on Jun 26, 2026

Patches for trt
chungen04
committed
Exported trtllm as engine class
chungen04
committed
Extend factory and config class for trtllm
chungen04
committed
dockerfile for trtllm
chungen04
committed
Add qwen3 8b example
chungen04
committed
Modified qwen3 8b example
chungen04
committed
marked trtllm support in README.md
chungen04
committed
Apply trtllm patch import-free at build time

chungen04
and
claude
committed
docker(trtllm): forward Mooncake/TorchSpec env to MPI workers

chungen04
and
claude
committed
docker(trtllm): key SaveHiddenStates Mooncake store on client_id

chungen04
and
claude
committed
trtllm: disable KV block reuse for hidden-state capture

chungen04
and
claude
committed
config(trtllm): raise max_num_tokens to match max_seq_length

chungen04
and
claude
committed
style(trtllm): apply ruff format to trtllm_engine

chungen04
and
claude
committed
trtllm: fold mpi_session patch into trtllm.patch and relax batch cap
chungen04
committed
trtllm: unify tp_size override across backends in run.sh
chungen04
committed
config(trtllm): tune qwen3-8b for single-node single-GPU
chungen04
committed
align(trtllm): match remote README launch form and Dockerfile patch step

chungen04
and
claude
committed
fix(trtllm): capture hidden states in native context order

chungen04
and
claude
committed
fix(trtllm): default placement_strategy=inference_first

chungen04
and
claude
committed
fix(trtllm): protect trust_remote_code from extra_args override

chungen04
and
claude
committed
style(trtllm): trim and align comments

chungen04
and
claude
committed
fix(trtllm): honor store_last_hidden_states in mooncake capture

chungen04
and
claude
committed
fix(trtllm): defer tensorrt_llm imports until engine init

chungen04
and
claude
committed
fix(run.sh): make engine-type probe non-fatal under set -e

chungen04
and
claude
committed
feat(trtllm): support data-parallel tp=1 inference (Ray-scoped placement + per-engine keys)

chungen04
and
claude
committed
feat(trtllm): enable DFlash training with the trtllm backend

chungen04
and
claude
committed
docs(trtllm): make branch comments self-contained

chungen04
and
claude
committed

Commits on Jun 27, 2026

Remove comment to instruct docker build
chungen04
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add TensorRT-LLM support for inference backend#121

Add TensorRT-LLM support for inference backend#121
chungen04 wants to merge 28 commits into
lightseekorg:mainfrom
chungen04:chungen/trtllm

Commits on Jun 26, 2026

Patches for trt

Exported trtllm as engine class

Extend factory and config class for trtllm

dockerfile for trtllm

Add qwen3 8b example

Modified qwen3 8b example

marked trtllm support in README.md

Apply trtllm patch import-free at build time

docker(trtllm): forward Mooncake/TorchSpec env to MPI workers

docker(trtllm): key SaveHiddenStates Mooncake store on client_id

trtllm: disable KV block reuse for hidden-state capture

config(trtllm): raise max_num_tokens to match max_seq_length

style(trtllm): apply ruff format to trtllm_engine

trtllm: fold mpi_session patch into trtllm.patch and relax batch cap

trtllm: unify tp_size override across backends in run.sh

config(trtllm): tune qwen3-8b for single-node single-GPU

align(trtllm): match remote README launch form and Dockerfile patch step

fix(trtllm): capture hidden states in native context order

fix(trtllm): default placement_strategy=inference_first

fix(trtllm): protect trust_remote_code from extra_args override

style(trtllm): trim and align comments

fix(trtllm): honor store_last_hidden_states in mooncake capture

fix(trtllm): defer tensorrt_llm imports until engine init

fix(run.sh): make engine-type probe non-fatal under set -e

feat(trtllm): support data-parallel tp=1 inference (Ray-scoped placement + per-engine keys)

feat(trtllm): enable DFlash training with the trtllm backend

docs(trtllm): make branch comments self-contained

Commits on Jun 27, 2026

Remove comment to instruct docker build