{"owner":"NVIDIA","repo":"TensorRT-LLM","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["CLAUDE.md","AGENTS.md"],"skills":{"CLAUDE.md":"# In ./CLAUDE.md\n@AGENTS.md\n","AGENTS.md":"# AGENTS.md\n\nTensorRT-LLM: open-source library for optimized LLM inference on NVIDIA GPUs.\nPython and C++ codebase with PyTorch and AutoDeploy execution paths.\n\n> If a `CLAUDE.local.md` file exists alongside this file, read and respect it — it contains developer-specific overrides that supplement this shared guidance.\n\n## Rules (Read First)\n\n**CRITICAL (YOU MUST):**\n- Read and follow `CODING_GUIDELINES.md` for ALL code changes (C++ and Python)\n- NVIDIA copyright header on ALL new files (update year on modified files)\n- `git commit -s` (DCO sign-off required). Never attribute AI tools in sign-off line. Always rely on `git` to do the sign off instead of directly adding sign off in commit message.\n- Do not add co-authors to the git commit message unless explicitly instructed to do so by the user.\n- `pre-commit` hooks run on commit — if files are modified by hooks, re-stage and commit again\n- LLM args or nested-config changes must run `python3 scripts/generate_llm_args_golden_manifest.py` and commit\n  `tensorrt_llm/usage/llm_args_golden_manifest.json`; new fields require telemetry/privacy CODEOWNER approval\n- PR title format: `[JIRA/NVBUG/None][type] description` (e.g., `[TRTLLM-5516][perf] optimize cuda graph padding`)\n- Set `LLM_MODELS_ROOT` env var when running tests that need model weights\n\n## Common Commands\n\n| Task | Command |\n|------|---------|\n| Unit tests | `pytest tests/unittest/` |\n| Specific test | `pytest tests/unittest/llmapi/test_llm_args.py` |\n| Pattern match | `pytest tests/unittest -k \"test_llm_args\"` |\n| Integration tests | `LLM_MODELS_ROOT=/path/to/models pytest tests/integration/defs/...` |\n| Serve model | `trtllm-serve <hf_model> --port 8000` |\n| Serve with config | `trtllm-serve <hf_model> --config config.yaml` |\n| Benchmark | `trtllm-bench --model <hf_model> throughput --dataset <path>` |\n| Find CI stage for test | `python scripts/test_to_stage_mapping.py --tests \"test_name\"` |\n\n### Installation & Build\n\nBuilding TensorRT-LLM requires Docker and may involve compiling C++ components.\nSee the [Installation Guide](docs/source/installation/installation-guide.md) for pre-built release containers and pip install,\n[build from source](docs/source/installation/build-from-source.md) for development builds,\nand [Container Images](docs/source/installation/containers.md) for information about the container images.\n\n### Reference Configs\n\n`examples/configs/database/` contains pareto-optimized serving configurations\nacross multiple models, GPUs, ISL/OSL combinations, and concurrency levels.\nUse these as starting points for deployment and benchmarking rather than hand-tuning parameters.\nSee [deployment guides](docs/source/deployment-guide/) for model-specific walkthroughs.\n\n## Architecture\n\nSee [architecture diagram](.github/tava_architecture_diagram.md) for the full Mermaid diagram.\n\n### Backends\n\n| Backend | Status | Entry Point | Key Path |\n|---------|--------|-------------|----------|\n| **PyTorch** | Default | `TorchLlmArgs` | `_torch/pyexecutor/` → `PyExecutor` → PyTorch Engine |\n| **AutoDeploy** | Beta | `_torch/auto_deploy/` shim | `_torch/auto_deploy/shim/ad_executor.py` → adapts `PyExecutor` → graph transforms + torch.export |\n\n### Shared C++ Core (via Nanobind)\n\nBoth backends share these C++ components:\n- **Scheduling pipeline**: Scheduler → BatchManager (in-flight batching) → KV Cache Manager\n- **Decoding pipeline**: Decoder (token generation orchestration) → Sampling\n\n### Request Flow\n```text\nHuggingFace Model → LLM API → Executor (PyTorch/AutoDeploy)\n    → Scheduler → Model Forward → Decoder → Sampling → Generated Tokens\n```\n\n### Serving\n- `trtllm-serve`: OpenAI-compatible REST + gRPC server, supports all backends\n- **Disaggregated serving**: separates prefill (context) and decode (generation) across GPUs\n  - KV cache exchange via NIXL (default), UCX, or MPI\n\n## Key Files\n\n| File | Role |\n|------|------|\n| `tensorrt_llm/llmapi/llm.py` | Main API entry point |\n| `tensorrt_llm/llmapi/llm_args.py` | Complete configuration schema (Pydantic) |\n| `tensorrt_llm/llmapi/llm_utils.py` | Model loading, model-specific default overrides |\n| `tensorrt_llm/models/modeling_utils.py` | Base classes for all models (`PretrainedConfig`, `PretrainedModel`) |\n| `tensorrt_llm/executor/executor.py` | Execution abstraction (`GenerationExecutor`) |\n| `tensorrt_llm/models/automodel.py` | Auto-discovery and model registry |\n| `tensorrt_llm/_torch/models/` | PyTorch backend model implementations (distinct from the top-level `models/` package) |\n| `tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md` | Attention, MLA, backend families, sparse backends, metadata contracts, and KV-cache behavior - **read before modifying `tensorrt_llm/_torch/modules/attention.py`, `tensorrt_llm/_torch/modules/mla.py`, or `tensorrt_llm/_torch/attention_backend/`** |\n| `tensorrt_llm/_torch/modules/fused_moe/MOE_DEVELOPER_GUIDE.md` | MoE architecture, backends, communication, development patterns — **read before modifying MoE code** |\n| `CODING_GUIDELINES.md` | C++ and Python coding standards (referenced throughout, must read before contributing) |\n\n## Design Patterns\n\n| Pattern | Key Points |\n|---------|------------|\n| **Config hierarchy** | `BaseLlmArgs` → `TorchLlmArgs`, model-specific defaults override generics, Pydantic validation |\n| **Model architecture** | Each model: `Config` (inherits `PretrainedConfig`) + `ForCausalLM` (inherits `PretrainedModel`) |\n| **Model defaults** | Architecture-specific overrides in `llm_utils.py` (attention kernels, quant, spec decoding, cache) |\n| **Attention backends** | `TorchLlmArgs.attn_backend` selects kernel: `TRTLLM` (default), `FlashInfer`, `FlashAttention` |\n| **Distributed execution** | Tensor/pipeline parallelism via `Mapping` class, multiple backends (MPI, Ray, RPC) |\n| **Auto-discovery** | Models self-register via `automodel.py`, resolved by HF config `architectures` field |\n\n## VisualGen\n\nVisualGen is a vertical alongside LLM for Diffusion-Transformer (DiT)-based image/video generation\n(text-to-image, text-to-video, image-to-video). It is **not** an LLM backend — it has\nits own engine, args, params, and outputs — but shares ops and kernels with the\nPyTorch backend where it makes sense (attention, quantization, parallelism).\n\nKey entry points:\n- Public Python API: `from tensorrt_llm import VisualGen, VisualGenArgs, VisualGenParams`.\n- Serving CLI: `trtllm-serve --model <HF id> --visual_gen_args <YAML path>`.\n\nKey files:\n- `tensorrt_llm/_torch/visual_gen/ENGINEERING_CRITERIA.md`: **Engineering criteria for any change under `tensorrt_llm/visual_gen/` or `tensorrt_llm/_torch/visual_gen/`** — API discipline, feature/test/lossy-vs-lossless requirements, examples & docs rules. Read before modifying anything in those trees.\n- `tensorrt_llm/visual_gen/`: VisualGen public Python API. **User-facing surface — before modifying anything here, pause and confirm with the user that a public API change is actually intended; do not infer it from the surrounding task.**\n- `tensorrt_llm/_torch/visual_gen/`: VisualGen internal implementation. All non-user-facing code belongs here.\n\n## Anti-Patterns / Gotchas\n\n- **Pre-commit modifies files in-place** — if hooks fail, files are already modified. Re-stage (`git add`) and commit again.\n- **Protected APIs exist** — changes to LLM API signatures will fail `tests/unittest/api_stability` tests. Get code owner review.\n- **Integration tests need GPUs + models** — always set `LLM_MODELS_ROOT` and ensure GPU access. Unit tests don't.\n- **Copyright year** — update to current year when modifying existing files; add full header to new files.\n- **Avoid broad exception handling** — catch specific exceptions, not bare `except:` (see `CODING_GUIDELINES.md`).\n- **One concern per PR** — avoid scope creep. If a PR touches unrelated areas, split it.\n- **User-facing configuration classes** - when editing or defining any user-facing configuration classes (particularly `BaseLlmArgs` or any class used in its fields), you **MUST** follow the Pydantic guidelines in `CODING_GUIDELINES.md`.\n\n## Development Workflow\n\n1. Set up build environment (see [installation docs](docs/source/installation/))\n2. Make changes following `CODING_GUIDELINES.md`\n3. Test locally with `pytest`\n\n## Branching policy and PRs\n\n- The main repository (`upstream`) is located at https://github.com/NVIDIA/TensorRT-LLM/\n- Branches should always be pushed to the user-specified fork (usually `origin`)\n- If pushing fails to due pre-push pre-commits hooks getting updated, just re-push immediately\n- PRs should be opened on the main repository\n   - Target `main` unless fixing a release branch bug\n   - See `CONTRIBUTING.md` for full PR policies\n\n### GitHub CLI authentication (`GH_CONFIG_DIR`)\n\nThe `gh` CLI uses `~/.config/gh` by default for authentication. Different GitHub hosts or forks may require a different config directory. **Before running any `gh` command** (e.g., `gh pr create`, `gh api`, `gh pr comment`):\n\n1. Check if the user has specified a custom `GH_CONFIG_DIR` (e.g., in `CLAUDE.local.md` or environment). If so, use it.\n2. If not explicitly set, default to `~/.config/gh`; do not ask for confirmation.\n3. Prefix all `gh` commands with the resolved config dir: `GH_CONFIG_DIR=<path> gh ...`\n4. If the command fails due to missing authentication or the wrong GitHub host/account, report the failure and ask for the correct `GH_CONFIG_DIR`.\n\n## CI / Testing\n\nSee [CI overview](docs/source/developer-guide/ci-overview.md) for full details.\n\n| Layer | Location | Notes |\n|-------|----------|-------|\n| Unit tests | `tests/unittest/` | Run in pre-merge CI; some tests require GPU |\n| API stability | `tests/unittest/api_stability/` | Protects committed API signatures |\n| Integration tests | `tests/integration/defs/` | Requires GPU + `LLM_MODELS_ROOT` |\n| Test lists | `tests/integration/test_lists/test-db/` | Per-GPU YAML files (`l0_a10.yml`, `l0_h100.yml`, etc.) |\n| Test waives | `tests/integration/test_lists/waives.txt` | Skip known-failing tests with NVBug links |\n| Performance | See [benchmarking guide](docs/source/developer-guide/perf-benchmarking.md) | `trtllm-bench` and `trtllm-serve` benchmarks |\n\n### Triggering CI\n\nCI is triggered by posting comments on the PR. Basic commands:\n- `/bot run` — trigger the standard CI pipeline\n- `/bot run --disable-fail-fast` — run all stages even if earlier ones fail (only add when explicitly needed)\n- `/bot run --extra-stage \"DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1\"` — include AutoDeploy CI stages (use for AutoDeploy-related PRs)\n\nFor a full list of up-to-date bot commands, post `/bot help` as a PR comment and check the bot's reply.\n\n### Trouble Shooting\n\n- Use `TLLM_LOG_LEVEL_BY_MODULE` to enable per-module log filtering (e.g., `\"debug:_torch,runtime;info:serve\"`); see [Module-Level Logging](docs/source/developer-guide/overview.md#module-level-logging) for details.\n\n## Key Documentation\n\n| Topic | Path |\n|-------|------|\n| Coding guidelines | `CODING_GUIDELINES.md` |\n| Architecture overview | `docs/source/developer-guide/overview.md` |\n| PyTorch backend | `docs/source/torch/arch_overview.md` |\n| Adding a new model | `docs/source/torch/adding_new_model.md` |\n| AutoDeploy | `docs/source/features/auto_deploy/auto-deploy.md` |\n| Disaggregated serving | `docs/source/features/disagg-serving.md` |\n| Speculative decoding | `docs/source/features/speculative-decoding.md` |\n| Quantization | `docs/source/features/quantization.md` |\n| Parallelism strategies | `docs/source/features/parallel-strategy.md` |\n| KV cache | `docs/source/features/kvcache.md` |\n| API change guidelines | `docs/source/developer-guide/api-change.md` |\n| Feature compatibility matrix | `docs/source/features/feature-combination-matrix.md` |\n| Supported models | `docs/source/models/supported-models.md` |\n| Deployment guides | `docs/source/deployment-guide/` |\n| Examples & customization | `docs/source/examples/` |\n| Performance analysis | `docs/source/developer-guide/perf-analysis.md` |\n"},"files":{"CLAUDE.md":"# In ./CLAUDE.md\n@AGENTS.md\n","AGENTS.md":"# AGENTS.md\n\nTensorRT-LLM: open-source library for optimized LLM inference on NVIDIA GPUs.\nPython and C++ codebase with PyTorch and AutoDeploy execution paths.\n\n> If a `CLAUDE.local.md` file exists alongside this file, read and respect it — it contains developer-specific overrides that supplement this shared guidance.\n\n## Rules (Read First)\n\n**CRITICAL (YOU MUST):**\n- Read and follow `CODING_GUIDELINES.md` for ALL code changes (C++ and Python)\n- NVIDIA copyright header on ALL new files (update year on modified files)\n- `git commit -s` (DCO sign-off required). Never attribute AI tools in sign-off line. Always rely on `git` to do the sign off instead of directly adding sign off in commit message.\n- Do not add co-authors to the git commit message unless explicitly instructed to do so by the user.\n- `pre-commit` hooks run on commit — if files are modified by hooks, re-stage and commit again\n- LLM args or nested-config changes must run `python3 scripts/generate_llm_args_golden_manifest.py` and commit\n  `tensorrt_llm/usage/llm_args_golden_manifest.json`; new fields require telemetry/privacy CODEOWNER approval\n- PR title format: `[JIRA/NVBUG/None][type] description` (e.g., `[TRTLLM-5516][perf] optimize cuda graph padding`)\n- Set `LLM_MODELS_ROOT` env var when running tests that need model weights\n\n## Common Commands\n\n| Task | Command |\n|------|---------|\n| Unit tests | `pytest tests/unittest/` |\n| Specific test | `pytest tests/unittest/llmapi/test_llm_args.py` |\n| Pattern match | `pytest tests/unittest -k \"test_llm_args\"` |\n| Integration tests | `LLM_MODELS_ROOT=/path/to/models pytest tests/integration/defs/...` |\n| Serve model | `trtllm-serve <hf_model> --port 8000` |\n| Serve with config | `trtllm-serve <hf_model> --config config.yaml` |\n| Benchmark | `trtllm-bench --model <hf_model> throughput --dataset <path>` |\n| Find CI stage for test | `python scripts/test_to_stage_mapping.py --tests \"test_name\"` |\n\n### Installation & Build\n\nBuilding TensorRT-LLM requires Docker and may involve compiling C++ components.\nSee the [Installation Guide](docs/source/installation/installation-guide.md) for pre-built release containers and pip install,\n[build from source](docs/source/installation/build-from-source.md) for development builds,\nand [Container Images](docs/source/installation/containers.md) for information about the container images.\n\n### Reference Configs\n\n`examples/configs/database/` contains pareto-optimized serving configurations\nacross multiple models, GPUs, ISL/OSL combinations, and concurrency levels.\nUse these as starting points for deployment and benchmarking rather than hand-tuning parameters.\nSee [deployment guides](docs/source/deployment-guide/) for model-specific walkthroughs.\n\n## Architecture\n\nSee [architecture diagram](.github/tava_architecture_diagram.md) for the full Mermaid diagram.\n\n### Backends\n\n| Backend | Status | Entry Point | Key Path |\n|---------|--------|-------------|----------|\n| **PyTorch** | Default | `TorchLlmArgs` | `_torch/pyexecutor/` → `PyExecutor` → PyTorch Engine |\n| **AutoDeploy** | Beta | `_torch/auto_deploy/` shim | `_torch/auto_deploy/shim/ad_executor.py` → adapts `PyExecutor` → graph transforms + torch.export |\n\n### Shared C++ Core (via Nanobind)\n\nBoth backends share these C++ components:\n- **Scheduling pipeline**: Scheduler → BatchManager (in-flight batching) → KV Cache Manager\n- **Decoding pipeline**: Decoder (token generation orchestration) → Sampling\n\n### Request Flow\n```text\nHuggingFace Model → LLM API → Executor (PyTorch/AutoDeploy)\n    → Scheduler → Model Forward → Decoder → Sampling → Generated Tokens\n```\n\n### Serving\n- `trtllm-serve`: OpenAI-compatible REST + gRPC server, supports all backends\n- **Disaggregated serving**: separates prefill (context) and decode (generation) across GPUs\n  - KV cache exchange via NIXL (default), UCX, or MPI\n\n## Key Files\n\n| File | Role |\n|------|------|\n| `tensorrt_llm/llmapi/llm.py` | Main API entry point |\n| `tensorrt_llm/llmapi/llm_args.py` | Complete configuration schema (Pydantic) |\n| `tensorrt_llm/llmapi/llm_utils.py` | Model loading, model-specific default overrides |\n| `tensorrt_llm/models/modeling_utils.py` | Base classes for all models (`PretrainedConfig`, `PretrainedModel`) |\n| `tensorrt_llm/executor/executor.py` | Execution abstraction (`GenerationExecutor`) |\n| `tensorrt_llm/models/automodel.py` | Auto-discovery and model registry |\n| `tensorrt_llm/_torch/models/` | PyTorch backend model implementations (distinct from the top-level `models/` package) |\n| `tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md` | Attention, MLA, backend families, sparse backends, metadata contracts, and KV-cache behavior - **read before modifying `tensorrt_llm/_torch/modules/attention.py`, `tensorrt_llm/_torch/modules/mla.py`, or `tensorrt_llm/_torch/attention_backend/`** |\n| `tensorrt_llm/_torch/modules/fused_moe/MOE_DEVELOPER_GUIDE.md` | MoE architecture, backends, communication, development patterns — **read before modifying MoE code** |\n| `CODING_GUIDELINES.md` | C++ and Python coding standards (referenced throughout, must read before contributing) |\n\n## Design Patterns\n\n| Pattern | Key Points |\n|---------|------------|\n| **Config hierarchy** | `BaseLlmArgs` → `TorchLlmArgs`, model-specific defaults override generics, Pydantic validation |\n| **Model architecture** | Each model: `Config` (inherits `PretrainedConfig`) + `ForCausalLM` (inherits `PretrainedModel`) |\n| **Model defaults** | Architecture-specific overrides in `llm_utils.py` (attention kernels, quant, spec decoding, cache) |\n| **Attention backends** | `TorchLlmArgs.attn_backend` selects kernel: `TRTLLM` (default), `FlashInfer`, `FlashAttention` |\n| **Distributed execution** | Tensor/pipeline parallelism via `Mapping` class, multiple backends (MPI, Ray, RPC) |\n| **Auto-discovery** | Models self-register via `automodel.py`, resolved by HF config `architectures` field |\n\n## VisualGen\n\nVisualGen is a vertical alongside LLM for Diffusion-Transformer (DiT)-based image/video generation\n(text-to-image, text-to-video, image-to-video). It is **not** an LLM backend — it has\nits own engine, args, params, and outputs — but shares ops and kernels with the\nPyTorch backend where it makes sense (attention, quantization, parallelism).\n\nKey entry points:\n- Public Python API: `from tensorrt_llm import VisualGen, VisualGenArgs, VisualGenParams`.\n- Serving CLI: `trtllm-serve --model <HF id> --visual_gen_args <YAML path>`.\n\nKey files:\n- `tensorrt_llm/_torch/visual_gen/ENGINEERING_CRITERIA.md`: **Engineering criteria for any change under `tensorrt_llm/visual_gen/` or `tensorrt_llm/_torch/visual_gen/`** — API discipline, feature/test/lossy-vs-lossless requirements, examples & docs rules. Read before modifying anything in those trees.\n- `tensorrt_llm/visual_gen/`: VisualGen public Python API. **User-facing surface — before modifying anything here, pause and confirm with the user that a public API change is actually intended; do not infer it from the surrounding task.**\n- `tensorrt_llm/_torch/visual_gen/`: VisualGen internal implementation. All non-user-facing code belongs here.\n\n## Anti-Patterns / Gotchas\n\n- **Pre-commit modifies files in-place** — if hooks fail, files are already modified. Re-stage (`git add`) and commit again.\n- **Protected APIs exist** — changes to LLM API signatures will fail `tests/unittest/api_stability` tests. Get code owner review.\n- **Integration tests need GPUs + models** — always set `LLM_MODELS_ROOT` and ensure GPU access. Unit tests don't.\n- **Copyright year** — update to current year when modifying existing files; add full header to new files.\n- **Avoid broad exception handling** — catch specific exceptions, not bare `except:` (see `CODING_GUIDELINES.md`).\n- **One concern per PR** — avoid scope creep. If a PR touches unrelated areas, split it.\n- **User-facing configuration classes** - when editing or defining any user-facing configuration classes (particularly `BaseLlmArgs` or any class used in its fields), you **MUST** follow the Pydantic guidelines in `CODING_GUIDELINES.md`.\n\n## Development Workflow\n\n1. Set up build environment (see [installation docs](docs/source/installation/))\n2. Make changes following `CODING_GUIDELINES.md`\n3. Test locally with `pytest`\n\n## Branching policy and PRs\n\n- The main repository (`upstream`) is located at https://github.com/NVIDIA/TensorRT-LLM/\n- Branches should always be pushed to the user-specified fork (usually `origin`)\n- If pushing fails to due pre-push pre-commits hooks getting updated, just re-push immediately\n- PRs should be opened on the main repository\n   - Target `main` unless fixing a release branch bug\n   - See `CONTRIBUTING.md` for full PR policies\n\n### GitHub CLI authentication (`GH_CONFIG_DIR`)\n\nThe `gh` CLI uses `~/.config/gh` by default for authentication. Different GitHub hosts or forks may require a different config directory. **Before running any `gh` command** (e.g., `gh pr create`, `gh api`, `gh pr comment`):\n\n1. Check if the user has specified a custom `GH_CONFIG_DIR` (e.g., in `CLAUDE.local.md` or environment). If so, use it.\n2. If not explicitly set, default to `~/.config/gh`; do not ask for confirmation.\n3. Prefix all `gh` commands with the resolved config dir: `GH_CONFIG_DIR=<path> gh ...`\n4. If the command fails due to missing authentication or the wrong GitHub host/account, report the failure and ask for the correct `GH_CONFIG_DIR`.\n\n## CI / Testing\n\nSee [CI overview](docs/source/developer-guide/ci-overview.md) for full details.\n\n| Layer | Location | Notes |\n|-------|----------|-------|\n| Unit tests | `tests/unittest/` | Run in pre-merge CI; some tests require GPU |\n| API stability | `tests/unittest/api_stability/` | Protects committed API signatures |\n| Integration tests | `tests/integration/defs/` | Requires GPU + `LLM_MODELS_ROOT` |\n| Test lists | `tests/integration/test_lists/test-db/` | Per-GPU YAML files (`l0_a10.yml`, `l0_h100.yml`, etc.) |\n| Test waives | `tests/integration/test_lists/waives.txt` | Skip known-failing tests with NVBug links |\n| Performance | See [benchmarking guide](docs/source/developer-guide/perf-benchmarking.md) | `trtllm-bench` and `trtllm-serve` benchmarks |\n\n### Triggering CI\n\nCI is triggered by posting comments on the PR. Basic commands:\n- `/bot run` — trigger the standard CI pipeline\n- `/bot run --disable-fail-fast` — run all stages even if earlier ones fail (only add when explicitly needed)\n- `/bot run --extra-stage \"DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1\"` — include AutoDeploy CI stages (use for AutoDeploy-related PRs)\n\nFor a full list of up-to-date bot commands, post `/bot help` as a PR comment and check the bot's reply.\n\n### Trouble Shooting\n\n- Use `TLLM_LOG_LEVEL_BY_MODULE` to enable per-module log filtering (e.g., `\"debug:_torch,runtime;info:serve\"`); see [Module-Level Logging](docs/source/developer-guide/overview.md#module-level-logging) for details.\n\n## Key Documentation\n\n| Topic | Path |\n|-------|------|\n| Coding guidelines | `CODING_GUIDELINES.md` |\n| Architecture overview | `docs/source/developer-guide/overview.md` |\n| PyTorch backend | `docs/source/torch/arch_overview.md` |\n| Adding a new model | `docs/source/torch/adding_new_model.md` |\n| AutoDeploy | `docs/source/features/auto_deploy/auto-deploy.md` |\n| Disaggregated serving | `docs/source/features/disagg-serving.md` |\n| Speculative decoding | `docs/source/features/speculative-decoding.md` |\n| Quantization | `docs/source/features/quantization.md` |\n| Parallelism strategies | `docs/source/features/parallel-strategy.md` |\n| KV cache | `docs/source/features/kvcache.md` |\n| API change guidelines | `docs/source/developer-guide/api-change.md` |\n| Feature compatibility matrix | `docs/source/features/feature-combination-matrix.md` |\n| Supported models | `docs/source/models/supported-models.md` |\n| Deployment guides | `docs/source/deployment-guide/` |\n| Examples & customization | `docs/source/examples/` |\n| Performance analysis | `docs/source/developer-guide/perf-analysis.md` |\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# In ./CLAUDE.md\n@AGENTS.md\n","category":"root","tokens":7},{"name":"AGENTS.md","path":"AGENTS.md","title":"AGENTS.md","content":"# AGENTS.md\n\nTensorRT-LLM: open-source library for optimized LLM inference on NVIDIA GPUs.\nPython and C++ codebase with PyTorch and AutoDeploy execution paths.\n\n> If a `CLAUDE.local.md` file exists alongside this file, read and respect it — it contains developer-specific overrides that supplement this shared guidance.\n\n## Rules (Read First)\n\n**CRITICAL (YOU MUST):**\n- Read and follow `CODING_GUIDELINES.md` for ALL code changes (C++ and Python)\n- NVIDIA copyright header on ALL new files (update year on modified files)\n- `git commit -s` (DCO sign-off required). Never attribute AI tools in sign-off line. Always rely on `git` to do the sign off instead of directly adding sign off in commit message.\n- Do not add co-authors to the git commit message unless explicitly instructed to do so by the user.\n- `pre-commit` hooks run on commit — if files are modified by hooks, re-stage and commit again\n- LLM args or nested-config changes must run `python3 scripts/generate_llm_args_golden_manifest.py` and commit\n  `tensorrt_llm/usage/llm_args_golden_manifest.json`; new fields require telemetry/privacy CODEOWNER approval\n- PR title format: `[JIRA/NVBUG/None][type] description` (e.g., `[TRTLLM-5516][perf] optimize cuda graph padding`)\n- Set `LLM_MODELS_ROOT` env var when running tests that need model weights\n\n## Common Commands\n\n| Task | Command |\n|------|---------|\n| Unit tests | `pytest tests/unittest/` |\n| Specific test | `pytest tests/unittest/llmapi/test_llm_args.py` |\n| Pattern match | `pytest tests/unittest -k \"test_llm_args\"` |\n| Integration tests | `LLM_MODELS_ROOT=/path/to/models pytest tests/integration/defs/...` |\n| Serve model | `trtllm-serve <hf_model> --port 8000` |\n| Serve with config | `trtllm-serve <hf_model> --config config.yaml` |\n| Benchmark | `trtllm-bench --model <hf_model> throughput --dataset <path>` |\n| Find CI stage for test | `python scripts/test_to_stage_mapping.py --tests \"test_name\"` |\n\n### Installation & Build\n\nBuilding TensorRT-LLM requires Docker and may involve compiling C++ components.\nSee the [Installation Guide](docs/source/installation/installation-guide.md) for pre-built release containers and pip install,\n[build from source](docs/source/installation/build-from-source.md) for development builds,\nand [Container Images](docs/source/installation/containers.md) for information about the container images.\n\n### Reference Configs\n\n`examples/configs/database/` contains pareto-optimized serving configurations\nacross multiple models, GPUs, ISL/OSL combinations, and concurrency levels.\nUse these as starting points for deployment and benchmarking rather than hand-tuning parameters.\nSee [deployment guides](docs/source/deployment-guide/) for model-specific walkthroughs.\n\n## Architecture\n\nSee [architecture diagram](.github/tava_architecture_diagram.md) for the full Mermaid diagram.\n\n### Backends\n\n| Backend | Status | Entry Point | Key Path |\n|---------|--------|-------------|----------|\n| **PyTorch** | Default | `TorchLlmArgs` | `_torch/pyexecutor/` → `PyExecutor` → PyTorch Engine |\n| **AutoDeploy** | Beta | `_torch/auto_deploy/` shim | `_torch/auto_deploy/shim/ad_executor.py` → adapts `PyExecutor` → graph transforms + torch.export |\n\n### Shared C++ Core (via Nanobind)\n\nBoth backends share these C++ components:\n- **Scheduling pipeline**: Scheduler → BatchManager (in-flight batching) → KV Cache Manager\n- **Decoding pipeline**: Decoder (token generation orchestration) → Sampling\n\n### Request Flow\n```text\nHuggingFace Model → LLM API → Executor (PyTorch/AutoDeploy)\n    → Scheduler → Model Forward → Decoder → Sampling → Generated Tokens\n```\n\n### Serving\n- `trtllm-serve`: OpenAI-compatible REST + gRPC server, supports all backends\n- **Disaggregated serving**: separates prefill (context) and decode (generation) across GPUs\n  - KV cache exchange via NIXL (default), UCX, or MPI\n\n## Key Files\n\n| File | Role |\n|------|------|\n| `tensorrt_llm/llmapi/llm.py` | Main API entry point |\n| `tensorrt_llm/llmapi/llm_args.py` | Complete configuration schema (Pydantic) |\n| `tensorrt_llm/llmapi/llm_utils.py` | Model loading, model-specific default overrides |\n| `tensorrt_llm/models/modeling_utils.py` | Base classes for all models (`PretrainedConfig`, `PretrainedModel`) |\n| `tensorrt_llm/executor/executor.py` | Execution abstraction (`GenerationExecutor`) |\n| `tensorrt_llm/models/automodel.py` | Auto-discovery and model registry |\n| `tensorrt_llm/_torch/models/` | PyTorch backend model implementations (distinct from the top-level `models/` package) |\n| `tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md` | Attention, MLA, backend families, sparse backends, metadata contracts, and KV-cache behavior - **read before modifying `tensorrt_llm/_torch/modules/attention.py`, `tensorrt_llm/_torch/modules/mla.py`, or `tensorrt_llm/_torch/attention_backend/`** |\n| `tensorrt_llm/_torch/modules/fused_moe/MOE_DEVELOPER_GUIDE.md` | MoE architecture, backends, communication, development patterns — **read before modifying MoE code** |\n| `CODING_GUIDELINES.md` | C++ and Python coding standards (referenced throughout, must read before contributing) |\n\n## Design Patterns\n\n| Pattern | Key Points |\n|---------|------------|\n| **Config hierarchy** | `BaseLlmArgs` → `TorchLlmArgs`, model-specific defaults override generics, Pydantic validation |\n| **Model architecture** | Each model: `Config` (inherits `PretrainedConfig`) + `ForCausalLM` (inherits `PretrainedModel`) |\n| **Model defaults** | Architecture-specific overrides in `llm_utils.py` (attention kernels, quant, spec decoding, cache) |\n| **Attention backends** | `TorchLlmArgs.attn_backend` selects kernel: `TRTLLM` (default), `FlashInfer`, `FlashAttention` |\n| **Distributed execution** | Tensor/pipeline parallelism via `Mapping` class, multiple backends (MPI, Ray, RPC) |\n| **Auto-discovery** | Models self-register via `automodel.py`, resolved by HF config `architectures` field |\n\n## VisualGen\n\nVisualGen is a vertical alongside LLM for Diffusion-Transformer (DiT)-based image/video generation\n(text-to-image, text-to-video, image-to-video). It is **not** an LLM backend — it has\nits own engine, args, params, and outputs — but shares ops and kernels with the\nPyTorch backend where it makes sense (attention, quantization, parallelism).\n\nKey entry points:\n- Public Python API: `from tensorrt_llm import VisualGen, VisualGenArgs, VisualGenParams`.\n- Serving CLI: `trtllm-serve --model <HF id> --visual_gen_args <YAML path>`.\n\nKey files:\n- `tensorrt_llm/_torch/visual_gen/ENGINEERING_CRITERIA.md`: **Engineering criteria for any change under `tensorrt_llm/visual_gen/` or `tensorrt_llm/_torch/visual_gen/`** — API discipline, feature/test/lossy-vs-lossless requirements, examples & docs rules. Read before modifying anything in those trees.\n- `tensorrt_llm/visual_gen/`: VisualGen public Python API. **User-facing surface — before modifying anything here, pause and confirm with the user that a public API change is actually intended; do not infer it from the surrounding task.**\n- `tensorrt_llm/_torch/visual_gen/`: VisualGen internal implementation. All non-user-facing code belongs here.\n\n## Anti-Patterns / Gotchas\n\n- **Pre-commit modifies files in-place** — if hooks fail, files are already modified. Re-stage (`git add`) and commit again.\n- **Protected APIs exist** — changes to LLM API signatures will fail `tests/unittest/api_stability` tests. Get code owner review.\n- **Integration tests need GPUs + models** — always set `LLM_MODELS_ROOT` and ensure GPU access. Unit tests don't.\n- **Copyright year** — update to current year when modifying existing files; add full header to new files.\n- **Avoid broad exception handling** — catch specific exceptions, not bare `except:` (see `CODING_GUIDELINES.md`).\n- **One concern per PR** — avoid scope creep. If a PR touches unrelated areas, split it.\n- **User-facing configuration classes** - when editing or defining any user-facing configuration classes (particularly `BaseLlmArgs` or any class used in its fields), you **MUST** follow the Pydantic guidelines in `CODING_GUIDELINES.md`.\n\n## Development Workflow\n\n1. Set up build environment (see [installation docs](docs/source/installation/))\n2. Make changes following `CODING_GUIDELINES.md`\n3. Test locally with `pytest`\n\n## Branching policy and PRs\n\n- The main repository (`upstream`) is located at https://github.com/NVIDIA/TensorRT-LLM/\n- Branches should always be pushed to the user-specified fork (usually `origin`)\n- If pushing fails to due pre-push pre-commits hooks getting updated, just re-push immediately\n- PRs should be opened on the main repository\n   - Target `main` unless fixing a release branch bug\n   - See `CONTRIBUTING.md` for full PR policies\n\n### GitHub CLI authentication (`GH_CONFIG_DIR`)\n\nThe `gh` CLI uses `~/.config/gh` by default for authentication. Different GitHub hosts or forks may require a different config directory. **Before running any `gh` command** (e.g., `gh pr create`, `gh api`, `gh pr comment`):\n\n1. Check if the user has specified a custom `GH_CONFIG_DIR` (e.g., in `CLAUDE.local.md` or environment). If so, use it.\n2. If not explicitly set, default to `~/.config/gh`; do not ask for confirmation.\n3. Prefix all `gh` commands with the resolved config dir: `GH_CONFIG_DIR=<path> gh ...`\n4. If the command fails due to missing authentication or the wrong GitHub host/account, report the failure and ask for the correct `GH_CONFIG_DIR`.\n\n## CI / Testing\n\nSee [CI overview](docs/source/developer-guide/ci-overview.md) for full details.\n\n| Layer | Location | Notes |\n|-------|----------|-------|\n| Unit tests | `tests/unittest/` | Run in pre-merge CI; some tests require GPU |\n| API stability | `tests/unittest/api_stability/` | Protects committed API signatures |\n| Integration tests | `tests/integration/defs/` | Requires GPU + `LLM_MODELS_ROOT` |\n| Test lists | `tests/integration/test_lists/test-db/` | Per-GPU YAML files (`l0_a10.yml`, `l0_h100.yml`, etc.) |\n| Test waives | `tests/integration/test_lists/waives.txt` | Skip known-failing tests with NVBug links |\n| Performance | See [benchmarking guide](docs/source/developer-guide/perf-benchmarking.md) | `trtllm-bench` and `trtllm-serve` benchmarks |\n\n### Triggering CI\n\nCI is triggered by posting comments on the PR. Basic commands:\n- `/bot run` — trigger the standard CI pipeline\n- `/bot run --disable-fail-fast` — run all stages even if earlier ones fail (only add when explicitly needed)\n- `/bot run --extra-stage \"DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1\"` — include AutoDeploy CI stages (use for AutoDeploy-related PRs)\n\nFor a full list of up-to-date bot commands, post `/bot help` as a PR comment and check the bot's reply.\n\n### Trouble Shooting\n\n- Use `TLLM_LOG_LEVEL_BY_MODULE` to enable per-module log filtering (e.g., `\"debug:_torch,runtime;info:serve\"`); see [Module-Level Logging](docs/source/developer-guide/overview.md#module-level-logging) for details.\n\n## Key Documentation\n\n| Topic | Path |\n|-------|------|\n| Coding guidelines | `CODING_GUIDELINES.md` |\n| Architecture overview | `docs/source/developer-guide/overview.md` |\n| PyTorch backend | `docs/source/torch/arch_overview.md` |\n| Adding a new model | `docs/source/torch/adding_new_model.md` |\n| AutoDeploy | `docs/source/features/auto_deploy/auto-deploy.md` |\n| Disaggregated serving | `docs/source/features/disagg-serving.md` |\n| Speculative decoding | `docs/source/features/speculative-decoding.md` |\n| Quantization | `docs/source/features/quantization.md` |\n| Parallelism strategies | `docs/source/features/parallel-strategy.md` |\n| KV cache | `docs/source/features/kvcache.md` |\n| API change guidelines | `docs/source/developer-guide/api-change.md` |\n| Feature compatibility matrix | `docs/source/features/feature-combination-matrix.md` |\n| Supported models | `docs/source/models/supported-models.md` |\n| Deployment guides | `docs/source/deployment-guide/` |\n| Examples & customization | `docs/source/examples/` |\n| Performance analysis | `docs/source/developer-guide/perf-analysis.md` |\n","category":"root","tokens":3002}]}