{"owner":"xorbitsai","repo":"inference","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md"],"skills":{"AGENTS.md":"# AI Agent Guidance\n\nGuidance for AI coding agents working in this repository.\n\n## Project Overview\n\nXinference is a Python model-serving project for language, embedding, rerank,\nimage, video, audio, and multimodal models. It exposes CLI commands, a Python\nclient, REST/OpenAI-compatible APIs, an xoscar-based distributed runtime, and a\nReact Web UI.\n\nPrimary package and entry points:\n\n- `xinference/`: Python package.\n- `xinference/deploy/cmdline.py`: CLI entry points for `xinference`,\n  `xinference-local`, `xinference-supervisor`, and `xinference-worker`.\n- `xinference/core/`: supervisor/worker runtime and actor orchestration.\n- `xinference/model/`: model families, engines, built-in model specs, and model\n  tests.\n- `xinference/api/`: API server and OpenAI-compatible routes.\n- `xinference/client/`: sync and async Python clients.\n- `frontend/`: Next.js Web UI.\n- `doc/source/`: Sphinx documentation.\n- `.github/workflows/python.yaml`: main lint and test CI.\n\n## Working Rules\n\n- Prefer small, focused changes that match the current module's style.\n- Preserve backward compatibility. If a breaking change is unavoidable, document\n  the reason and add deprecation behavior where practical.\n- Do not edit `xinference/thirdparty/` unless the task explicitly concerns\n  vendored code.\n- Avoid broad refactors while fixing a localized bug.\n- Keep public API behavior, request/response schemas, model registration names,\n  and CLI flags stable unless the task requires changing them.\n- Add or update tests for behavior changes. For model-runtime changes, prefer\n  tests close to the affected model family under `xinference/model/**/tests/`.\n- Use type hints for new Python code when practical; this project encourages\n  PEP 484 style annotations.\n- Treat docs and examples as user-facing API. Keep command examples accurate.\n\n## Environment Setup\n\nRecommended local setup:\n\n```bash\nconda create --name xinf python=3.12 nodejs\nconda activate xinf\npip install -e \".[dev]\"\n```\n\nNotes:\n\n- Wheel, sdist, and editable builds (including `pip install -e .`) run the\n  Web UI build through the in-tree build backend (`build_backend.py` /\n  `build_web.py`) unless `NO_WEB_UI=1` is set.\n- For Python 3.12 and newer, CI installs `setuptools<82`; use the same pin if\n  packaging or editable installs fail.\n- The project supports Python 3.10 through 3.13 in CI.\n- Optional model engines have extras in `pyproject.toml`, such as\n  `transformers`, `vllm`, `mlx`, `embedding`, `rerank`, `image`, `video`, and\n  `audio`.\n\n## Formatting and Linting\n\nPython formatting and checks are managed through pre-commit:\n\n```bash\npip install pre-commit\npre-commit run --files <modified-files>\n```\n\nFor a branch-wide check against upstream main:\n\n```bash\npre-commit run --from-ref=upstream/main --to-ref=HEAD --all-files\n```\n\nConfigured hooks include Black, end-of-file/trailing-whitespace checks, Ruff,\nisort, mypy with missing imports ignored, and codespell. Configuration lives in\n`.pre-commit-config.yaml` and `pyproject.toml`.\n\n## Python Tests\n\nRun focused tests first:\n\n```bash\npytest -vv path/to/test_file.py\n```\n\nThe broad CI-style non-GPU test command is approximately:\n\n```bash\npytest --timeout=3000 -W ignore::PendingDeprecationWarning -vv \\\n  --cov-config=pyproject.toml --cov-report=xml --cov=xinference \\\n  --ignore xinference/core/tests/test_continuous_batching.py \\\n  --ignore xinference/model/image/tests/test_stable_diffusion.py \\\n  --ignore xinference/model/image/tests/test_got_ocr2.py \\\n  --ignore xinference/model/audio/tests \\\n  --ignore xinference/model/embedding/tests/test_integrated_embedding.py \\\n  --ignore xinference/model/llm/transformers/tests/test_tensorizer.py \\\n  --ignore xinference/model/llm/tests/test_llm_model.py \\\n  --ignore xinference/model/llm/vllm \\\n  --ignore xinference/model/llm/sglang \\\n  --ignore xinference/client/tests/test_client.py \\\n  --ignore xinference/client/tests/test_async_client.py \\\n  --ignore xinference/model/llm/mlx \\\n  xinference\n```\n\nUse narrower commands for daily development. Many model tests require large\ndependencies, GPU, Metal, network access, or model downloads.\n\n## Frontend\n\nThe Web UI is under `frontend` and is a Next.js app (React, TypeScript,\nTailwind CSS), built as a static export and served by the Python backend from\n`xinference/ui/web/dist`.\n\nCommon commands:\n\n```bash\ncd frontend\nnpm ci\nnpm run dev\nnpm run build\nnpx eslint .\n```\n\nUse `npm run format` only when you intentionally want Prettier writes across\nthe frontend tree.\n\n## Documentation\n\nDocumentation source is in `doc/source`.\n\nCommon docs dependencies are included in the `doc` extra:\n\n```bash\npip install -e \".[doc]\"\ncd doc\nmake html\n```\n\nWhen changing CLI behavior, API behavior, deployment behavior, or model support,\nupdate the relevant documentation pages in `doc/source`.\n\n## Model and Runtime Conventions\n\n- Keep model-family logic inside the relevant `xinference/model/<family>/`\n  package.\n- Keep built-in model metadata changes close to existing specs and tests.\n- Be careful with lazy imports and optional dependencies. Import heavyweight\n  model libraries only where needed so unrelated installs still work.\n- Preserve platform guards for Linux-only, CUDA-only, and macOS Metal/MLX paths.\n- For distributed runtime changes, consider both local mode and\n  supervisor/worker mode.\n- For OpenAI-compatible behavior, verify request/response fields and streaming\n  behavior against existing API and client tests.\n\n## CI Expectations\n\nThe main CI workflow:\n\n- Runs `pre-commit run --all-files`.\n- Runs UI `npm ci`, `npx eslint .`, and Prettier check.\n- Tests Python 3.10 through 3.13 across Linux, macOS, and Windows.\n- Has special GPU and macOS Metal jobs for model-specific paths.\n\nBefore marking a change done, run the smallest meaningful validation command\nthat covers the behavior you changed, and mention any broader checks that were\nnot run because of environment cost or missing hardware.\n\n## Git and Review Hygiene\n\n- Keep commits scoped to the requested change.\n- Do not rewrite or revert user changes in an existing worktree unless asked.\n- If the active checkout is busy or on an unrelated branch, use a separate\n  worktree and a semantic branch name such as `fix/...`, `feat/...`, or\n  `docs/...`.\n- In PR reviews, inspect current GitHub review threads before adding duplicate\n  comments.\n"},"files":{"AGENTS.md":"# AI Agent Guidance\n\nGuidance for AI coding agents working in this repository.\n\n## Project Overview\n\nXinference is a Python model-serving project for language, embedding, rerank,\nimage, video, audio, and multimodal models. It exposes CLI commands, a Python\nclient, REST/OpenAI-compatible APIs, an xoscar-based distributed runtime, and a\nReact Web UI.\n\nPrimary package and entry points:\n\n- `xinference/`: Python package.\n- `xinference/deploy/cmdline.py`: CLI entry points for `xinference`,\n  `xinference-local`, `xinference-supervisor`, and `xinference-worker`.\n- `xinference/core/`: supervisor/worker runtime and actor orchestration.\n- `xinference/model/`: model families, engines, built-in model specs, and model\n  tests.\n- `xinference/api/`: API server and OpenAI-compatible routes.\n- `xinference/client/`: sync and async Python clients.\n- `frontend/`: Next.js Web UI.\n- `doc/source/`: Sphinx documentation.\n- `.github/workflows/python.yaml`: main lint and test CI.\n\n## Working Rules\n\n- Prefer small, focused changes that match the current module's style.\n- Preserve backward compatibility. If a breaking change is unavoidable, document\n  the reason and add deprecation behavior where practical.\n- Do not edit `xinference/thirdparty/` unless the task explicitly concerns\n  vendored code.\n- Avoid broad refactors while fixing a localized bug.\n- Keep public API behavior, request/response schemas, model registration names,\n  and CLI flags stable unless the task requires changing them.\n- Add or update tests for behavior changes. For model-runtime changes, prefer\n  tests close to the affected model family under `xinference/model/**/tests/`.\n- Use type hints for new Python code when practical; this project encourages\n  PEP 484 style annotations.\n- Treat docs and examples as user-facing API. Keep command examples accurate.\n\n## Environment Setup\n\nRecommended local setup:\n\n```bash\nconda create --name xinf python=3.12 nodejs\nconda activate xinf\npip install -e \".[dev]\"\n```\n\nNotes:\n\n- Wheel, sdist, and editable builds (including `pip install -e .`) run the\n  Web UI build through the in-tree build backend (`build_backend.py` /\n  `build_web.py`) unless `NO_WEB_UI=1` is set.\n- For Python 3.12 and newer, CI installs `setuptools<82`; use the same pin if\n  packaging or editable installs fail.\n- The project supports Python 3.10 through 3.13 in CI.\n- Optional model engines have extras in `pyproject.toml`, such as\n  `transformers`, `vllm`, `mlx`, `embedding`, `rerank`, `image`, `video`, and\n  `audio`.\n\n## Formatting and Linting\n\nPython formatting and checks are managed through pre-commit:\n\n```bash\npip install pre-commit\npre-commit run --files <modified-files>\n```\n\nFor a branch-wide check against upstream main:\n\n```bash\npre-commit run --from-ref=upstream/main --to-ref=HEAD --all-files\n```\n\nConfigured hooks include Black, end-of-file/trailing-whitespace checks, Ruff,\nisort, mypy with missing imports ignored, and codespell. Configuration lives in\n`.pre-commit-config.yaml` and `pyproject.toml`.\n\n## Python Tests\n\nRun focused tests first:\n\n```bash\npytest -vv path/to/test_file.py\n```\n\nThe broad CI-style non-GPU test command is approximately:\n\n```bash\npytest --timeout=3000 -W ignore::PendingDeprecationWarning -vv \\\n  --cov-config=pyproject.toml --cov-report=xml --cov=xinference \\\n  --ignore xinference/core/tests/test_continuous_batching.py \\\n  --ignore xinference/model/image/tests/test_stable_diffusion.py \\\n  --ignore xinference/model/image/tests/test_got_ocr2.py \\\n  --ignore xinference/model/audio/tests \\\n  --ignore xinference/model/embedding/tests/test_integrated_embedding.py \\\n  --ignore xinference/model/llm/transformers/tests/test_tensorizer.py \\\n  --ignore xinference/model/llm/tests/test_llm_model.py \\\n  --ignore xinference/model/llm/vllm \\\n  --ignore xinference/model/llm/sglang \\\n  --ignore xinference/client/tests/test_client.py \\\n  --ignore xinference/client/tests/test_async_client.py \\\n  --ignore xinference/model/llm/mlx \\\n  xinference\n```\n\nUse narrower commands for daily development. Many model tests require large\ndependencies, GPU, Metal, network access, or model downloads.\n\n## Frontend\n\nThe Web UI is under `frontend` and is a Next.js app (React, TypeScript,\nTailwind CSS), built as a static export and served by the Python backend from\n`xinference/ui/web/dist`.\n\nCommon commands:\n\n```bash\ncd frontend\nnpm ci\nnpm run dev\nnpm run build\nnpx eslint .\n```\n\nUse `npm run format` only when you intentionally want Prettier writes across\nthe frontend tree.\n\n## Documentation\n\nDocumentation source is in `doc/source`.\n\nCommon docs dependencies are included in the `doc` extra:\n\n```bash\npip install -e \".[doc]\"\ncd doc\nmake html\n```\n\nWhen changing CLI behavior, API behavior, deployment behavior, or model support,\nupdate the relevant documentation pages in `doc/source`.\n\n## Model and Runtime Conventions\n\n- Keep model-family logic inside the relevant `xinference/model/<family>/`\n  package.\n- Keep built-in model metadata changes close to existing specs and tests.\n- Be careful with lazy imports and optional dependencies. Import heavyweight\n  model libraries only where needed so unrelated installs still work.\n- Preserve platform guards for Linux-only, CUDA-only, and macOS Metal/MLX paths.\n- For distributed runtime changes, consider both local mode and\n  supervisor/worker mode.\n- For OpenAI-compatible behavior, verify request/response fields and streaming\n  behavior against existing API and client tests.\n\n## CI Expectations\n\nThe main CI workflow:\n\n- Runs `pre-commit run --all-files`.\n- Runs UI `npm ci`, `npx eslint .`, and Prettier check.\n- Tests Python 3.10 through 3.13 across Linux, macOS, and Windows.\n- Has special GPU and macOS Metal jobs for model-specific paths.\n\nBefore marking a change done, run the smallest meaningful validation command\nthat covers the behavior you changed, and mention any broader checks that were\nnot run because of environment cost or missing hardware.\n\n## Git and Review Hygiene\n\n- Keep commits scoped to the requested change.\n- Do not rewrite or revert user changes in an existing worktree unless asked.\n- If the active checkout is busy or on an unrelated branch, use a separate\n  worktree and a semantic branch name such as `fix/...`, `feat/...`, or\n  `docs/...`.\n- In PR reviews, inspect current GitHub review threads before adding duplicate\n  comments.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","title":"AGENTS.md","content":"# AI Agent Guidance\n\nGuidance for AI coding agents working in this repository.\n\n## Project Overview\n\nXinference is a Python model-serving project for language, embedding, rerank,\nimage, video, audio, and multimodal models. It exposes CLI commands, a Python\nclient, REST/OpenAI-compatible APIs, an xoscar-based distributed runtime, and a\nReact Web UI.\n\nPrimary package and entry points:\n\n- `xinference/`: Python package.\n- `xinference/deploy/cmdline.py`: CLI entry points for `xinference`,\n  `xinference-local`, `xinference-supervisor`, and `xinference-worker`.\n- `xinference/core/`: supervisor/worker runtime and actor orchestration.\n- `xinference/model/`: model families, engines, built-in model specs, and model\n  tests.\n- `xinference/api/`: API server and OpenAI-compatible routes.\n- `xinference/client/`: sync and async Python clients.\n- `frontend/`: Next.js Web UI.\n- `doc/source/`: Sphinx documentation.\n- `.github/workflows/python.yaml`: main lint and test CI.\n\n## Working Rules\n\n- Prefer small, focused changes that match the current module's style.\n- Preserve backward compatibility. If a breaking change is unavoidable, document\n  the reason and add deprecation behavior where practical.\n- Do not edit `xinference/thirdparty/` unless the task explicitly concerns\n  vendored code.\n- Avoid broad refactors while fixing a localized bug.\n- Keep public API behavior, request/response schemas, model registration names,\n  and CLI flags stable unless the task requires changing them.\n- Add or update tests for behavior changes. For model-runtime changes, prefer\n  tests close to the affected model family under `xinference/model/**/tests/`.\n- Use type hints for new Python code when practical; this project encourages\n  PEP 484 style annotations.\n- Treat docs and examples as user-facing API. Keep command examples accurate.\n\n## Environment Setup\n\nRecommended local setup:\n\n```bash\nconda create --name xinf python=3.12 nodejs\nconda activate xinf\npip install -e \".[dev]\"\n```\n\nNotes:\n\n- Wheel, sdist, and editable builds (including `pip install -e .`) run the\n  Web UI build through the in-tree build backend (`build_backend.py` /\n  `build_web.py`) unless `NO_WEB_UI=1` is set.\n- For Python 3.12 and newer, CI installs `setuptools<82`; use the same pin if\n  packaging or editable installs fail.\n- The project supports Python 3.10 through 3.13 in CI.\n- Optional model engines have extras in `pyproject.toml`, such as\n  `transformers`, `vllm`, `mlx`, `embedding`, `rerank`, `image`, `video`, and\n  `audio`.\n\n## Formatting and Linting\n\nPython formatting and checks are managed through pre-commit:\n\n```bash\npip install pre-commit\npre-commit run --files <modified-files>\n```\n\nFor a branch-wide check against upstream main:\n\n```bash\npre-commit run --from-ref=upstream/main --to-ref=HEAD --all-files\n```\n\nConfigured hooks include Black, end-of-file/trailing-whitespace checks, Ruff,\nisort, mypy with missing imports ignored, and codespell. Configuration lives in\n`.pre-commit-config.yaml` and `pyproject.toml`.\n\n## Python Tests\n\nRun focused tests first:\n\n```bash\npytest -vv path/to/test_file.py\n```\n\nThe broad CI-style non-GPU test command is approximately:\n\n```bash\npytest --timeout=3000 -W ignore::PendingDeprecationWarning -vv \\\n  --cov-config=pyproject.toml --cov-report=xml --cov=xinference \\\n  --ignore xinference/core/tests/test_continuous_batching.py \\\n  --ignore xinference/model/image/tests/test_stable_diffusion.py \\\n  --ignore xinference/model/image/tests/test_got_ocr2.py \\\n  --ignore xinference/model/audio/tests \\\n  --ignore xinference/model/embedding/tests/test_integrated_embedding.py \\\n  --ignore xinference/model/llm/transformers/tests/test_tensorizer.py \\\n  --ignore xinference/model/llm/tests/test_llm_model.py \\\n  --ignore xinference/model/llm/vllm \\\n  --ignore xinference/model/llm/sglang \\\n  --ignore xinference/client/tests/test_client.py \\\n  --ignore xinference/client/tests/test_async_client.py \\\n  --ignore xinference/model/llm/mlx \\\n  xinference\n```\n\nUse narrower commands for daily development. Many model tests require large\ndependencies, GPU, Metal, network access, or model downloads.\n\n## Frontend\n\nThe Web UI is under `frontend` and is a Next.js app (React, TypeScript,\nTailwind CSS), built as a static export and served by the Python backend from\n`xinference/ui/web/dist`.\n\nCommon commands:\n\n```bash\ncd frontend\nnpm ci\nnpm run dev\nnpm run build\nnpx eslint .\n```\n\nUse `npm run format` only when you intentionally want Prettier writes across\nthe frontend tree.\n\n## Documentation\n\nDocumentation source is in `doc/source`.\n\nCommon docs dependencies are included in the `doc` extra:\n\n```bash\npip install -e \".[doc]\"\ncd doc\nmake html\n```\n\nWhen changing CLI behavior, API behavior, deployment behavior, or model support,\nupdate the relevant documentation pages in `doc/source`.\n\n## Model and Runtime Conventions\n\n- Keep model-family logic inside the relevant `xinference/model/<family>/`\n  package.\n- Keep built-in model metadata changes close to existing specs and tests.\n- Be careful with lazy imports and optional dependencies. Import heavyweight\n  model libraries only where needed so unrelated installs still work.\n- Preserve platform guards for Linux-only, CUDA-only, and macOS Metal/MLX paths.\n- For distributed runtime changes, consider both local mode and\n  supervisor/worker mode.\n- For OpenAI-compatible behavior, verify request/response fields and streaming\n  behavior against existing API and client tests.\n\n## CI Expectations\n\nThe main CI workflow:\n\n- Runs `pre-commit run --all-files`.\n- Runs UI `npm ci`, `npx eslint .`, and Prettier check.\n- Tests Python 3.10 through 3.13 across Linux, macOS, and Windows.\n- Has special GPU and macOS Metal jobs for model-specific paths.\n\nBefore marking a change done, run the smallest meaningful validation command\nthat covers the behavior you changed, and mention any broader checks that were\nnot run because of environment cost or missing hardware.\n\n## Git and Review Hygiene\n\n- Keep commits scoped to the requested change.\n- Do not rewrite or revert user changes in an existing worktree unless asked.\n- If the active checkout is busy or on an unrelated branch, use a separate\n  worktree and a semantic branch name such as `fix/...`, `feat/...`, or\n  `docs/...`.\n- In PR reviews, inspect current GitHub review threads before adding duplicate\n  comments.\n","category":"root","tokens":1583}]}