{"owner":"arc53","repo":"DocsGPT","hasSkills":true,"totalSkillsCount":4,"totalTokensCount":4706,"categories":["root-instruction","mcp-config","marketplace","plugin-manifest"],"hasMcp":true,"mcpConfig":{"mcpServers":{"DocsGPT":{"command":"npx","args":["-y","@modelcontextprotocol/server-DocsGPT"]}}},"found":["AGENTS.md",".cursor/mcp.json",".claude-plugin/marketplace.json",".claude-plugin/plugin.json"],"skills":{"AGENTS.md":"# AGENTS.md\n\n- Read `CONTRIBUTING.md` before making non-trivial changes.\n- For day-to-day development and feature work, follow the development-environment workflow rather than defaulting to `setup.sh` / `setup.ps1`.\n- Avoid using the setup scripts during normal feature work unless the user explicitly asks for them. Users configure `.env` usually.\n- Try to follow red/green TDD\n\n### Check existing dev prerequisites first\n\nFor feature work, do **not** assume the environment needs to be recreated.\n\n- Check whether the user already has a Python virtual environment such as `venv/` or `.venv/`.\n- Check whether Postgres is already running and reachable via `POSTGRES_URI` (the canonical user-data store).\n- Check whether Redis is already running.\n- Reuse what is already working. Do not stop or recreate Postgres, Redis, or the Python environment unless the task is environment setup or troubleshooting.\n\n> MongoDB is **not** required for the default install. It is only needed if\n> the user opts into the Mongo vector-store backend (`VECTOR_STORE=mongodb`)\n> or is running the one-shot `scripts/db/backfill.py` to migrate existing\n> user data from the legacy Mongo-based install. `pymongo` is not a\n> dependency or an extra: install it separately with\n> `uv pip install 'pymongo>=4.6'` (a later `uv sync` removes it again).\n> The vector store needs Atlas `$vectorSearch`, not a plain `mongo` container.\n\n## Normal local development commands\n\nUse these commands once the dev prerequisites above are satisfied.\n\n### Backend\n\n```bash\nuv sync                    # deps from uv.lock + the dev group (test tools, ruff) + docsgpt itself, editable\nsource .venv/bin/activate  # macOS/Linux\n# pip instead: pip install -r docsgpt/requirements.txt -r tests/requirements.txt && pip install -e .\n# Optional extras (not installed by default); name every extra you want, `uv sync` removes the others:\n# uv sync --extra docling   # docling parser engine (OCR backend, structured output)\n# uv sync --extra milvus    # VECTOR_STORE=milvus\n# pip: pip install -r docsgpt/requirements-docling.txt (or requirements-milvus.txt); each file = core + the extra.\n# `uv pip install -r docsgpt/requirements-docling.txt` needs UV_INDEX_STRATEGY=unsafe-best-match\n# (the file adds the PyTorch CPU index; prefer `uv sync --extra docling`).\n```\n\nPython 3.12 (`requires-python >=3.12`; CI runs 3.12). A dev `.env` starts from\n`cp .env-template .env` and needs at least `POSTGRES_URI` and `INTERNAL_KEY`\n(see `docs/content/Deploying/Development-Environment.mdx`).\n\nThe backend is also an installable package (`pyproject.toml`, hatchling).\n`uv sync` (or `pip install -e .`) installs it editable and puts a `docsgpt` command on PATH;\n`python -m docsgpt <command>` works without that step. The quickest dev loop is\n`docsgpt dev`: it runs the API and the worker (with beat) from the checkout, both\nreloading on save, in one terminal; `--ui` adds the Vite dev server and\n`--mock-llm` runs `scripts/mock_llm.py` so no provider key is needed.\n`docsgpt doctor` checks PostgreSQL (reachable, schema current), Redis, the model\nprovider and the API port; run it first when something does not start.\nOther commands: `docsgpt api --reload`, `docsgpt worker`, `docsgpt beat`, `docsgpt migrate`,\n`docsgpt grant-admin`, `docsgpt prefetch-models`, `docsgpt verify-offline`. Runtime data (`.env`,\n`inputs/`, `indexes/`) lives in the checkout by default; `DOCSGPT_HOME` moves\nthat data home, and `DOCSGPT_ENV_FILE` selects only the `.env` file (see\n`docsgpt/core/paths.py`). `bash scripts/build_frontend.sh` builds the web UI\ninto `docsgpt/static` (gitignored); the API serves it when present\n(`docsgpt/ui.py`, switch `SERVE_UI`), and the package workflows run the script\nbefore `uv build` so the wheel ships it.\n\nDependencies are declared in `pyproject.toml` and locked in `uv.lock`; the\n`docsgpt/requirements*.txt` files are exported from the lock. To add or\nbump a package: edit `pyproject.toml`, run `uv lock`, then\n`bash scripts/export_requirements.sh` (CI fails if the exports are stale).\nNever edit the requirements files by hand.\n\nRun the API. For local dev, prefer the ASGI entrypoint under uvicorn — it\nserves the **whole** app, matches production, and hot-reloads:\n\n```bash\nuvicorn docsgpt.asgi:asgi_app --host 0.0.0.0 --port 7091 --reload\n```\n\n`flask --app docsgpt/app.py run --host=0.0.0.0 --port=7091` is a faster\ninner loop (quick startup, the Werkzeug interactive debugger), but it serves\n**only** the WSGI Flask app and omits the routes mounted on the ASGI shell\nin `docsgpt/asgi.py`:\n\n- the `/mcp` FastMCP endpoint,\n- the chat reconnect reader `GET /api/messages/<id>/events`,\n- the notification stream `GET /api/events`,\n- the remote-device command stream `GET /api/devices/sessions/<id>/events`, and\n- artifact downloads `GET /api/artifacts/<id>/download`.\n\nThese are native-async Starlette routes because they hold a response open\nfor a long time; on Flask each would pin a WSGI threadpool slot. Under\n`flask run` those paths 404: chat still works (`POST /stream` is a Flask\nroute), but live notifications, stream auto-resume, paired devices and\nartifact downloads don't. Use `flask run` only when you don't need them.\n`docsgpt api` and `docsgpt dev` serve the ASGI app. The user-facing copy of\nthis list is the \"ASGI-only features\" section of\n`docs/content/Deploying/Development-Environment.mdx`; keep the two in step.\n\nProduction uses `gunicorn -k docsgpt.gunicorn_worker.BoundedDrainUvicornWorker`\nagainst the same `docsgpt.asgi:asgi_app` target; see `docsgpt/Dockerfile` for\nthe full flag set.\n\nRun the Celery worker, with the embedded beat scheduler (`-B`), in a separate\nterminal:\n\n```bash\ncelery -A docsgpt.app.celery worker -l INFO -B\n```\n\n`docsgpt worker` (or `python -m docsgpt worker`) runs the same thing: it adds\n`-B` itself (`--no-beat` drops it) and picks the solo pool on macOS and Windows.\n\n**Beat must run somewhere.** It fires scheduled agent runs, source syncs, background-job sweeps,\nreconciliation, retention cleanups and the version check; without it they\nsilently never happen. Extra beat instances are safe (RedBeat holds a lock in\nRedis). Celery rejects `-B` on Windows: drop it there and run\n`celery -A docsgpt.app.celery beat -l INFO` (or `docsgpt beat`) next to the worker.\n\n**The worker is required for retrieval, not optional.** `EMBEDDINGS_DELEGATE_TO_WORKER`\ndefaults on, so the API embeds each query by dispatching to the worker rather than\nloading a model of its own — which keeps the API process around 285 MB instead of\nabout 660 MB. Without a worker consuming `EMBEDDINGS_QUEUE`, every search fails after\n`EMBEDDINGS_DELEGATE_TIMEOUT`. To run the API on its own, either set\n`EMBEDDINGS_DELEGATE_TO_WORKER=false` (loads the model in-process) or point\n`EMBEDDINGS_BASE_URL` at an embeddings service.\n\nOn macOS, prefer the solo pool for Celery:\n\n```bash\npython -m celery -A docsgpt.app.celery worker -l INFO -B --pool=solo -X analytics\n```\n\n`-X analytics` keeps the conversation-analytics sweep, which holds a worker for\nminutes, off the solo worker. With `DECISION_ANALYTICS_ENABLED` on, run\n`docsgpt worker --analytics` (or `celery ... worker -Q analytics -n analytics@%h`)\nnext to it; `docsgpt worker` excludes the queue itself, and `docsgpt dev` starts\nthe analytics worker while the setting is on.\n\nNote that `--pool=solo` costs roughly 350 ms per query embed against ~55 ms on the\ndefault prefork pool — nearly all of it the solo worker picking the message up, not\nthe embedding itself. That only affects local dev; production runs prefork.\n\nA bare worker (no `-Q`) consumes every configured queue, so one worker does the\nwhole job — app tasks, ingests, query embedding, and document parsing (the\n`read_document` tool / workflow native-file parse) alike. Use `-Q` only to split load:\nrun the main worker with `-Q docsgpt`, a dedicated (e.g. GPU-enabled) parser worker\nwith `-Q parsing` for heavy OCR, `-Q embeddings` to keep query latency off the ingest\npool, and `-Q ingest` for whole-source jobs (`INGEST_QUEUE`: uploads, remote and\nconnector ingests, re-ingests, syncs, graph builds, wiki conversion). One of those can\nhold a slot for hours, so on a shared worker they starve beat tasks; attachments and\nsingle wiki-page re-embeds stay on `docsgpt`. `analytics` needs a consumer too\n(`-Q analytics -c 1`, or listed in another worker's `-Q`). The ingest tasks parse\nin-process; only `read_document` is routed to `parsing`. A worker started with an\nexplicit `-Q` list consumes only that list, so every queue must be covered by some\nworker. When you split workers, keep `-B` on at least one of them.\n\n### Frontend\n\nInstall dependencies only when needed, then run the dev server:\n\n```bash\ncd frontend\nnpm install --include=dev\nnpm run dev\n```\n\n### Docs site\n\n```bash\ncd docs\nnpm install\n```\n\n### Python / backend changes validation\n\n```bash\nruff check .\npython -m pytest\n```\n\nThe suite needs the test dependencies (`uv sync` installs the `dev` group; pip users\ninstall `tests/requirements.txt`, and pytest-cov is mandatory because `pytest.ini`\npasses `--cov`). DB-backed tests start a throwaway cluster through\n`pytest-postgresql`, so the PostgreSQL server binaries (`pg_ctl`, `initdb`) must be\non `PATH` or reachable through `pg_config`; a running Postgres is not required.\nSee CONTRIBUTING.md \"Running the tests\".\n\nOn **macOS**, run the suite with `KMP_DUPLICATE_LIB_OK=TRUE`:\n\n```bash\nKMP_DUPLICATE_LIB_OK=TRUE python -m pytest\n```\n\n`faiss-cpu` and `torch` each ship their own LLVM OpenMP runtime, and loading\nboth into one process makes `libomp.dylib` abort the interpreter\n(`OMP: Error #15`). It is a macOS-only packaging clash, not a code fault: Linux\nresolves both to `libgomp`, which tolerates duplicates, so CI (`ubuntu-latest`)\nand the Docker images are unaffected. Without the variable, whether the run\naborts depends on which tests happen to load faiss and torch in the same\nprocess, so a green run on one selection and an abort on another is expected.\n\n### Frontend changes\n\n```bash\ncd frontend && npm run lint\ncd frontend && npm run build\n```\n\n### Documentation changes\n\n```bash\ncd docs && npm run dates:check && npm run build\n```\n\nRead `docs/AGENTS.md` first: every page carries a `lastUpdated` date that you bump when\nyou change what the page tells the reader.\n\nIf Vale is installed locally and you edited prose, also run:\n\n```bash\nvale .\n```\n\n## Repository map\n\n- `docsgpt/`: Flask backend, API routes, agent logic, retrieval, parsing, security, storage, Celery worker, and WSGI entrypoints.\n- `tests/`: backend unit/integration tests and test-only Python dependencies.\n- `frontend/`: Vite + React + TypeScript application.\n- `frontend/src/`: main UI code, including `components`, `conversation`, `hooks`, `locale`, `settings`, `upload`, and Redux store wiring in `store.ts`.\n- `docs/`: separate documentation site built with Next.js/Nextra.\n- `extensions/`: integrations and widgets — currently the Chatwoot webhook bridge and the React widget (published to npm as `docsgpt`). The Discord, Slack and Telegram bots live in their own repos (`arc53/discord-docsgpt-extension`, `arc53/slack-bot-docsgpt-extenstion`, `arc53/tg-bot-docsgpt-extenstion`); the old Chrome extension was removed and has no public successor.\n- `deployment/`: Docker Compose variants and Kubernetes manifests.\n\n## Coding rules\n\n### Backend\n\n- Follow PEP 8 and keep Python line length at or under 120 characters.\n- Use type hints for function arguments and return values.\n- Add Google-style docstrings to new or substantially changed functions and classes.\n- Add or update tests under `tests/` for backend behavior changes.\n- Keep changes narrow in `api`, `auth`, `security`, `parser`, `retriever`, and `storage` areas.\n\n### Backend Abstractions\n\n- LLM providers implement a common interface in `docsgpt/llm/` (add new providers by extending the base class).\n- Vector stores are abstracted in `docsgpt/vectorstore/`.\n- Parsers live in `docsgpt/parser/` and handle different document formats in the ingestion stage.\n- Agents and tools are in `docsgpt/agents/` and `docsgpt/agents/tools/`.\n- Celery setup/config lives in `docsgpt/celery_init.py` and `docsgpt/celeryconfig.py`.\n- Settings and env vars are managed via Pydantic in `docsgpt/core/settings/` (one module per domain, composed into `Settings`). Every field needs a `description`; regenerate the docs reference with `python -m docsgpt.core.settings.reference --write`.\n- REST routes are documented from the flask-restx Swagger document; after adding or changing a route, regenerate the docs snapshot with `python -m docsgpt.api.reference --write` (CI fails if `docs/data/swagger.json` is stale).\n\n### Frontend\n\n- Follow the existing ESLint + Prettier setup.\n- Prefer small, reusable functional components and hooks.\n- If shared state must be added, use Redux rather than introducing a new global state library.\n- Avoid broad UI refactors unless the task explicitly asks for them.\n- Do not re-create components if we already have some in the app.\n- Follow `frontend/DESIGN.md`: compose `components/ui/` parts and pick their look with props, use theme tokens, and\n  keep typography, spacing, radius and motion on its roles.\n- Every user-visible string, attributes included (`aria-label`, `label`, `placeholder`, `title`, `alt`), is a `t()`\n  key in all eight locales under `frontend/src/locale/` (`de en es fr jp ru zh zh-TW`). Admin pages stay English.\n\n#### Icons\n\nDocsGPT historically mixed three icon sources: `lucide-react`, inline SVG components, and\n`.svg` assets loaded via `<img src=…>`. For new code:\n\n1. **Prefer `lucide-react`** for standard UI affordances (close, chevron, search, trash,\n   plus, etc.). It tokenizes via `currentColor`, ships tree-shaken icons, and the codebase\n   already imports it in 30+ places. `<X className=\"size-4\" />`, `<ChevronDown />`, etc.\n2. **Use `assets/<name>.svg?react`** when you need a brand-specific or domain illustration\n   that doesn't exist in lucide (the app logo, robot fallback, send arrow,\n   etc.). Always set `fill=\"currentColor\"` / `stroke=\"currentColor\"` in the SVG file so\n   consumers can theme via Tailwind text classes.\n3. **Avoid `<img src={Asset}>` for new icons.** It blocks `currentColor` theming and\n   forces dark-variant duplicates (the audit removed several orphan dark/purple/white\n   variants in this branch). The pattern is acceptable for existing call sites — don't\n   bulk-migrate without a reason.\n\nThree pre-existing dark-variant pairs (`documentation`, `no-files`, `science-spark`) are\nhand-tuned multi-color illustrations, not pure inverts; they keep their `-dark` companion\nfiles until a per-illustration refactor.\n\n## PR readiness\n\nBefore opening a PR:\n\n- run the relevant validation commands above\n- confirm backend changes still work end-to-end after ingesting sample data when applicable\n- clearly summarize user-visible behavior changes\n- mention any config, dependency, or deployment implications\n- Ask your user to attach a screenshot or a video to it",".cursor/mcp.json":"# Model Context Protocol (MCP) Configuration\n\nPath: `.cursor/mcp.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.cursor/mcp.json)",".claude-plugin/marketplace.json":"# Claude Plugin Marketplace Catalog\n\nPath: `.claude-plugin/marketplace.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.claude-plugin/marketplace.json)",".claude-plugin/plugin.json":"# claude-plugin Plugin Manifest\n\nPath: `.claude-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.claude-plugin/plugin.json)"},"files":{"AGENTS.md":"# AGENTS.md\n\n- Read `CONTRIBUTING.md` before making non-trivial changes.\n- For day-to-day development and feature work, follow the development-environment workflow rather than defaulting to `setup.sh` / `setup.ps1`.\n- Avoid using the setup scripts during normal feature work unless the user explicitly asks for them. Users configure `.env` usually.\n- Try to follow red/green TDD\n\n### Check existing dev prerequisites first\n\nFor feature work, do **not** assume the environment needs to be recreated.\n\n- Check whether the user already has a Python virtual environment such as `venv/` or `.venv/`.\n- Check whether Postgres is already running and reachable via `POSTGRES_URI` (the canonical user-data store).\n- Check whether Redis is already running.\n- Reuse what is already working. Do not stop or recreate Postgres, Redis, or the Python environment unless the task is environment setup or troubleshooting.\n\n> MongoDB is **not** required for the default install. It is only needed if\n> the user opts into the Mongo vector-store backend (`VECTOR_STORE=mongodb`)\n> or is running the one-shot `scripts/db/backfill.py` to migrate existing\n> user data from the legacy Mongo-based install. `pymongo` is not a\n> dependency or an extra: install it separately with\n> `uv pip install 'pymongo>=4.6'` (a later `uv sync` removes it again).\n> The vector store needs Atlas `$vectorSearch`, not a plain `mongo` container.\n\n## Normal local development commands\n\nUse these commands once the dev prerequisites above are satisfied.\n\n### Backend\n\n```bash\nuv sync                    # deps from uv.lock + the dev group (test tools, ruff) + docsgpt itself, editable\nsource .venv/bin/activate  # macOS/Linux\n# pip instead: pip install -r docsgpt/requirements.txt -r tests/requirements.txt && pip install -e .\n# Optional extras (not installed by default); name every extra you want, `uv sync` removes the others:\n# uv sync --extra docling   # docling parser engine (OCR backend, structured output)\n# uv sync --extra milvus    # VECTOR_STORE=milvus\n# pip: pip install -r docsgpt/requirements-docling.txt (or requirements-milvus.txt); each file = core + the extra.\n# `uv pip install -r docsgpt/requirements-docling.txt` needs UV_INDEX_STRATEGY=unsafe-best-match\n# (the file adds the PyTorch CPU index; prefer `uv sync --extra docling`).\n```\n\nPython 3.12 (`requires-python >=3.12`; CI runs 3.12). A dev `.env` starts from\n`cp .env-template .env` and needs at least `POSTGRES_URI` and `INTERNAL_KEY`\n(see `docs/content/Deploying/Development-Environment.mdx`).\n\nThe backend is also an installable package (`pyproject.toml`, hatchling).\n`uv sync` (or `pip install -e .`) installs it editable and puts a `docsgpt` command on PATH;\n`python -m docsgpt <command>` works without that step. The quickest dev loop is\n`docsgpt dev`: it runs the API and the worker (with beat) from the checkout, both\nreloading on save, in one terminal; `--ui` adds the Vite dev server and\n`--mock-llm` runs `scripts/mock_llm.py` so no provider key is needed.\n`docsgpt doctor` checks PostgreSQL (reachable, schema current), Redis, the model\nprovider and the API port; run it first when something does not start.\nOther commands: `docsgpt api --reload`, `docsgpt worker`, `docsgpt beat`, `docsgpt migrate`,\n`docsgpt grant-admin`, `docsgpt prefetch-models`, `docsgpt verify-offline`. Runtime data (`.env`,\n`inputs/`, `indexes/`) lives in the checkout by default; `DOCSGPT_HOME` moves\nthat data home, and `DOCSGPT_ENV_FILE` selects only the `.env` file (see\n`docsgpt/core/paths.py`). `bash scripts/build_frontend.sh` builds the web UI\ninto `docsgpt/static` (gitignored); the API serves it when present\n(`docsgpt/ui.py`, switch `SERVE_UI`), and the package workflows run the script\nbefore `uv build` so the wheel ships it.\n\nDependencies are declared in `pyproject.toml` and locked in `uv.lock`; the\n`docsgpt/requirements*.txt` files are exported from the lock. To add or\nbump a package: edit `pyproject.toml`, run `uv lock`, then\n`bash scripts/export_requirements.sh` (CI fails if the exports are stale).\nNever edit the requirements files by hand.\n\nRun the API. For local dev, prefer the ASGI entrypoint under uvicorn — it\nserves the **whole** app, matches production, and hot-reloads:\n\n```bash\nuvicorn docsgpt.asgi:asgi_app --host 0.0.0.0 --port 7091 --reload\n```\n\n`flask --app docsgpt/app.py run --host=0.0.0.0 --port=7091` is a faster\ninner loop (quick startup, the Werkzeug interactive debugger), but it serves\n**only** the WSGI Flask app and omits the routes mounted on the ASGI shell\nin `docsgpt/asgi.py`:\n\n- the `/mcp` FastMCP endpoint,\n- the chat reconnect reader `GET /api/messages/<id>/events`,\n- the notification stream `GET /api/events`,\n- the remote-device command stream `GET /api/devices/sessions/<id>/events`, and\n- artifact downloads `GET /api/artifacts/<id>/download`.\n\nThese are native-async Starlette routes because they hold a response open\nfor a long time; on Flask each would pin a WSGI threadpool slot. Under\n`flask run` those paths 404: chat still works (`POST /stream` is a Flask\nroute), but live notifications, stream auto-resume, paired devices and\nartifact downloads don't. Use `flask run` only when you don't need them.\n`docsgpt api` and `docsgpt dev` serve the ASGI app. The user-facing copy of\nthis list is the \"ASGI-only features\" section of\n`docs/content/Deploying/Development-Environment.mdx`; keep the two in step.\n\nProduction uses `gunicorn -k docsgpt.gunicorn_worker.BoundedDrainUvicornWorker`\nagainst the same `docsgpt.asgi:asgi_app` target; see `docsgpt/Dockerfile` for\nthe full flag set.\n\nRun the Celery worker, with the embedded beat scheduler (`-B`), in a separate\nterminal:\n\n```bash\ncelery -A docsgpt.app.celery worker -l INFO -B\n```\n\n`docsgpt worker` (or `python -m docsgpt worker`) runs the same thing: it adds\n`-B` itself (`--no-beat` drops it) and picks the solo pool on macOS and Windows.\n\n**Beat must run somewhere.** It fires scheduled agent runs, source syncs, background-job sweeps,\nreconciliation, retention cleanups and the version check; without it they\nsilently never happen. Extra beat instances are safe (RedBeat holds a lock in\nRedis). Celery rejects `-B` on Windows: drop it there and run\n`celery -A docsgpt.app.celery beat -l INFO` (or `docsgpt beat`) next to the worker.\n\n**The worker is required for retrieval, not optional.** `EMBEDDINGS_DELEGATE_TO_WORKER`\ndefaults on, so the API embeds each query by dispatching to the worker rather than\nloading a model of its own — which keeps the API process around 285 MB instead of\nabout 660 MB. Without a worker consuming `EMBEDDINGS_QUEUE`, every search fails after\n`EMBEDDINGS_DELEGATE_TIMEOUT`. To run the API on its own, either set\n`EMBEDDINGS_DELEGATE_TO_WORKER=false` (loads the model in-process) or point\n`EMBEDDINGS_BASE_URL` at an embeddings service.\n\nOn macOS, prefer the solo pool for Celery:\n\n```bash\npython -m celery -A docsgpt.app.celery worker -l INFO -B --pool=solo -X analytics\n```\n\n`-X analytics` keeps the conversation-analytics sweep, which holds a worker for\nminutes, off the solo worker. With `DECISION_ANALYTICS_ENABLED` on, run\n`docsgpt worker --analytics` (or `celery ... worker -Q analytics -n analytics@%h`)\nnext to it; `docsgpt worker` excludes the queue itself, and `docsgpt dev` starts\nthe analytics worker while the setting is on.\n\nNote that `--pool=solo` costs roughly 350 ms per query embed against ~55 ms on the\ndefault prefork pool — nearly all of it the solo worker picking the message up, not\nthe embedding itself. That only affects local dev; production runs prefork.\n\nA bare worker (no `-Q`) consumes every configured queue, so one worker does the\nwhole job — app tasks, ingests, query embedding, and document parsing (the\n`read_document` tool / workflow native-file parse) alike. Use `-Q` only to split load:\nrun the main worker with `-Q docsgpt`, a dedicated (e.g. GPU-enabled) parser worker\nwith `-Q parsing` for heavy OCR, `-Q embeddings` to keep query latency off the ingest\npool, and `-Q ingest` for whole-source jobs (`INGEST_QUEUE`: uploads, remote and\nconnector ingests, re-ingests, syncs, graph builds, wiki conversion). One of those can\nhold a slot for hours, so on a shared worker they starve beat tasks; attachments and\nsingle wiki-page re-embeds stay on `docsgpt`. `analytics` needs a consumer too\n(`-Q analytics -c 1`, or listed in another worker's `-Q`). The ingest tasks parse\nin-process; only `read_document` is routed to `parsing`. A worker started with an\nexplicit `-Q` list consumes only that list, so every queue must be covered by some\nworker. When you split workers, keep `-B` on at least one of them.\n\n### Frontend\n\nInstall dependencies only when needed, then run the dev server:\n\n```bash\ncd frontend\nnpm install --include=dev\nnpm run dev\n```\n\n### Docs site\n\n```bash\ncd docs\nnpm install\n```\n\n### Python / backend changes validation\n\n```bash\nruff check .\npython -m pytest\n```\n\nThe suite needs the test dependencies (`uv sync` installs the `dev` group; pip users\ninstall `tests/requirements.txt`, and pytest-cov is mandatory because `pytest.ini`\npasses `--cov`). DB-backed tests start a throwaway cluster through\n`pytest-postgresql`, so the PostgreSQL server binaries (`pg_ctl`, `initdb`) must be\non `PATH` or reachable through `pg_config`; a running Postgres is not required.\nSee CONTRIBUTING.md \"Running the tests\".\n\nOn **macOS**, run the suite with `KMP_DUPLICATE_LIB_OK=TRUE`:\n\n```bash\nKMP_DUPLICATE_LIB_OK=TRUE python -m pytest\n```\n\n`faiss-cpu` and `torch` each ship their own LLVM OpenMP runtime, and loading\nboth into one process makes `libomp.dylib` abort the interpreter\n(`OMP: Error #15`). It is a macOS-only packaging clash, not a code fault: Linux\nresolves both to `libgomp`, which tolerates duplicates, so CI (`ubuntu-latest`)\nand the Docker images are unaffected. Without the variable, whether the run\naborts depends on which tests happen to load faiss and torch in the same\nprocess, so a green run on one selection and an abort on another is expected.\n\n### Frontend changes\n\n```bash\ncd frontend && npm run lint\ncd frontend && npm run build\n```\n\n### Documentation changes\n\n```bash\ncd docs && npm run dates:check && npm run build\n```\n\nRead `docs/AGENTS.md` first: every page carries a `lastUpdated` date that you bump when\nyou change what the page tells the reader.\n\nIf Vale is installed locally and you edited prose, also run:\n\n```bash\nvale .\n```\n\n## Repository map\n\n- `docsgpt/`: Flask backend, API routes, agent logic, retrieval, parsing, security, storage, Celery worker, and WSGI entrypoints.\n- `tests/`: backend unit/integration tests and test-only Python dependencies.\n- `frontend/`: Vite + React + TypeScript application.\n- `frontend/src/`: main UI code, including `components`, `conversation`, `hooks`, `locale`, `settings`, `upload`, and Redux store wiring in `store.ts`.\n- `docs/`: separate documentation site built with Next.js/Nextra.\n- `extensions/`: integrations and widgets — currently the Chatwoot webhook bridge and the React widget (published to npm as `docsgpt`). The Discord, Slack and Telegram bots live in their own repos (`arc53/discord-docsgpt-extension`, `arc53/slack-bot-docsgpt-extenstion`, `arc53/tg-bot-docsgpt-extenstion`); the old Chrome extension was removed and has no public successor.\n- `deployment/`: Docker Compose variants and Kubernetes manifests.\n\n## Coding rules\n\n### Backend\n\n- Follow PEP 8 and keep Python line length at or under 120 characters.\n- Use type hints for function arguments and return values.\n- Add Google-style docstrings to new or substantially changed functions and classes.\n- Add or update tests under `tests/` for backend behavior changes.\n- Keep changes narrow in `api`, `auth`, `security`, `parser`, `retriever`, and `storage` areas.\n\n### Backend Abstractions\n\n- LLM providers implement a common interface in `docsgpt/llm/` (add new providers by extending the base class).\n- Vector stores are abstracted in `docsgpt/vectorstore/`.\n- Parsers live in `docsgpt/parser/` and handle different document formats in the ingestion stage.\n- Agents and tools are in `docsgpt/agents/` and `docsgpt/agents/tools/`.\n- Celery setup/config lives in `docsgpt/celery_init.py` and `docsgpt/celeryconfig.py`.\n- Settings and env vars are managed via Pydantic in `docsgpt/core/settings/` (one module per domain, composed into `Settings`). Every field needs a `description`; regenerate the docs reference with `python -m docsgpt.core.settings.reference --write`.\n- REST routes are documented from the flask-restx Swagger document; after adding or changing a route, regenerate the docs snapshot with `python -m docsgpt.api.reference --write` (CI fails if `docs/data/swagger.json` is stale).\n\n### Frontend\n\n- Follow the existing ESLint + Prettier setup.\n- Prefer small, reusable functional components and hooks.\n- If shared state must be added, use Redux rather than introducing a new global state library.\n- Avoid broad UI refactors unless the task explicitly asks for them.\n- Do not re-create components if we already have some in the app.\n- Follow `frontend/DESIGN.md`: compose `components/ui/` parts and pick their look with props, use theme tokens, and\n  keep typography, spacing, radius and motion on its roles.\n- Every user-visible string, attributes included (`aria-label`, `label`, `placeholder`, `title`, `alt`), is a `t()`\n  key in all eight locales under `frontend/src/locale/` (`de en es fr jp ru zh zh-TW`). Admin pages stay English.\n\n#### Icons\n\nDocsGPT historically mixed three icon sources: `lucide-react`, inline SVG components, and\n`.svg` assets loaded via `<img src=…>`. For new code:\n\n1. **Prefer `lucide-react`** for standard UI affordances (close, chevron, search, trash,\n   plus, etc.). It tokenizes via `currentColor`, ships tree-shaken icons, and the codebase\n   already imports it in 30+ places. `<X className=\"size-4\" />`, `<ChevronDown />`, etc.\n2. **Use `assets/<name>.svg?react`** when you need a brand-specific or domain illustration\n   that doesn't exist in lucide (the app logo, robot fallback, send arrow,\n   etc.). Always set `fill=\"currentColor\"` / `stroke=\"currentColor\"` in the SVG file so\n   consumers can theme via Tailwind text classes.\n3. **Avoid `<img src={Asset}>` for new icons.** It blocks `currentColor` theming and\n   forces dark-variant duplicates (the audit removed several orphan dark/purple/white\n   variants in this branch). The pattern is acceptable for existing call sites — don't\n   bulk-migrate without a reason.\n\nThree pre-existing dark-variant pairs (`documentation`, `no-files`, `science-spark`) are\nhand-tuned multi-color illustrations, not pure inverts; they keep their `-dark` companion\nfiles until a per-illustration refactor.\n\n## PR readiness\n\nBefore opening a PR:\n\n- run the relevant validation commands above\n- confirm backend changes still work end-to-end after ingesting sample data when applicable\n- clearly summarize user-visible behavior changes\n- mention any config, dependency, or deployment implications\n- Ask your user to attach a screenshot or a video to it",".cursor/mcp.json":"# Model Context Protocol (MCP) Configuration\n\nPath: `.cursor/mcp.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.cursor/mcp.json)",".claude-plugin/marketplace.json":"# Claude Plugin Marketplace Catalog\n\nPath: `.claude-plugin/marketplace.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.claude-plugin/marketplace.json)",".claude-plugin/plugin.json":"# claude-plugin Plugin Manifest\n\nPath: `.claude-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.claude-plugin/plugin.json)"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\n- Read `CONTRIBUTING.md` before making non-trivial changes.\n- For day-to-day development and feature work, follow the development-environment workflow rather than defaulting to `setup.sh` / `setup.ps1`.\n- Avoid using the setup scripts during normal feature work unless the user explicitly asks for them. Users configure `.env` usually.\n- Try to follow red/green TDD\n\n### Check existing dev prerequisites first\n\nFor feature work, do **not** assume the environment needs to be recreated.\n\n- Check whether the user already has a Python virtual environment such as `venv/` or `.venv/`.\n- Check whether Postgres is already running and reachable via `POSTGRES_URI` (the canonical user-data store).\n- Check whether Redis is already running.\n- Reuse what is already working. Do not stop or recreate Postgres, Redis, or the Python environment unless the task is environment setup or troubleshooting.\n\n> MongoDB is **not** required for the default install. It is only needed if\n> the user opts into the Mongo vector-store backend (`VECTOR_STORE=mongodb`)\n> or is running the one-shot `scripts/db/backfill.py` to migrate existing\n> user data from the legacy Mongo-based install. `pymongo` is not a\n> dependency or an extra: install it separately with\n> `uv pip install 'pymongo>=4.6'` (a later `uv sync` removes it again).\n> The vector store needs Atlas `$vectorSearch`, not a plain `mongo` container.\n\n## Normal local development commands\n\nUse these commands once the dev prerequisites above are satisfied.\n\n### Backend\n\n```bash\nuv sync                    # deps from uv.lock + the dev group (test tools, ruff) + docsgpt itself, editable\nsource .venv/bin/activate  # macOS/Linux\n# pip instead: pip install -r docsgpt/requirements.txt -r tests/requirements.txt && pip install -e .\n# Optional extras (not installed by default); name every extra you want, `uv sync` removes the others:\n# uv sync --extra docling   # docling parser engine (OCR backend, structured output)\n# uv sync --extra milvus    # VECTOR_STORE=milvus\n# pip: pip install -r docsgpt/requirements-docling.txt (or requirements-milvus.txt); each file = core + the extra.\n# `uv pip install -r docsgpt/requirements-docling.txt` needs UV_INDEX_STRATEGY=unsafe-best-match\n# (the file adds the PyTorch CPU index; prefer `uv sync --extra docling`).\n```\n\nPython 3.12 (`requires-python >=3.12`; CI runs 3.12). A dev `.env` starts from\n`cp .env-template .env` and needs at least `POSTGRES_URI` and `INTERNAL_KEY`\n(see `docs/content/Deploying/Development-Environment.mdx`).\n\nThe backend is also an installable package (`pyproject.toml`, hatchling).\n`uv sync` (or `pip install -e .`) installs it editable and puts a `docsgpt` command on PATH;\n`python -m docsgpt <command>` works without that step. The quickest dev loop is\n`docsgpt dev`: it runs the API and the worker (with beat) from the checkout, both\nreloading on save, in one terminal; `--ui` adds the Vite dev server and\n`--mock-llm` runs `scripts/mock_llm.py` so no provider key is needed.\n`docsgpt doctor` checks PostgreSQL (reachable, schema current), Redis, the model\nprovider and the API port; run it first when something does not start.\nOther commands: `docsgpt api --reload`, `docsgpt worker`, `docsgpt beat`, `docsgpt migrate`,\n`docsgpt grant-admin`, `docsgpt prefetch-models`, `docsgpt verify-offline`. Runtime data (`.env`,\n`inputs/`, `indexes/`) lives in the checkout by default; `DOCSGPT_HOME` moves\nthat data home, and `DOCSGPT_ENV_FILE` selects only the `.env` file (see\n`docsgpt/core/paths.py`). `bash scripts/build_frontend.sh` builds the web UI\ninto `docsgpt/static` (gitignored); the API serves it when present\n(`docsgpt/ui.py`, switch `SERVE_UI`), and the package workflows run the script\nbefore `uv build` so the wheel ships it.\n\nDependencies are declared in `pyproject.toml` and locked in `uv.lock`; the\n`docsgpt/requirements*.txt` files are exported from the lock. To add or\nbump a package: edit `pyproject.toml`, run `uv lock`, then\n`bash scripts/export_requirements.sh` (CI fails if the exports are stale).\nNever edit the requirements files by hand.\n\nRun the API. For local dev, prefer the ASGI entrypoint under uvicorn — it\nserves the **whole** app, matches production, and hot-reloads:\n\n```bash\nuvicorn docsgpt.asgi:asgi_app --host 0.0.0.0 --port 7091 --reload\n```\n\n`flask --app docsgpt/app.py run --host=0.0.0.0 --port=7091` is a faster\ninner loop (quick startup, the Werkzeug interactive debugger), but it serves\n**only** the WSGI Flask app and omits the routes mounted on the ASGI shell\nin `docsgpt/asgi.py`:\n\n- the `/mcp` FastMCP endpoint,\n- the chat reconnect reader `GET /api/messages/<id>/events`,\n- the notification stream `GET /api/events`,\n- the remote-device command stream `GET /api/devices/sessions/<id>/events`, and\n- artifact downloads `GET /api/artifacts/<id>/download`.\n\nThese are native-async Starlette routes because they hold a response open\nfor a long time; on Flask each would pin a WSGI threadpool slot. Under\n`flask run` those paths 404: chat still works (`POST /stream` is a Flask\nroute), but live notifications, stream auto-resume, paired devices and\nartifact downloads don't. Use `flask run` only when you don't need them.\n`docsgpt api` and `docsgpt dev` serve the ASGI app. The user-facing copy of\nthis list is the \"ASGI-only features\" section of\n`docs/content/Deploying/Development-Environment.mdx`; keep the two in step.\n\nProduction uses `gunicorn -k docsgpt.gunicorn_worker.BoundedDrainUvicornWorker`\nagainst the same `docsgpt.asgi:asgi_app` target; see `docsgpt/Dockerfile` for\nthe full flag set.\n\nRun the Celery worker, with the embedded beat scheduler (`-B`), in a separate\nterminal:\n\n```bash\ncelery -A docsgpt.app.celery worker -l INFO -B\n```\n\n`docsgpt worker` (or `python -m docsgpt worker`) runs the same thing: it adds\n`-B` itself (`--no-beat` drops it) and picks the solo pool on macOS and Windows.\n\n**Beat must run somewhere.** It fires scheduled agent runs, source syncs, background-job sweeps,\nreconciliation, retention cleanups and the version check; without it they\nsilently never happen. Extra beat instances are safe (RedBeat holds a lock in\nRedis). Celery rejects `-B` on Windows: drop it there and run\n`celery -A docsgpt.app.celery beat -l INFO` (or `docsgpt beat`) next to the worker.\n\n**The worker is required for retrieval, not optional.** `EMBEDDINGS_DELEGATE_TO_WORKER`\ndefaults on, so the API embeds each query by dispatching to the worker rather than\nloading a model of its own — which keeps the API process around 285 MB instead of\nabout 660 MB. Without a worker consuming `EMBEDDINGS_QUEUE`, every search fails after\n`EMBEDDINGS_DELEGATE_TIMEOUT`. To run the API on its own, either set\n`EMBEDDINGS_DELEGATE_TO_WORKER=false` (loads the model in-process) or point\n`EMBEDDINGS_BASE_URL` at an embeddings service.\n\nOn macOS, prefer the solo pool for Celery:\n\n```bash\npython -m celery -A docsgpt.app.celery worker -l INFO -B --pool=solo -X analytics\n```\n\n`-X analytics` keeps the conversation-analytics sweep, which holds a worker for\nminutes, off the solo worker. With `DECISION_ANALYTICS_ENABLED` on, run\n`docsgpt worker --analytics` (or `celery ... worker -Q analytics -n analytics@%h`)\nnext to it; `docsgpt worker` excludes the queue itself, and `docsgpt dev` starts\nthe analytics worker while the setting is on.\n\nNote that `--pool=solo` costs roughly 350 ms per query embed against ~55 ms on the\ndefault prefork pool — nearly all of it the solo worker picking the message up, not\nthe embedding itself. That only affects local dev; production runs prefork.\n\nA bare worker (no `-Q`) consumes every configured queue, so one worker does the\nwhole job — app tasks, ingests, query embedding, and document parsing (the\n`read_document` tool / workflow native-file parse) alike. Use `-Q` only to split load:\nrun the main worker with `-Q docsgpt`, a dedicated (e.g. GPU-enabled) parser worker\nwith `-Q parsing` for heavy OCR, `-Q embeddings` to keep query latency off the ingest\npool, and `-Q ingest` for whole-source jobs (`INGEST_QUEUE`: uploads, remote and\nconnector ingests, re-ingests, syncs, graph builds, wiki conversion). One of those can\nhold a slot for hours, so on a shared worker they starve beat tasks; attachments and\nsingle wiki-page re-embeds stay on `docsgpt`. `analytics` needs a consumer too\n(`-Q analytics -c 1`, or listed in another worker's `-Q`). The ingest tasks parse\nin-process; only `read_document` is routed to `parsing`. A worker started with an\nexplicit `-Q` list consumes only that list, so every queue must be covered by some\nworker. When you split workers, keep `-B` on at least one of them.\n\n### Frontend\n\nInstall dependencies only when needed, then run the dev server:\n\n```bash\ncd frontend\nnpm install --include=dev\nnpm run dev\n```\n\n### Docs site\n\n```bash\ncd docs\nnpm install\n```\n\n### Python / backend changes validation\n\n```bash\nruff check .\npython -m pytest\n```\n\nThe suite needs the test dependencies (`uv sync` installs the `dev` group; pip users\ninstall `tests/requirements.txt`, and pytest-cov is mandatory because `pytest.ini`\npasses `--cov`). DB-backed tests start a throwaway cluster through\n`pytest-postgresql`, so the PostgreSQL server binaries (`pg_ctl`, `initdb`) must be\non `PATH` or reachable through `pg_config`; a running Postgres is not required.\nSee CONTRIBUTING.md \"Running the tests\".\n\nOn **macOS**, run the suite with `KMP_DUPLICATE_LIB_OK=TRUE`:\n\n```bash\nKMP_DUPLICATE_LIB_OK=TRUE python -m pytest\n```\n\n`faiss-cpu` and `torch` each ship their own LLVM OpenMP runtime, and loading\nboth into one process makes `libomp.dylib` abort the interpreter\n(`OMP: Error #15`). It is a macOS-only packaging clash, not a code fault: Linux\nresolves both to `libgomp`, which tolerates duplicates, so CI (`ubuntu-latest`)\nand the Docker images are unaffected. Without the variable, whether the run\naborts depends on which tests happen to load faiss and torch in the same\nprocess, so a green run on one selection and an abort on another is expected.\n\n### Frontend changes\n\n```bash\ncd frontend && npm run lint\ncd frontend && npm run build\n```\n\n### Documentation changes\n\n```bash\ncd docs && npm run dates:check && npm run build\n```\n\nRead `docs/AGENTS.md` first: every page carries a `lastUpdated` date that you bump when\nyou change what the page tells the reader.\n\nIf Vale is installed locally and you edited prose, also run:\n\n```bash\nvale .\n```\n\n## Repository map\n\n- `docsgpt/`: Flask backend, API routes, agent logic, retrieval, parsing, security, storage, Celery worker, and WSGI entrypoints.\n- `tests/`: backend unit/integration tests and test-only Python dependencies.\n- `frontend/`: Vite + React + TypeScript application.\n- `frontend/src/`: main UI code, including `components`, `conversation`, `hooks`, `locale`, `settings`, `upload`, and Redux store wiring in `store.ts`.\n- `docs/`: separate documentation site built with Next.js/Nextra.\n- `extensions/`: integrations and widgets — currently the Chatwoot webhook bridge and the React widget (published to npm as `docsgpt`). The Discord, Slack and Telegram bots live in their own repos (`arc53/discord-docsgpt-extension`, `arc53/slack-bot-docsgpt-extenstion`, `arc53/tg-bot-docsgpt-extenstion`); the old Chrome extension was removed and has no public successor.\n- `deployment/`: Docker Compose variants and Kubernetes manifests.\n\n## Coding rules\n\n### Backend\n\n- Follow PEP 8 and keep Python line length at or under 120 characters.\n- Use type hints for function arguments and return values.\n- Add Google-style docstrings to new or substantially changed functions and classes.\n- Add or update tests under `tests/` for backend behavior changes.\n- Keep changes narrow in `api`, `auth`, `security`, `parser`, `retriever`, and `storage` areas.\n\n### Backend Abstractions\n\n- LLM providers implement a common interface in `docsgpt/llm/` (add new providers by extending the base class).\n- Vector stores are abstracted in `docsgpt/vectorstore/`.\n- Parsers live in `docsgpt/parser/` and handle different document formats in the ingestion stage.\n- Agents and tools are in `docsgpt/agents/` and `docsgpt/agents/tools/`.\n- Celery setup/config lives in `docsgpt/celery_init.py` and `docsgpt/celeryconfig.py`.\n- Settings and env vars are managed via Pydantic in `docsgpt/core/settings/` (one module per domain, composed into `Settings`). Every field needs a `description`; regenerate the docs reference with `python -m docsgpt.core.settings.reference --write`.\n- REST routes are documented from the flask-restx Swagger document; after adding or changing a route, regenerate the docs snapshot with `python -m docsgpt.api.reference --write` (CI fails if `docs/data/swagger.json` is stale).\n\n### Frontend\n\n- Follow the existing ESLint + Prettier setup.\n- Prefer small, reusable functional components and hooks.\n- If shared state must be added, use Redux rather than introducing a new global state library.\n- Avoid broad UI refactors unless the task explicitly asks for them.\n- Do not re-create components if we already have some in the app.\n- Follow `frontend/DESIGN.md`: compose `components/ui/` parts and pick their look with props, use theme tokens, and\n  keep typography, spacing, radius and motion on its roles.\n- Every user-visible string, attributes included (`aria-label`, `label`, `placeholder`, `title`, `alt`), is a `t()`\n  key in all eight locales under `frontend/src/locale/` (`de en es fr jp ru zh zh-TW`). Admin pages stay English.\n\n#### Icons\n\nDocsGPT historically mixed three icon sources: `lucide-react`, inline SVG components, and\n`.svg` assets loaded via `<img src=…>`. For new code:\n\n1. **Prefer `lucide-react`** for standard UI affordances (close, chevron, search, trash,\n   plus, etc.). It tokenizes via `currentColor`, ships tree-shaken icons, and the codebase\n   already imports it in 30+ places. `<X className=\"size-4\" />`, `<ChevronDown />`, etc.\n2. **Use `assets/<name>.svg?react`** when you need a brand-specific or domain illustration\n   that doesn't exist in lucide (the app logo, robot fallback, send arrow,\n   etc.). Always set `fill=\"currentColor\"` / `stroke=\"currentColor\"` in the SVG file so\n   consumers can theme via Tailwind text classes.\n3. **Avoid `<img src={Asset}>` for new icons.** It blocks `currentColor` theming and\n   forces dark-variant duplicates (the audit removed several orphan dark/purple/white\n   variants in this branch). The pattern is acceptable for existing call sites — don't\n   bulk-migrate without a reason.\n\nThree pre-existing dark-variant pairs (`documentation`, `no-files`, `science-spark`) are\nhand-tuned multi-color illustrations, not pure inverts; they keep their `-dark` companion\nfiles until a per-illustration refactor.\n\n## PR readiness\n\nBefore opening a PR:\n\n- run the relevant validation commands above\n- confirm backend changes still work end-to-end after ingesting sample data when applicable\n- clearly summarize user-visible behavior changes\n- mention any config, dependency, or deployment implications\n- Ask your user to attach a screenshot or a video to it","isInternal":false,"tokens":3806,"sizeBytes":15005},{"name":"marketplace.json","path":".claude-plugin/marketplace.json","rawUrl":"https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.claude-plugin/marketplace.json","title":"Claude Plugin Marketplace Catalog","category":"marketplace","format":"json","content":"# Claude Plugin Marketplace Catalog\n\nPath: `.claude-plugin/marketplace.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.claude-plugin/marketplace.json)","tokens":300,"sizeBytes":192},{"name":"plugin.json","path":".claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.claude-plugin/plugin.json","title":"claude-plugin Plugin Manifest","category":"plugin-manifest","format":"json","content":"# claude-plugin Plugin Manifest\n\nPath: `.claude-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.claude-plugin/plugin.json)","tokens":300,"sizeBytes":178},{"name":"mcp.json","path":".cursor/mcp.json","rawUrl":"https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.cursor/mcp.json","title":"Model Context Protocol (MCP) Configuration","category":"mcp-config","format":"json","content":"# Model Context Protocol (MCP) Configuration\n\nPath: `.cursor/mcp.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.cursor/mcp.json)","tokens":300,"sizeBytes":171}],"systemPromptSnippet":"<yakaai_skills repo=\"arc53/DocsGPT\">\n<!-- File: AGENTS.md (Tokens: ~3806 | Category: root-instruction) -->\n# AGENTS.md\n\n- Read `CONTRIBUTING.md` before making non-trivial changes.\n- For day-to-day development and feature work, follow the development-environment workflow rather than defaulting to `setup.sh` / `setup.ps1`.\n- Avoid using the setup scripts during normal feature work unless the user explicitly asks for them. Users configure `.env` usually.\n- Try to follow red/green TDD\n\n### Check existing dev prerequisites first\n\nFor feature work, do **not** assume the environment needs to be recreated.\n\n- Check whether the user already has a Python virtual environment such as `venv/` or `.venv/`.\n- Check whether Postgres is already running and reachable via `POSTGRES_URI` (the canonical user-data store).\n- Check whether Redis is already running.\n- Reuse what is already working. Do not stop or recreate Postgres, Redis, or the Python environment unless the task is environment setup or troubleshooting.\n\n> MongoDB is **not** required for the default install. It is only needed if\n> the user opts into the Mongo vector-store backend (`VECTOR_STORE=mongodb`)\n> or is running the one-shot `scripts/db/backfill.py` to migrate existing\n> user data from the legacy Mongo-based install. `pymongo` is not a\n> dependency or an extra: install it separately with\n> `uv pip install 'pymongo>=4.6'` (a later `uv sync` removes it again).\n> The vector store needs Atlas `$vectorSearch`, not a plain `mongo` container.\n\n## Normal local development commands\n\nUse these commands once the dev prerequisites above are satisfied.\n\n### Backend\n\n```bash\nuv sync                    # deps from uv.lock + the dev group (test tools, ruff) + docsgpt itself, editable\nsource .venv/bin/activate  # macOS/Linux\n# pip instead: pip install -r docsgpt/requirements.txt -r tests/requirements.txt && pip install -e .\n# Optional extras (not installed by default); name every extra you want, `uv sync` removes the others:\n# uv sync --extra docling   # docling parser engine (OCR backend, structured output)\n# uv sync --extra milvus    # VECTOR_STORE=milvus\n# pip: pip install -r docsgpt/requirements-docling.txt (or requirements-milvus.txt); each file = core + the extra.\n# `uv pip install -r docsgpt/requirements-docling.txt` needs UV_INDEX_STRATEGY=unsafe-best-match\n# (the file adds the PyTorch CPU index; prefer `uv sync --extra docling`).\n```\n\nPython 3.12 (`requires-python >=3.12`; CI runs 3.12). A dev `.env` starts from\n`cp .env-template .env` and needs at least `POSTGRES_URI` and `INTERNAL_KEY`\n(see `docs/content/Deploying/Development-Environment.mdx`).\n\nThe backend is also an installable package (`pyproject.toml`, hatchling).\n`uv sync` (or `pip install -e .`) installs it editable and puts a `docsgpt` command on PATH;\n`python -m docsgpt <command>` works without that step. The quickest dev loop is\n`docsgpt dev`: it runs the API and the worker (with beat) from the checkout, both\nreloading on save, in one terminal; `--ui` adds the Vite dev server and\n`--mock-llm` runs `scripts/mock_llm.py` so no provider key is needed.\n`docsgpt doctor` checks PostgreSQL (reachable, schema current), Redis, the model\nprovider and the API port; run it first when something does not start.\nOther commands: `docsgpt api --reload`, `docsgpt worker`, `docsgpt beat`, `docsgpt migrate`,\n`docsgpt grant-admin`, `docsgpt prefetch-models`, `docsgpt verify-offline`. Runtime data (`.env`,\n`inputs/`, `indexes/`) lives in the checkout by default; `DOCSGPT_HOME` moves\nthat data home, and `DOCSGPT_ENV_FILE` selects only the `.env` file (see\n`docsgpt/core/paths.py`). `bash scripts/build_frontend.sh` builds the web UI\ninto `docsgpt/static` (gitignored); the API serves it when present\n(`docsgpt/ui.py`, switch `SERVE_UI`), and the package workflows run the script\nbefore `uv build` so the wheel ships it.\n\nDependencies are declared in `pyproject.toml` and locked in `uv.lock`; the\n`docsgpt/requirements*.txt` files are exported from the lock. To add or\nbump a package: edit `pyproject.toml`, run `uv lock`, then\n`bash scripts/export_requirements.sh` (CI fails if the exports are stale).\nNever edit the requirements files by hand.\n\nRun the API. For local dev, prefer the ASGI entrypoint under uvicorn — it\nserves the **whole** app, matches production, and hot-reloads:\n\n```bash\nuvicorn docsgpt.asgi:asgi_app --host 0.0.0.0 --port 7091 --reload\n```\n\n`flask --app docsgpt/app.py run --host=0.0.0.0 --port=7091` is a faster\ninner loop (quick startup, the Werkzeug interactive debugger), but it serves\n**only** the WSGI Flask app and omits the routes mounted on the ASGI shell\nin `docsgpt/asgi.py`:\n\n- the `/mcp` FastMCP endpoint,\n- the chat reconnect reader `GET /api/messages/<id>/events`,\n- the notification stream `GET /api/events`,\n- the remote-device command stream `GET /api/devices/sessions/<id>/events`, and\n- artifact downloads `GET /api/artifacts/<id>/download`.\n\nThese are native-async Starlette routes because they hold a response open\nfor a long time; on Flask each would pin a WSGI threadpool slot. Under\n`flask run` those paths 404: chat still works (`POST /stream` is a Flask\nroute), but live notifications, stream auto-resume, paired devices and\nartifact downloads don't. Use `flask run` only when you don't need them.\n`docsgpt api` and `docsgpt dev` serve the ASGI app. The user-facing copy of\nthis list is the \"ASGI-only features\" section of\n`docs/content/Deploying/Development-Environment.mdx`; keep the two in step.\n\nProduction uses `gunicorn -k docsgpt.gunicorn_worker.BoundedDrainUvicornWorker`\nagainst the same `docsgpt.asgi:asgi_app` target; see `docsgpt/Dockerfile` for\nthe full flag set.\n\nRun the Celery worker, with the embedded beat scheduler (`-B`), in a separate\nterminal:\n\n```bash\ncelery -A docsgpt.app.celery worker -l INFO -B\n```\n\n`docsgpt worker` (or `python -m docsgpt worker`) runs the same thing: it adds\n`-B` itself (`--no-beat` drops it) and picks the solo pool on macOS and Windows.\n\n**Beat must run somewhere.** It fires scheduled agent runs, source syncs, background-job sweeps,\nreconciliation, retention cleanups and the version check; without it they\nsilently never happen. Extra beat instances are safe (RedBeat holds a lock in\nRedis). Celery rejects `-B` on Windows: drop it there and run\n`celery -A docsgpt.app.celery beat -l INFO` (or `docsgpt beat`) next to the worker.\n\n**The worker is required for retrieval, not optional.** `EMBEDDINGS_DELEGATE_TO_WORKER`\ndefaults on, so the API embeds each query by dispatching to the worker rather than\nloading a model of its own — which keeps the API process around 285 MB instead of\nabout 660 MB. Without a worker consuming `EMBEDDINGS_QUEUE`, every search fails after\n`EMBEDDINGS_DELEGATE_TIMEOUT`. To run the API on its own, either set\n`EMBEDDINGS_DELEGATE_TO_WORKER=false` (loads the model in-process) or point\n`EMBEDDINGS_BASE_URL` at an embeddings service.\n\nOn macOS, prefer the solo pool for Celery:\n\n```bash\npython -m celery -A docsgpt.app.celery worker -l INFO -B --pool=solo -X analytics\n```\n\n`-X analytics` keeps the conversation-analytics sweep, which holds a worker for\nminutes, off the solo worker. With `DECISION_ANALYTICS_ENABLED` on, run\n`docsgpt worker --analytics` (or `celery ... worker -Q analytics -n analytics@%h`)\nnext to it; `docsgpt worker` excludes the queue itself, and `docsgpt dev` starts\nthe analytics worker while the setting is on.\n\nNote that `--pool=solo` costs roughly 350 ms per query embed against ~55 ms on the\ndefault prefork pool — nearly all of it the solo worker picking the message up, not\nthe embedding itself. That only affects local dev; production runs prefork.\n\nA bare worker (no `-Q`) consumes every configured queue, so one worker does the\nwhole job — app tasks, ingests, query embedding, and document parsing (the\n`read_document` tool / workflow native-file parse) alike. Use `-Q` only to split load:\nrun the main worker with `-Q docsgpt`, a dedicated (e.g. GPU-enabled) parser worker\nwith `-Q parsing` for heavy OCR, `-Q embeddings` to keep query latency off the ingest\npool, and `-Q ingest` for whole-source jobs (`INGEST_QUEUE`: uploads, remote and\nconnector ingests, re-ingests, syncs, graph builds, wiki conversion). One of those can\nhold a slot for hours, so on a shared worker they starve beat tasks; attachments and\nsingle wiki-page re-embeds stay on `docsgpt`. `analytics` needs a consumer too\n(`-Q analytics -c 1`, or listed in another worker's `-Q`). The ingest tasks parse\nin-process; only `read_document` is routed to `parsing`. A worker started with an\nexplicit `-Q` list consumes only that list, so every queue must be covered by some\nworker. When you split workers, keep `-B` on at least one of them.\n\n### Frontend\n\nInstall dependencies only when needed, then run the dev server:\n\n```bash\ncd frontend\nnpm install --include=dev\nnpm run dev\n```\n\n### Docs site\n\n```bash\ncd docs\nnpm install\n```\n\n### Python / backend changes validation\n\n```bash\nruff check .\npython -m pytest\n```\n\nThe suite needs the test dependencies (`uv sync` installs the `dev` group; pip users\ninstall `tests/requirements.txt`, and pytest-cov is mandatory because `pytest.ini`\npasses `--cov`). DB-backed tests start a throwaway cluster through\n`pytest-postgresql`, so the PostgreSQL server binaries (`pg_ctl`, `initdb`) must be\non `PATH` or reachable through `pg_config`; a running Postgres is not required.\nSee CONTRIBUTING.md \"Running the tests\".\n\nOn **macOS**, run the suite with `KMP_DUPLICATE_LIB_OK=TRUE`:\n\n```bash\nKMP_DUPLICATE_LIB_OK=TRUE python -m pytest\n```\n\n`faiss-cpu` and `torch` each ship their own LLVM OpenMP runtime, and loading\nboth into one process makes `libomp.dylib` abort the interpreter\n(`OMP: Error #15`). It is a macOS-only packaging clash, not a code fault: Linux\nresolves both to `libgomp`, which tolerates duplicates, so CI (`ubuntu-latest`)\nand the Docker images are unaffected. Without the variable, whether the run\naborts depends on which tests happen to load faiss and torch in the same\nprocess, so a green run on one selection and an abort on another is expected.\n\n### Frontend changes\n\n```bash\ncd frontend && npm run lint\ncd frontend && npm run build\n```\n\n### Documentation changes\n\n```bash\ncd docs && npm run dates:check && npm run build\n```\n\nRead `docs/AGENTS.md` first: every page carries a `lastUpdated` date that you bump when\nyou change what the page tells the reader.\n\nIf Vale is installed locally and you edited prose, also run:\n\n```bash\nvale .\n```\n\n## Repository map\n\n- `docsgpt/`: Flask backend, API routes, agent logic, retrieval, parsing, security, storage, Celery worker, and WSGI entrypoints.\n- `tests/`: backend unit/integration tests and test-only Python dependencies.\n- `frontend/`: Vite + React + TypeScript application.\n- `frontend/src/`: main UI code, including `components`, `conversation`, `hooks`, `locale`, `settings`, `upload`, and Redux store wiring in `store.ts`.\n- `docs/`: separate documentation site built with Next.js/Nextra.\n- `extensions/`: integrations and widgets — currently the Chatwoot webhook bridge and the React widget (published to npm as `docsgpt`). The Discord, Slack and Telegram bots live in their own repos (`arc53/discord-docsgpt-extension`, `arc53/slack-bot-docsgpt-extenstion`, `arc53/tg-bot-docsgpt-extenstion`); the old Chrome extension was removed and has no public successor.\n- `deployment/`: Docker Compose variants and Kubernetes manifests.\n\n## Coding rules\n\n### Backend\n\n- Follow PEP 8 and keep Python line length at or under 120 characters.\n- Use type hints for function arguments and return values.\n- Add Google-style docstrings to new or substantially changed functions and classes.\n- Add or update tests under `tests/` for backend behavior changes.\n- Keep changes narrow in `api`, `auth`, `security`, `parser`, `retriever`, and `storage` areas.\n\n### Backend Abstractions\n\n- LLM providers implement a common interface in `docsgpt/llm/` (add new providers by extending the base class).\n- Vector stores are abstracted in `docsgpt/vectorstore/`.\n- Parsers live in `docsgpt/parser/` and handle different document formats in the ingestion stage.\n- Agents and tools are in `docsgpt/agents/` and `docsgpt/agents/tools/`.\n- Celery setup/config lives in `docsgpt/celery_init.py` and `docsgpt/celeryconfig.py`.\n- Settings and env vars are managed via Pydantic in `docsgpt/core/settings/` (one module per domain, composed into `Settings`). Every field needs a `description`; regenerate the docs reference with `python -m docsgpt.core.settings.reference --write`.\n- REST routes are documented from the flask-restx Swagger document; after adding or changing a route, regenerate the docs snapshot with `python -m docsgpt.api.reference --write` (CI fails if `docs/data/swagger.json` is stale).\n\n### Frontend\n\n- Follow the existing ESLint + Prettier setup.\n- Prefer small, reusable functional components and hooks.\n- If shared state must be added, use Redux rather than introducing a new global state library.\n- Avoid broad UI refactors unless the task explicitly asks for them.\n- Do not re-create components if we already have some in the app.\n- Follow `frontend/DESIGN.md`: compose `components/ui/` parts and pick their look with props, use theme tokens, and\n  keep typography, spacing, radius and motion on its roles.\n- Every user-visible string, attributes included (`aria-label`, `label`, `placeholder`, `title`, `alt`), is a `t()`\n  key in all eight locales under `frontend/src/locale/` (`de en es fr jp ru zh zh-TW`). Admin pages stay English.\n\n#### Icons\n\nDocsGPT historically mixed three icon sources: `lucide-react`, inline SVG components, and\n`.svg` assets loaded via `<img src=…>`. For new code:\n\n1. **Prefer `lucide-react`** for standard UI affordances (close, chevron, search, trash,\n   plus, etc.). It tokenizes via `currentColor`, ships tree-shaken icons, and the codebase\n   already imports it in 30+ places. `<X className=\"size-4\" />`, `<ChevronDown />`, etc.\n2. **Use `assets/<name>.svg?react`** when you need a brand-specific or domain illustration\n   that doesn't exist in lucide (the app logo, robot fallback, send arrow,\n   etc.). Always set `fill=\"currentColor\"` / `stroke=\"currentColor\"` in the SVG file so\n   consumers can theme via Tailwind text classes.\n3. **Avoid `<img src={Asset}>` for new icons.** It blocks `currentColor` theming and\n   forces dark-variant duplicates (the audit removed several orphan dark/purple/white\n   variants in this branch). The pattern is acceptable for existing call sites — don't\n   bulk-migrate without a reason.\n\nThree pre-existing dark-variant pairs (`documentation`, `no-files`, `science-spark`) are\nhand-tuned multi-color illustrations, not pure inverts; they keep their `-dark` companion\nfiles until a per-illustration refactor.\n\n## PR readiness\n\nBefore opening a PR:\n\n- run the relevant validation commands above\n- confirm backend changes still work end-to-end after ingesting sample data when applicable\n- clearly summarize user-visible behavior changes\n- mention any config, dependency, or deployment implications\n- Ask your user to attach a screenshot or a video to it\n\n<!-- File: .claude-plugin/marketplace.json (Tokens: ~300 | Category: marketplace) -->\n# Claude Plugin Marketplace Catalog\n\nPath: `.claude-plugin/marketplace.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.claude-plugin/marketplace.json)\n\n<!-- File: .claude-plugin/plugin.json (Tokens: ~300 | Category: plugin-manifest) -->\n# claude-plugin Plugin Manifest\n\nPath: `.claude-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.claude-plugin/plugin.json)\n\n<!-- File: .cursor/mcp.json (Tokens: ~300 | Category: mcp-config) -->\n# Model Context Protocol (MCP) Configuration\n\nPath: `.cursor/mcp.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/arc53/DocsGPT/HEAD/.cursor/mcp.json)\n</yakaai_skills>"}