{"owner":"yusufkaraaslan","repo":"Skill_Seekers","hasSkills":true,"hasMcp":true,"mcpConfig":{"mcpServers":{"Skill_Seekers":{"command":"npx","args":["-y","@modelcontextprotocol/server-Skill_Seekers"]}}},"found":["CLAUDE.md","AGENTS.md"],"skills":{"CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\n**Skill Seekers** converts documentation from 18 source types into production-ready formats for 21+ AI platforms (LLM platforms, RAG frameworks, vector databases, AI coding assistants). Published on PyPI as `skill-seekers`.\n\n**Version:** 3.9.0 (dev; last release 3.8.0) — source of truth is `src/skill_seekers/_version.py` | **Python:** 3.10+ | **Website:** https://skillseekersweb.com/\n\n**Architecture:** See `docs/UML_ARCHITECTURE.md` for UML diagrams and module overview. StarUML project at `docs/UML/skill_seekers.mdj`. Refactor state/history: `docs/UNIFICATION_PLAN.md` (Grand Unification — all 5 phases done; remaining cosmetic items listed there).\n\n## Essential Commands\n\n```bash\n# REQUIRED before running tests or CLI (src/ layout)\npip install -e .\n\n# Run all tests (NEVER skip - all must pass before commits)\npytest tests/ -v\n\n# Fast iteration (skip slow MCP tests ~20min)\npytest tests/ --ignore=tests/test_mcp_fastmcp.py --ignore=tests/test_mcp_server.py --ignore=tests/test_install_skill_e2e.py -q\n\n# Single test\npytest tests/test_scraper_features.py::test_detect_language -vv -s\n\n# Code quality (must pass before push - matches CI)\nuvx ruff check src/ tests/\nuvx ruff format --check src/ tests/\nmypy src/skill_seekers  # continue-on-error in CI\n\n# Auto-fix lint/format issues\nuvx ruff check --fix --unsafe-fixes src/ tests/\nuvx ruff format src/ tests/\n\n# Build & publish\nuv build\nuv publish\n```\n\n## CI Matrix\n\nRuns on push/PR to `main` or `development`. Lint job (Python 3.12, Ubuntu) + Test job (Ubuntu + macOS, Python 3.10/3.11/3.12, excludes macOS+3.10). Both must pass for merge.\n\n## Git Workflow\n\n- **Main branch:** `main` (requires tests + 1 review)\n- **Development branch:** `development` (default PR target, requires tests)\n- **Feature branches:** `feature/{task-id}-{description}` from `development`\n- PRs always target `development`, never `main` directly\n\n## Architecture\n\n### CLI: Unified create command\n\nEntry point `src/skill_seekers/cli/main.py`. The `create` command is the **primary** entry point for skill creation — it auto-detects source type and routes to the appropriate `SkillConverter`. The `scan` command (added in #327) is a separate discovery step for projects with multiple frameworks; it emits one config file per detected framework and you then run `create` on each.\n\n```\nskill-seekers create <source>     # Auto-detect: URL, owner/repo, ./path, file.pdf, etc.\nskill-seekers scan <dir>          # AI-driven discovery → emits one config per detected framework + <project>-codebase.json\nskill-seekers package <dir>       # Package for platform (--target claude/gemini/openai/markdown/minimax/opencode/kimi/deepseek/qwen/openrouter/together/fireworks/atlas/langchain/llama-index/haystack/chroma/faiss/weaviate/qdrant/pinecone/ibm-bob)\n```\n\n### Scan command (issue #327)\n\n`skill-seekers scan <dir>` is an AI-driven project knowledge-base bootstrapper. Pipeline in `src/skill_seekers/cli/scan_command.py`:\n\n1. `collect_signals()` in `signal_collectors.py` — deterministic, bounded gathering of manifests + README + Dockerfile/CI + sampled source files + git remote. **Per-kind byte budgets** (24 KB manifest / 6 KB README / 6 KB CI / 28 KB samples, total 64 KB) so a fat package.json can't crowd out other kinds. `_SOURCE_DIRS` covers ~14 layouts (Go `cmd/`, Rust `crates/`, JS monorepo `apps/packages/`, Maven `source/`, Django at root); also walks root one level deep for flat-layout Python.\n2. `detect_with_ai(bundle, AgentClient)` — one LLM call, structured JSON output. **Source signals are first-2-KB of each file** (whole-file sampling, no regex parsing — added in WS4 because regex missed Go multi-line imports + Rust `mod`/`extern crate`). Canonical-slug prompt + the canonical-name resolver are coupled — change one, update the other.\n3. `resolve_or_generate_with_status()` — for each detection: try `out_dir/<slug>.json` (cache from prior run), then `resolve_config_path` from `config_fetcher` with multiple canonical name candidates (`_canonical_name_candidates` handles `\"Godot Engine\"` → `\"godot\"`, plus CJK / European suffixes like `\"Godot 引擎\"`, `\"React フレームワーク\"`, `\"Lodash Bibliothek\"`), then `generate_config_with_ai` as the last resort. Always appends `.json` to lookup names so local-disk and user-dir resolution actually finds files. Always stamps `metadata.detected_version` (nested, not top-level — `metadata.version` already exists and means config-schema version).\n4. `emit_codebase_config()` — always writes `<project>-codebase.json` (a `type: local` source pointed at the project root).\n5. `diff_against_existing()` — keyed by **filename slug** (not internal `data[\"name\"]`) so re-scans don't churn when the AI returns a display name vs the registry canonical slug.\n6. `_archive_removed()` — when a config disappears from detections, MOVE (not delete — user may have hand-edited) to `out_dir/.archived/<UTC-timestamp>/`. Runs after diff, before fresh writes.\n7. `maybe_publish()` — **native async** (WS11). Opt-in submission of freshly AI-generated configs to the community registry. Pre-checks `GITHUB_TOKEN`. Idempotency guard: `_find_existing_issue` queries GitHub Search API for an existing open issue with the same config name before submitting. Retries transient failures (rate limit, 5xx) with 0s/5s/15s backoff. `_prompt_async` wraps `input()` via `asyncio.to_thread` so the event loop isn't blocked.\n\n**CLI dispatch** uses the `COMMAND_CLASSES` table in `main.py` (added in WS1). `scan` and `doctor` are dispatched as `Cls(args).execute()` consuming the parsed argparse namespace directly — no `_reconstruct_argv` hack, no duplicate argparse. `ScanCommand.execute()` is the single `asyncio.run` boundary wrapping `run_scan` (sync) + `maybe_publish` (async). Remaining ~14 commands still use the legacy `COMMAND_MODULES` dispatch; they're flagged for migration.\n\n**Cost guardrails**: `--max-ai-generations N` (default 10) caps unbounded AI generation; `--dry-run` previews without writing or invoking AI; `--probe-urls` HEAD-checks AI-generated URLs with retry-on-404 and stamps `metadata._url_unverified` on confirmed-bad URLs.\n\n**Safety**: All writes use `_atomic_write_json` (`os.replace` after writing to `.tmp`) so a `KeyboardInterrupt` mid-write can't corrupt configs. `_safe_size` guards `stat()` so broken symlinks don't crash the scan. `ScanCommand.execute` calls `logging.basicConfig` so `logger.warning`/`error` is visible; exit code is non-zero when no configs and no codebase config were emitted.\n\n**Public constant**: `SourceDetector.CODE_PROJECT_MARKERS` (was `_CODE_PROJECT_MARKERS`) — shared between source_detector + signal_collectors. ~50 manifest types now (Pipfile, environment.yml, deno.json, flake.nix, Chart.yaml, deps.edn, dune-project, BUILD.bazel, …). Public so cross-module access doesn't reach into a private attribute.\n\n### SkillConverter Pattern (Template Method + Factory)\n\nAll 18 source types implement the `SkillConverter` base class (`skill_converter.py`):\n\n```python\nconverter = get_converter(\"web\", config)  # Factory lookup\nconverter.run()  # Template: extract() → build_skill()\n```\n\nRegistry in `CONVERTER_REGISTRY` maps source type → (module, class). `create_command.py` builds config from `ExecutionContext`, calls `get_converter()`, then runs centralized enhancement. `get_converter(\"config\", {...})` constructs `UnifiedScraper` from the same factory-shaped dict (no special cases in create_command/MCP). The base resolves `skill_dir` once (strips trailing separators) and derives `data_file` via `data_file_for()` — subclasses must not re-derive paths.\n\n### DocumentSkillBuilder (build side of 9 document scrapers)\n\n`cli/document_skill_builder.py:DocumentSkillBuilder` sits between `SkillConverter` and the 9 document scrapers (epub, word, pptx, html, pdf, jupyter, man, rss, chat). It owns `categorize_content`, reference-file writing (tables, truncation, image guard), `index.md` + `SKILL.md` generation, and `load_extracted_data`. Variation points are class attrs (`DOC_NOUN`, `SOURCE_LABEL`, `LOAD_TOTAL_KEY`, `PATTERN_KEYWORDS`, `RANGE_LABEL`, …) and small hook methods (`category_stem`, `_write_reference_section`, `_write_skill_md_metadata`). Output is pinned **byte-identical** by golden trees in `tests/golden/phase2/` — `UPDATE_GOLDENS=1` rewrites them, only do that deliberately. Surviving full-method overrides are domain-shaped and commented per scraper.\n\n### UnifiedScraper (multi-source configs)\n\n`unified_scraper.py` dispatches via the class-level `SOURCE_DISPATCH` table; `_scrape_with_converter()` is the shared engine for the 13 mechanical source types (`get_converter()` + public `converter.extract()` + cache copy + sub-skill build), so **new types registered in `CONVERTER_REGISTRY` work in unified configs automatically**. documentation/github/local stay bespoke (commented why). `run()` deliberately does NOT follow the base template (TestRunOrchestration pins that run() triggers workflows).\n\n### Data Flow (5 phases)\n\n1. **Scrape** - Source-specific scraper extracts content to `output/{name}_data/pages/*.json`\n2. **Build** - `build_skill()` categorizes pages, extracts patterns, generates `output/{name}/SKILL.md`\n3. **Enhance** (optional) - LLM rewrites SKILL.md (`--enhance-level 0-3`, auto-detects API vs LOCAL mode)\n4. **Package** - Platform adaptor formats output (`.zip`, `.tar.gz`, JSON, vector index)\n5. **Upload** (optional) - Platform API upload\n\n### Platform Adaptor Pattern (Strategy + Factory)\n\nFactory: `get_adaptor(platform, config)` in `adaptors/__init__.py` returns a `SkillAdaptor` instance. Base class `SkillAdaptor` + `SkillMetadata` in `adaptors/base.py`.\n\n```\nsrc/skill_seekers/cli/adaptors/\n├── __init__.py              # Factory: get_adaptor(platform, config), ADAPTORS registry\n├── base.py                  # Abstract base: SkillAdaptor, SkillMetadata\n├── openai_compatible.py     # Shared base for OpenAI-compatible platforms\n├── claude.py                # --target claude\n├── gemini.py                # --target gemini\n├── openai.py                # --target openai\n├── markdown.py              # --target markdown\n├── minimax.py               # --target minimax\n├── opencode.py              # --target opencode\n├── kimi.py                  # --target kimi\n├── deepseek.py              # --target deepseek\n├── qwen.py                  # --target qwen\n├── openrouter.py            # --target openrouter\n├── together.py              # --target together\n├── fireworks.py             # --target fireworks\n├── langchain.py             # --target langchain\n├── llama_index.py           # --target llama-index\n├── haystack.py              # --target haystack\n├── chroma.py                # --target chroma\n├── faiss_helpers.py         # --target faiss\n├── qdrant.py                # --target qdrant\n├── weaviate.py              # --target weaviate\n├── pinecone_adaptor.py      # --target pinecone\n└── streaming_adaptor.py     # --target streaming\n```\n\nAll adaptors use `--target`. All adaptors are imported with `try/except ImportError` so missing optional deps don't break the registry.\n\n### 18 Source Type Converters\n\nEach in `src/skill_seekers/cli/{type}_scraper.py` as a `SkillConverter` subclass (no `main()`). The `create_command.py` uses `source_detector.py` to auto-detect, then calls `get_converter()`. Converters: web (doc_scraper), github, pdf, word, epub, video, local (codebase_scraper), jupyter, html, openapi, asciidoc, pptx, rss, manpage, confluence, notion, chat, config (unified_scraper).\n\n### CLI Argument System (single-definition parsers)\n\n```\nsrc/skill_seekers/cli/\n├── parsers/              # Central SubcommandParser classes — the ONLY definition of each command's flags\n│   └── create_parser.py  # Progressive help disclosure (--help-web, --help-github, etc.)\n├── arguments/            # Argument definitions\n│   ├── common.py         # add_all_standard_arguments() - shared across all scrapers\n│   └── create.py         # UNIVERSAL_ARGUMENTS, WEB_ARGUMENTS, GITHUB_ARGUMENTS, etc.\n├── exit_codes.py         # EXIT_SUCCESS/ERROR/VALIDATION/INTERRUPT\n└── source_detector.py    # Auto-detect source type from input string\n```\n\nCommand modules' standalone `main(args=None)` paths build their parser FROM the central `SubcommandParser` class — **add/change a flag in `parsers/*.py` only**. Drift guards (`tests/test_cli_parsers.py::TestCentralModuleParserSync` and `TestCentralParserSingleSource`) fail CI on any divergence of dests/defaults/option strings.\n\n`ExecutionContext.override()` is context-local (a `ContextVar` layered over the unchanged base singleton) — thread/async safe for the MCP server; propagate to worker threads via `copy_context`.\n\n### Standalone subsystems (outside `cli/`)\n\nFour top-level packages sit beside `cli/` and are largely independent of the scrape→build→package flow:\n\n- `embedding/` — FastAPI embedding-generation server (`python -m skill_seekers.embedding.server`) with a caching layer (`cache.py`) and multi-backend generators (OpenAI, sentence-transformers, Anthropic). Feeds the vector-DB adaptors.\n- `sync/` — real-time doc-sync system: `detector.py` (content-hash / last-modified change detection), `monitor.py` (scheduled incremental re-scrapes), `notifier.py` (email/Slack/webhook). Keeps generated skills fresh as upstream docs change.\n- `benchmark/` — performance suite (`runner.py`, `framework.py`) measuring scrape/embedding/storage/e2e timing, memory, and CPU; emits comparison + optimization reports.\n- `workflows/` — bundled default enhancement-workflow presets consumed by the enhancement step.\n\n### C3.x Codebase Analysis Pipeline\n\nLocal codebase analysis features, all opt-out (`--skip-*` flags):\n- C3.1 `pattern_recognizer.py` - Design pattern detection (10 GoF patterns, 9 languages)\n- C3.2 `test_example_extractor.py` - Usage examples from tests\n- C3.3 `how_to_guide_builder.py` - AI-enhanced educational guides\n- C3.4 `config_extractor.py` - Configuration pattern extraction\n- C3.5 `generate_router.py` - Architecture overview generation\n- C3.10 `signal_flow_analyzer.py` - Godot signal flow analysis\n\n### MCP Server\n\n`src/skill_seekers/mcp/server_fastmcp.py` - 40 tools via FastMCP. Transport: stdio (Claude Code) or HTTP (Cursor/Windsurf). Optional dependency: `pip install -e \".[mcp]\"`\n\n- **Tools run in-process** via `run_cli_main()` in `mcp/tools/_common.py`: same argv parsed by the command's REAL parser (sys.argv patch under a lock), stdout/stderr capture + contextvar log capture, identical `(stdout, stderr, returncode)` contract. No subprocess startup; old hard timeouts are advisory.\n- **Exceptions BY DESIGN**: `enhance_skill` (LOCAL agent) and `install_skill`'s enhancement step stay subprocess — the agent must be a real child process for the fork-bomb-guard env semantics (`SKILL_SEEKER_ENHANCE_ACTIVE`). Never make these in-process.\n- **Domain logic lives in `skill_seekers.services/`** (marketplace_manager, marketplace_publisher, config_publisher, source_manager, git_repo) — importable by CLI without the `[mcp]` extra; old `skill_seekers.mcp.*` paths are back-compat shims. No `sys.path` hacks anywhere in `mcp/`.\n\n### Enhancement (AgentClient is the single AI transport)\n\nEvery AI call goes through `AgentClient` (`src/skill_seekers/cli/agent_client.py`): central truncation gate, timeout policy, error classification. `API_PROVIDERS` (provider registry) and `AGENT_PRESETS` (local-agent command templates) live ONLY there. Each `API_PROVIDERS` entry declares its wire `protocol` (`anthropic`/`openai`/`google`) and `supports_images` capability — `_call_api` branches on the resolved protocol, NOT the provider name, so an OpenAI/Anthropic-compatible provider needs no new branch. Adaptors declare provider/endpoint/model/prompt and route through `SkillAdaptor._enhance_skill_md_via_client` (atomic save with backup). Multimodal image input goes through `AgentClient.call_with_image()` (used by `video_visual` frame OCR across all image-capable providers); it no longer bypasses AgentClient with a direct SDK call.\n\n- **API mode** (if API key set): Anthropic, Google Gemini, OpenAI, Moonshot/Kimi, MiniMax — detected in registry order; `SKILL_SEEKER_PROVIDER` forces one. Models: `SKILL_SEEKER_MODEL` (global) or `ANTHROPIC_MODEL`/`GOOGLE_MODEL`/`OPENAI_MODEL`/`MOONSHOT_MODEL`/`MINIMAX_MODEL`; `ANTHROPIC_BASE_URL` for compatible endpoints. MiniMax adds `MINIMAX_API_REGION` (`global_en`/`cn_zh`) and `MINIMAX_API_PROTOCOL` (`openai`/`anthropic`). Vision OCR provider: `SKILL_SEEKER_VISION_PROVIDER` (`auto` picks the first image-capable provider with a key).\n- **LOCAL mode** (fallback): Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents — command built by `build_local_agent_command()`.\n- Control: `--enhance-level 0` (off) / `1` (SKILL.md only) / `2` (default, balanced) / `3` (full)\n- Agent selection: `--agent claude|codex|copilot|opencode|kimi|custom`\n\n## Key Implementation Details\n\n### Smart Categorization (`doc_scraper.py:smart_categorize()`)\n\nScores pages against category keywords: 3 points for URL match, 2 for title, 1 for content. Threshold of 2+ required. Falls back to \"other\".\n\n### Content Extraction (`doc_scraper.py`)\n\n`FALLBACK_MAIN_SELECTORS` constant + `_find_main_content()` helper handle CSS selector fallback. Links are extracted from the full page before early return (not just main content). `body` is deliberately excluded from fallbacks.\n\n### Three-Stream GitHub Architecture (`unified_codebase_analyzer.py`)\n\nStream 1: Code Analysis (AST, patterns, tests, guides). Stream 2: Documentation (README, docs/, wiki). Stream 3: Community (issues, PRs, metadata). Depth control: `basic` (1-2 min) or `c3x` (20-60 min).\n\n## Testing\n\n### Test markers (pytest.ini)\n\n```bash\npytest tests/ -v                                    # Default: fast tests only\npytest tests/ -v -m slow                            # Include slow tests (>5s)\npytest tests/ -v -m integration                     # External services required\npytest tests/ -v -m e2e                             # Resource-intensive\npytest tests/ -v -m \"not slow and not integration\"  # Fastest subset\n```\n\n### Known legitimate skips (~11)\n\n- 2: chromadb incompatible with Python 3.14 (pydantic v1)\n- 2: weaviate-client not installed\n- 2: Qdrant not running (requires docker)\n- 2: langchain/llama_index not installed\n- 3: GITHUB_TOKEN not set\n\n### sys.modules gotcha\n\n`test_swift_detection.py` deletes `skill_seekers.cli` modules from `sys.modules`. It must save and restore both `sys.modules` entries AND parent package attributes (`setattr`). See the test file for the pattern.\n\n## Dependencies\n\nCore deps include `langchain`, `llama-index`, `anthropic`, `httpx`, `PyMuPDF`, `pydantic`. Platform-specific deps are optional:\n\n```bash\npip install -e \".[mcp]\"       # MCP server\npip install -e \".[gemini]\"    # Google Gemini\npip install -e \".[openai]\"    # OpenAI\npip install -e \".[docx]\"      # Word documents\npip install -e \".[epub]\"      # EPUB books\npip install -e \".[video]\"     # Video (lightweight)\npip install -e \".[video-full]\"# Video (Whisper + visual)\npip install -e \".[jupyter]\"   # Jupyter notebooks\npip install -e \".[pptx]\"      # PowerPoint\npip install -e \".[rss]\"       # RSS/Atom feeds\npip install -e \".[confluence]\"# Confluence wiki\npip install -e \".[notion]\"    # Notion pages\npip install -e \".[chroma]\"    # ChromaDB\npip install -e \".[all]\"       # Everything (except video-full)\n```\n\nDev dependencies use PEP 735 `[dependency-groups]` in pyproject.toml.\n\n## Environment Variables\n\n```bash\nANTHROPIC_API_KEY=sk-ant-...          # Claude AI (or compatible endpoint)\nANTHROPIC_BASE_URL=https://...        # Optional: Claude-compatible API endpoint\nGOOGLE_API_KEY=AIza...                # Google Gemini (optional)\nOPENAI_API_KEY=sk-...                 # OpenAI (optional)\nGITHUB_TOKEN=ghp_...                  # Higher GitHub rate limits\n```\n\n## Adding New Features\n\n### New platform adaptor\n1. Create `src/skill_seekers/cli/adaptors/{platform}.py` inheriting `SkillAdaptor` from `base.py`\n2. Register in `adaptors/__init__.py` (add try/except import + add to `ADAPTORS` dict)\n3. Add optional dep to `pyproject.toml`\n4. Add tests in `tests/`\n\n### New source type converter\n1. Create `src/skill_seekers/cli/{type}_scraper.py` — for document-shaped sources inherit `DocumentSkillBuilder` (categorization/references/index/SKILL.md come free; implement `extract()` + hooks), otherwise inherit `SkillConverter` and implement `extract()` and `build_skill()`. Set `SOURCE_TYPE`.\n2. Register in `CONVERTER_REGISTRY` in `skill_converter.py` — this also makes the type work in unified configs automatically (UnifiedScraper engine)\n3. Add source type config building in `create_command.py:_build_config()`\n4. Add auto-detection in `source_detector.py`\n5. Add optional dep if needed\n6. Add tests\n\n### New CLI argument\n- Subcommand flag: define ONLY in the central parser class (`parsers/{cmd}_parser.py`) — module `main()` builds from it; the drift-guard test fails otherwise\n- Universal: `UNIVERSAL_ARGUMENTS` in `arguments/create.py`\n- Source-specific: appropriate dict (`WEB_ARGUMENTS`, `GITHUB_ARGUMENTS`, etc.)\n- Shared across scrapers: `add_all_standard_arguments()` in `arguments/common.py`\n","AGENTS.md":"# AGENTS.md - Skill Seekers\n\nComprehensive reference for AI coding agents. Skill Seekers is a Python CLI tool (v3.6.0) that converts documentation sites, GitHub repos, PDFs, videos, notebooks, wikis, and more into AI-ready skills for 21+ LLM platforms and RAG pipelines.\n\n## Project Overview\n\n**Skill Seekers** is a universal preprocessing layer that transforms raw documentation and code into structured knowledge assets. It supports 17+ source types and exports to 21+ AI platforms including Claude, Gemini, OpenAI, LangChain, LlamaIndex, and various vector databases.\n\n### Key Capabilities\n- **Source Types (17):** Documentation websites, GitHub repos, PDFs, Word docs, EPUBs, videos, local codebases, Jupyter notebooks, HTML, OpenAPI specs, AsciiDoc, PowerPoint, Confluence, Notion, RSS feeds, man pages, chat exports\n- **Export Targets (21):** Claude, Gemini, OpenAI, MiniMax, OpenCode, Kimi, DeepSeek, Qwen, OpenRouter, Together AI, Fireworks AI, Markdown, LangChain, LlamaIndex, Haystack, Weaviate, ChromaDB, FAISS, Qdrant, Pinecone\n- **MCP Server:** FastMCP-based Model Context Protocol server for AI assistant integration\n\n## Setup\n\n```bash\n# REQUIRED before running tests (src/ layout — tests hard-exit if package not installed)\npip install -e .\n\n# With dev tools (pytest, ruff, mypy, coverage)\npip install -e \".[dev]\"\n\n# With specific LLM platform support\npip install -e \".[gemini]\"      # Google Gemini\npip install -e \".[openai]\"      # OpenAI ChatGPT\npip install -e \".[all-llms]\"    # All LLM platforms\n\n# With all optional dependencies (except video-full)\npip install -e \".[all]\"\n\n# Full video processing (heavy dependencies)\npip install -e \".[video-full]\"\n```\n\nNote: `tests/conftest.py` checks that `skill_seekers` is importable and calls `sys.exit(1)` if not. Always install in editable mode first.\n\n### Environment Variables\n\nCreate a `.env` file or export these variables:\n```bash\nANTHROPIC_API_KEY      # For Claude AI enhancement\nGOOGLE_API_KEY         # For Gemini support\nOPENAI_API_KEY         # For OpenAI support\nGITHUB_TOKEN           # For GitHub repo scraping (higher rate limits)\n```\n\n## Build / Test / Lint\n\n```bash\n# Full suite (never skip — all must pass)\npytest tests/ -v\n\n# Fast iteration (skip slow, integration, E2E, network, MCP)\npytest tests/ -m \"not slow and not integration and not e2e and not network and not serial and not mcp_only\" -q\n\n# Fast parallel (install pytest-xdist first)\npytest tests/ -n auto --dist=loadfile -m \"not slow and not integration and not e2e and not network and not serial and not mcp_only\" -q\n\n# 3-phase runner script (recommended for local dev)\nbash scripts/run_tests_fast.sh\n\n# Single test\npytest tests/test_scraper_features.py::test_detect_language -v\n\n# Skip slow/integration\npytest tests/ -v -m \"not slow and not integration\"\n\n# With coverage\npytest tests/ --cov=src/skill_seekers --cov-report=term\n\n# Lint + format check (matches CI)\nruff check src/ tests/\nruff format --check src/ tests/\n\n# Type check (non-blocking — mypy is continue-on-error in CI)\nmypy src/skill_seekers --show-error-codes --pretty\n```\n\n**Pytest config:** `asyncio_mode = \"auto\"`, so `@pytest.mark.asyncio` is implicit. Test markers: `slow`, `integration`, `e2e`, `venv`, `bootstrap`, `benchmark`, `asyncio`, `serial`, `network`, `mcp_only`.\n\n**CI note:** CI pins `ruff==0.15.8` (not the `>=0.14.13` dev dep). If formatting behaves differently locally, check the CI version.\n\n**CI test phases:** Tests are split into 3 parallel jobs:\n- `test-fast` — 3386 unit tests with xdist across OS/Python matrix\n- `test-serial` — 69 serial/integration/E2E/network tests\n- `test-mcp` — 193 MCP tests (requires `[mcp]` extras)\n\n## Code Style\n\n### Formatting Rules (ruff — from pyproject.toml)\n- **Line length:** 100 characters\n- **Target Python:** 3.10+\n- **Enabled lint rules:** E, W, F, I, B, C4, UP, ARG, SIM\n- **Ignored rules:** E501 (line length handled by formatter), F541 (f-string style), ARG002 (unused method args for interface compliance), B007 (intentional unused loop vars), I001 (formatter handles imports), SIM114 (readability preference)\n\n### Imports\n- Sort with isort (via ruff); `skill_seekers` is first-party\n- Standard library → third-party → first-party, separated by blank lines\n- Use `from __future__ import annotations` only if needed for forward refs\n- Guard optional imports with try/except ImportError (see `adaptors/__init__.py` pattern):\n  ```python\n  try:\n      from .claude import ClaudeAdaptor\n      from .minimax import MiniMaxAdaptor\n  except ImportError:\n      ClaudeAdaptor = None\n      MiniMaxAdaptor = None\n  ```\n\n### Naming Conventions\n- **Files:** `snake_case.py` (e.g., `source_detector.py`, `config_validator.py`)\n- **Classes:** `PascalCase` (e.g., `SkillAdaptor`, `ClaudeAdaptor`, `SourceDetector`)\n- **Functions/methods:** `snake_case` (e.g., `get_adaptor()`, `detect_language()`)\n- **Constants:** `UPPER_CASE` (e.g., `ADAPTORS`, `DEFAULT_CHUNK_TOKENS`, `VALID_SOURCE_TYPES`)\n- **Private:** prefix with `_` (e.g., `_read_existing_content()`, `_validate_unified()`)\n\n### Type Hints\n- Gradual typing — add hints where practical, not enforced everywhere\n- Use modern syntax: `str | None` not `Optional[str]`, `list[str]` not `List[str]`\n- MyPy config: `disallow_untyped_defs = false`, `check_untyped_defs = true`, `ignore_missing_imports = true`\n- Tests are excluded from strict type checking (`disallow_untyped_defs = false`, `check_untyped_defs = false` for `tests.*`)\n\n### Docstrings\n- Module-level docstring on every file (triple-quoted, describes purpose)\n- Google-style docstrings for public functions/classes\n- Include `Args:`, `Returns:`, `Raises:` sections where useful\n\n### Error Handling\n- Use specific exceptions, never bare `except:`\n- Provide helpful error messages with context\n- Use `raise ValueError(...)` for invalid arguments, `raise RuntimeError(...)` for state errors\n- Guard optional dependency imports with try/except and give clear install instructions on failure\n- Chain exceptions with `raise ... from e` when wrapping\n\n### Suppressing Lint Warnings\n- Use inline `# noqa: XXXX` comments (e.g., `# noqa: F401` for re-exports, `# noqa: ARG001` for required but unused params)\n\n## Project Layout\n\n```\nsrc/skill_seekers/           # Main package (src/ layout)\n  cli/                       # CLI commands and entry points (100+ files)\n    adaptors/                # Platform adaptors (Strategy pattern, inherit SkillAdaptor)\n    arguments/               # CLI argument definitions (one per source type)\n    parsers/                 # Subcommand parsers (one per source type)\n    storage/                 # Cloud storage (inherit BaseStorageAdaptor)\n    main.py                  # Unified CLI entry point (COMMAND_MODULES dict)\n    source_detector.py       # Auto-detects source type from user input\n    create_command.py        # Unified `create` command routing\n    config_validator.py      # VALID_SOURCE_TYPES set + per-type validation\n    unified_scraper.py       # Multi-source orchestrator (scraped_data + dispatch)\n    unified_skill_builder.py # Pairwise synthesis + generic merge\n  mcp/                       # MCP server (FastMCP + legacy)\n    tools/                   # MCP tool implementations by category (10 files)\n    server_fastmcp.py        # FastMCP server implementation\n    server_legacy.py         # Legacy MCP server\n  sync/                      # Sync monitoring (Pydantic models)\n  benchmark/                 # Benchmarking framework\n  embedding/                 # FastAPI embedding server\n  workflows/                 # 67 YAML workflow presets\n  _version.py                # Reads version from pyproject.toml\ntests/                       # 160 test files (pytest)\n  test_adaptors/             # 22 adaptor-specific test files\n  conftest.py                # Test configuration with package check\nconfigs/                     # Preset JSON scraping configs\ndocs/                        # Documentation (guides, integrations, architecture)\n```\n\n## Key Patterns\n\n**Adaptor (Strategy) pattern** — all platform logic in `cli/adaptors/`. Inherit `SkillAdaptor`, implement `format_skill_md()`, `package()`, `upload()`. Register in `adaptors/__init__.py` ADAPTORS dict.\n\n**Scraper pattern** — each source type has: `cli/<type>_scraper.py` (with `<Type>ToSkillConverter` class + `main()`), `arguments/<type>.py`, `parsers/<type>_parser.py`. Register in `parsers/__init__.py` PARSERS list, `main.py` COMMAND_MODULES dict, `config_validator.py` VALID_SOURCE_TYPES set.\n\n**Unified pipeline** — `unified_scraper.py` dispatches to per-type `_scrape_<type>()` methods. `unified_skill_builder.py` uses pairwise synthesis for docs+github+pdf combos and `_generic_merge()` for all other combinations.\n\n**MCP tools** — grouped in `mcp/tools/` by category. `scrape_generic_tool` handles all new source types.\n\n**CLI subcommands** — git-style in `cli/main.py`. Each delegates to a module's `main()` function.\n\n**Supported source types (17):** documentation (web), github, pdf, local, word, video, epub, jupyter, html, openapi, asciidoc, pptx, confluence, notion, rss, manpage, chat. Each detected automatically by `source_detector.py`.\n\n**Supported platforms (21):** claude, gemini, openai, minimax, opencode, kimi, deepseek, qwen, openrouter, together, fireworks, markdown, langchain, llama-index, haystack, weaviate, chroma, faiss, qdrant, pinecone.\n\n## CLI Commands\n\n```bash\n# Core commands\nskill-seekers create <source>              # Create skill from any source (auto-detects type)\nskill-seekers scan <dir>                   # AI-detect a project's tech stack and emit per-framework configs\nskill-seekers enhance <directory>          # AI-powered enhancement\nskill-seekers package <directory>          # Package skill for target platform\nskill-seekers upload <file>                # Upload skill to target platform\nskill-seekers install <source>             # One-command workflow (scrape + enhance + package + upload)\n\n# Utilities\nskill-seekers estimate <source>            # Estimate page count before scraping\nskill-seekers doctor                       # Health check for dependencies\nskill-seekers config                       # Configure API keys and settings\nskill-seekers workflows                    # List and apply workflow presets\nskill-seekers resume <job_id>              # Resume interrupted scraping\n\n# Advanced\nskill-seekers stream <source>              # Streaming ingestion\nskill-seekers update <directory>           # Incremental update\nskill-seekers multilang <directory>        # Multi-language support\n```\n\n## Testing Instructions\n\n### Test Structure\n- Unit tests: `tests/test_*.py` — test individual modules\n- Adaptor tests: `tests/test_adaptors/test_*_adaptor.py` — test platform adaptors\n- E2E tests: `tests/test_*_e2e.py` — end-to-end integration tests\n\n### Running Tests\n```bash\n# Fast test run (skip slow/integration tests)\npytest tests/ -v -m \"not slow and not integration\"\n\n# Full test suite\npytest tests/ -v\n\n# With coverage report\npytest tests/ --cov=src/skill_seekers --cov-report=term-missing\n\n# Specific test categories\npytest tests/ -v -m \"slow\"           # Only slow tests\npytest tests/ -v -m \"integration\"    # Only integration tests\npytest tests/ -v -m \"e2e\"            # Only E2E tests\n```\n\n### Test Fixtures\nTest fixtures are located in `tests/fixtures/` and include sample configs, HTML files, and mock data.\n\n## Git Workflow\n\n- **`main`** — production, protected\n- **`development`** — default PR target, active dev\n- Feature branches created from `development`\n\n## Pre-commit Checklist\n\n```bash\nruff check src/ tests/\nruff format --check src/ tests/\npytest tests/ -v -x   # stop on first failure\n```\n\nNever commit API keys. Use env vars: `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`, `GITHUB_TOKEN`.\n\n## CI/CD\n\nGitHub Actions (7 workflows in `.github/workflows/`):\n- **tests.yml** — ruff + mypy lint job, then pytest matrix (Ubuntu + macOS, Python 3.10-3.12) with Codecov upload\n- **release.yml** — tag-triggered: tests → version verification → PyPI publish via `uv build`\n- **test-vector-dbs.yml** — tests vector DB adaptors (weaviate, chroma, faiss, qdrant)\n- **docker-publish.yml** — multi-platform Docker builds (amd64, arm64) for CLI + MCP images\n- **quality-metrics.yml** — quality analysis with configurable threshold\n- **scheduled-updates.yml** — weekly skill updates for popular frameworks\n- **vector-db-export.yml** — weekly vector DB exports\n\n## Deployment\n\n### Docker\nMulti-stage Dockerfile with Python 3.12 slim base:\n```bash\n# Build CLI image\ndocker build -t skill-seekers:local -f Dockerfile .\n\n# Run CLI\ndocker run -v $(pwd)/output:/output skill-seekers:local create https://docs.example.com\n\n# Run MCP server\ndocker build -t skill-seekers-mcp:local -f Dockerfile.mcp .\ndocker run -p 8765:8765 skill-seekers-mcp:local\n```\n\n### MCP Server\nThe MCP server provides Model Context Protocol integration:\n```bash\n# Start FastMCP server\nskill-seekers-mcp\n\n# Or use the Python module\npython -m skill_seekers.mcp.server_fastmcp\n```\n\n## Security Considerations\n\n- **API Keys:** Never commit API keys to version control. Use environment variables or `.env` files (already in `.gitignore`)\n- **Docker:** Runs as non-root user (`skillseeker`, UID 1000)\n- **Dependencies:** Regular security updates via `pip audit` or `safety check`\n- **Sandboxing:** Video processing uses optional dependencies that can be heavy; install `[video-full]` only when needed\n\n## Additional Resources\n\n- **Website:** https://skillseekersweb.com/\n- **Documentation:** https://skillseekersweb.com/\n- **PyPI:** https://pypi.org/project/skill-seekers/\n- **Repository:** https://github.com/yusufkaraaslan/Skill_Seekers\n- **Config Browser:** https://skillseekersweb.com/\n- **Project Board:** https://github.com/users/yusufkaraaslan/projects/2\n"},"files":{"CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\n**Skill Seekers** converts documentation from 18 source types into production-ready formats for 21+ AI platforms (LLM platforms, RAG frameworks, vector databases, AI coding assistants). Published on PyPI as `skill-seekers`.\n\n**Version:** 3.9.0 (dev; last release 3.8.0) — source of truth is `src/skill_seekers/_version.py` | **Python:** 3.10+ | **Website:** https://skillseekersweb.com/\n\n**Architecture:** See `docs/UML_ARCHITECTURE.md` for UML diagrams and module overview. StarUML project at `docs/UML/skill_seekers.mdj`. Refactor state/history: `docs/UNIFICATION_PLAN.md` (Grand Unification — all 5 phases done; remaining cosmetic items listed there).\n\n## Essential Commands\n\n```bash\n# REQUIRED before running tests or CLI (src/ layout)\npip install -e .\n\n# Run all tests (NEVER skip - all must pass before commits)\npytest tests/ -v\n\n# Fast iteration (skip slow MCP tests ~20min)\npytest tests/ --ignore=tests/test_mcp_fastmcp.py --ignore=tests/test_mcp_server.py --ignore=tests/test_install_skill_e2e.py -q\n\n# Single test\npytest tests/test_scraper_features.py::test_detect_language -vv -s\n\n# Code quality (must pass before push - matches CI)\nuvx ruff check src/ tests/\nuvx ruff format --check src/ tests/\nmypy src/skill_seekers  # continue-on-error in CI\n\n# Auto-fix lint/format issues\nuvx ruff check --fix --unsafe-fixes src/ tests/\nuvx ruff format src/ tests/\n\n# Build & publish\nuv build\nuv publish\n```\n\n## CI Matrix\n\nRuns on push/PR to `main` or `development`. Lint job (Python 3.12, Ubuntu) + Test job (Ubuntu + macOS, Python 3.10/3.11/3.12, excludes macOS+3.10). Both must pass for merge.\n\n## Git Workflow\n\n- **Main branch:** `main` (requires tests + 1 review)\n- **Development branch:** `development` (default PR target, requires tests)\n- **Feature branches:** `feature/{task-id}-{description}` from `development`\n- PRs always target `development`, never `main` directly\n\n## Architecture\n\n### CLI: Unified create command\n\nEntry point `src/skill_seekers/cli/main.py`. The `create` command is the **primary** entry point for skill creation — it auto-detects source type and routes to the appropriate `SkillConverter`. The `scan` command (added in #327) is a separate discovery step for projects with multiple frameworks; it emits one config file per detected framework and you then run `create` on each.\n\n```\nskill-seekers create <source>     # Auto-detect: URL, owner/repo, ./path, file.pdf, etc.\nskill-seekers scan <dir>          # AI-driven discovery → emits one config per detected framework + <project>-codebase.json\nskill-seekers package <dir>       # Package for platform (--target claude/gemini/openai/markdown/minimax/opencode/kimi/deepseek/qwen/openrouter/together/fireworks/atlas/langchain/llama-index/haystack/chroma/faiss/weaviate/qdrant/pinecone/ibm-bob)\n```\n\n### Scan command (issue #327)\n\n`skill-seekers scan <dir>` is an AI-driven project knowledge-base bootstrapper. Pipeline in `src/skill_seekers/cli/scan_command.py`:\n\n1. `collect_signals()` in `signal_collectors.py` — deterministic, bounded gathering of manifests + README + Dockerfile/CI + sampled source files + git remote. **Per-kind byte budgets** (24 KB manifest / 6 KB README / 6 KB CI / 28 KB samples, total 64 KB) so a fat package.json can't crowd out other kinds. `_SOURCE_DIRS` covers ~14 layouts (Go `cmd/`, Rust `crates/`, JS monorepo `apps/packages/`, Maven `source/`, Django at root); also walks root one level deep for flat-layout Python.\n2. `detect_with_ai(bundle, AgentClient)` — one LLM call, structured JSON output. **Source signals are first-2-KB of each file** (whole-file sampling, no regex parsing — added in WS4 because regex missed Go multi-line imports + Rust `mod`/`extern crate`). Canonical-slug prompt + the canonical-name resolver are coupled — change one, update the other.\n3. `resolve_or_generate_with_status()` — for each detection: try `out_dir/<slug>.json` (cache from prior run), then `resolve_config_path` from `config_fetcher` with multiple canonical name candidates (`_canonical_name_candidates` handles `\"Godot Engine\"` → `\"godot\"`, plus CJK / European suffixes like `\"Godot 引擎\"`, `\"React フレームワーク\"`, `\"Lodash Bibliothek\"`), then `generate_config_with_ai` as the last resort. Always appends `.json` to lookup names so local-disk and user-dir resolution actually finds files. Always stamps `metadata.detected_version` (nested, not top-level — `metadata.version` already exists and means config-schema version).\n4. `emit_codebase_config()` — always writes `<project>-codebase.json` (a `type: local` source pointed at the project root).\n5. `diff_against_existing()` — keyed by **filename slug** (not internal `data[\"name\"]`) so re-scans don't churn when the AI returns a display name vs the registry canonical slug.\n6. `_archive_removed()` — when a config disappears from detections, MOVE (not delete — user may have hand-edited) to `out_dir/.archived/<UTC-timestamp>/`. Runs after diff, before fresh writes.\n7. `maybe_publish()` — **native async** (WS11). Opt-in submission of freshly AI-generated configs to the community registry. Pre-checks `GITHUB_TOKEN`. Idempotency guard: `_find_existing_issue` queries GitHub Search API for an existing open issue with the same config name before submitting. Retries transient failures (rate limit, 5xx) with 0s/5s/15s backoff. `_prompt_async` wraps `input()` via `asyncio.to_thread` so the event loop isn't blocked.\n\n**CLI dispatch** uses the `COMMAND_CLASSES` table in `main.py` (added in WS1). `scan` and `doctor` are dispatched as `Cls(args).execute()` consuming the parsed argparse namespace directly — no `_reconstruct_argv` hack, no duplicate argparse. `ScanCommand.execute()` is the single `asyncio.run` boundary wrapping `run_scan` (sync) + `maybe_publish` (async). Remaining ~14 commands still use the legacy `COMMAND_MODULES` dispatch; they're flagged for migration.\n\n**Cost guardrails**: `--max-ai-generations N` (default 10) caps unbounded AI generation; `--dry-run` previews without writing or invoking AI; `--probe-urls` HEAD-checks AI-generated URLs with retry-on-404 and stamps `metadata._url_unverified` on confirmed-bad URLs.\n\n**Safety**: All writes use `_atomic_write_json` (`os.replace` after writing to `.tmp`) so a `KeyboardInterrupt` mid-write can't corrupt configs. `_safe_size` guards `stat()` so broken symlinks don't crash the scan. `ScanCommand.execute` calls `logging.basicConfig` so `logger.warning`/`error` is visible; exit code is non-zero when no configs and no codebase config were emitted.\n\n**Public constant**: `SourceDetector.CODE_PROJECT_MARKERS` (was `_CODE_PROJECT_MARKERS`) — shared between source_detector + signal_collectors. ~50 manifest types now (Pipfile, environment.yml, deno.json, flake.nix, Chart.yaml, deps.edn, dune-project, BUILD.bazel, …). Public so cross-module access doesn't reach into a private attribute.\n\n### SkillConverter Pattern (Template Method + Factory)\n\nAll 18 source types implement the `SkillConverter` base class (`skill_converter.py`):\n\n```python\nconverter = get_converter(\"web\", config)  # Factory lookup\nconverter.run()  # Template: extract() → build_skill()\n```\n\nRegistry in `CONVERTER_REGISTRY` maps source type → (module, class). `create_command.py` builds config from `ExecutionContext`, calls `get_converter()`, then runs centralized enhancement. `get_converter(\"config\", {...})` constructs `UnifiedScraper` from the same factory-shaped dict (no special cases in create_command/MCP). The base resolves `skill_dir` once (strips trailing separators) and derives `data_file` via `data_file_for()` — subclasses must not re-derive paths.\n\n### DocumentSkillBuilder (build side of 9 document scrapers)\n\n`cli/document_skill_builder.py:DocumentSkillBuilder` sits between `SkillConverter` and the 9 document scrapers (epub, word, pptx, html, pdf, jupyter, man, rss, chat). It owns `categorize_content`, reference-file writing (tables, truncation, image guard), `index.md` + `SKILL.md` generation, and `load_extracted_data`. Variation points are class attrs (`DOC_NOUN`, `SOURCE_LABEL`, `LOAD_TOTAL_KEY`, `PATTERN_KEYWORDS`, `RANGE_LABEL`, …) and small hook methods (`category_stem`, `_write_reference_section`, `_write_skill_md_metadata`). Output is pinned **byte-identical** by golden trees in `tests/golden/phase2/` — `UPDATE_GOLDENS=1` rewrites them, only do that deliberately. Surviving full-method overrides are domain-shaped and commented per scraper.\n\n### UnifiedScraper (multi-source configs)\n\n`unified_scraper.py` dispatches via the class-level `SOURCE_DISPATCH` table; `_scrape_with_converter()` is the shared engine for the 13 mechanical source types (`get_converter()` + public `converter.extract()` + cache copy + sub-skill build), so **new types registered in `CONVERTER_REGISTRY` work in unified configs automatically**. documentation/github/local stay bespoke (commented why). `run()` deliberately does NOT follow the base template (TestRunOrchestration pins that run() triggers workflows).\n\n### Data Flow (5 phases)\n\n1. **Scrape** - Source-specific scraper extracts content to `output/{name}_data/pages/*.json`\n2. **Build** - `build_skill()` categorizes pages, extracts patterns, generates `output/{name}/SKILL.md`\n3. **Enhance** (optional) - LLM rewrites SKILL.md (`--enhance-level 0-3`, auto-detects API vs LOCAL mode)\n4. **Package** - Platform adaptor formats output (`.zip`, `.tar.gz`, JSON, vector index)\n5. **Upload** (optional) - Platform API upload\n\n### Platform Adaptor Pattern (Strategy + Factory)\n\nFactory: `get_adaptor(platform, config)` in `adaptors/__init__.py` returns a `SkillAdaptor` instance. Base class `SkillAdaptor` + `SkillMetadata` in `adaptors/base.py`.\n\n```\nsrc/skill_seekers/cli/adaptors/\n├── __init__.py              # Factory: get_adaptor(platform, config), ADAPTORS registry\n├── base.py                  # Abstract base: SkillAdaptor, SkillMetadata\n├── openai_compatible.py     # Shared base for OpenAI-compatible platforms\n├── claude.py                # --target claude\n├── gemini.py                # --target gemini\n├── openai.py                # --target openai\n├── markdown.py              # --target markdown\n├── minimax.py               # --target minimax\n├── opencode.py              # --target opencode\n├── kimi.py                  # --target kimi\n├── deepseek.py              # --target deepseek\n├── qwen.py                  # --target qwen\n├── openrouter.py            # --target openrouter\n├── together.py              # --target together\n├── fireworks.py             # --target fireworks\n├── langchain.py             # --target langchain\n├── llama_index.py           # --target llama-index\n├── haystack.py              # --target haystack\n├── chroma.py                # --target chroma\n├── faiss_helpers.py         # --target faiss\n├── qdrant.py                # --target qdrant\n├── weaviate.py              # --target weaviate\n├── pinecone_adaptor.py      # --target pinecone\n└── streaming_adaptor.py     # --target streaming\n```\n\nAll adaptors use `--target`. All adaptors are imported with `try/except ImportError` so missing optional deps don't break the registry.\n\n### 18 Source Type Converters\n\nEach in `src/skill_seekers/cli/{type}_scraper.py` as a `SkillConverter` subclass (no `main()`). The `create_command.py` uses `source_detector.py` to auto-detect, then calls `get_converter()`. Converters: web (doc_scraper), github, pdf, word, epub, video, local (codebase_scraper), jupyter, html, openapi, asciidoc, pptx, rss, manpage, confluence, notion, chat, config (unified_scraper).\n\n### CLI Argument System (single-definition parsers)\n\n```\nsrc/skill_seekers/cli/\n├── parsers/              # Central SubcommandParser classes — the ONLY definition of each command's flags\n│   └── create_parser.py  # Progressive help disclosure (--help-web, --help-github, etc.)\n├── arguments/            # Argument definitions\n│   ├── common.py         # add_all_standard_arguments() - shared across all scrapers\n│   └── create.py         # UNIVERSAL_ARGUMENTS, WEB_ARGUMENTS, GITHUB_ARGUMENTS, etc.\n├── exit_codes.py         # EXIT_SUCCESS/ERROR/VALIDATION/INTERRUPT\n└── source_detector.py    # Auto-detect source type from input string\n```\n\nCommand modules' standalone `main(args=None)` paths build their parser FROM the central `SubcommandParser` class — **add/change a flag in `parsers/*.py` only**. Drift guards (`tests/test_cli_parsers.py::TestCentralModuleParserSync` and `TestCentralParserSingleSource`) fail CI on any divergence of dests/defaults/option strings.\n\n`ExecutionContext.override()` is context-local (a `ContextVar` layered over the unchanged base singleton) — thread/async safe for the MCP server; propagate to worker threads via `copy_context`.\n\n### Standalone subsystems (outside `cli/`)\n\nFour top-level packages sit beside `cli/` and are largely independent of the scrape→build→package flow:\n\n- `embedding/` — FastAPI embedding-generation server (`python -m skill_seekers.embedding.server`) with a caching layer (`cache.py`) and multi-backend generators (OpenAI, sentence-transformers, Anthropic). Feeds the vector-DB adaptors.\n- `sync/` — real-time doc-sync system: `detector.py` (content-hash / last-modified change detection), `monitor.py` (scheduled incremental re-scrapes), `notifier.py` (email/Slack/webhook). Keeps generated skills fresh as upstream docs change.\n- `benchmark/` — performance suite (`runner.py`, `framework.py`) measuring scrape/embedding/storage/e2e timing, memory, and CPU; emits comparison + optimization reports.\n- `workflows/` — bundled default enhancement-workflow presets consumed by the enhancement step.\n\n### C3.x Codebase Analysis Pipeline\n\nLocal codebase analysis features, all opt-out (`--skip-*` flags):\n- C3.1 `pattern_recognizer.py` - Design pattern detection (10 GoF patterns, 9 languages)\n- C3.2 `test_example_extractor.py` - Usage examples from tests\n- C3.3 `how_to_guide_builder.py` - AI-enhanced educational guides\n- C3.4 `config_extractor.py` - Configuration pattern extraction\n- C3.5 `generate_router.py` - Architecture overview generation\n- C3.10 `signal_flow_analyzer.py` - Godot signal flow analysis\n\n### MCP Server\n\n`src/skill_seekers/mcp/server_fastmcp.py` - 40 tools via FastMCP. Transport: stdio (Claude Code) or HTTP (Cursor/Windsurf). Optional dependency: `pip install -e \".[mcp]\"`\n\n- **Tools run in-process** via `run_cli_main()` in `mcp/tools/_common.py`: same argv parsed by the command's REAL parser (sys.argv patch under a lock), stdout/stderr capture + contextvar log capture, identical `(stdout, stderr, returncode)` contract. No subprocess startup; old hard timeouts are advisory.\n- **Exceptions BY DESIGN**: `enhance_skill` (LOCAL agent) and `install_skill`'s enhancement step stay subprocess — the agent must be a real child process for the fork-bomb-guard env semantics (`SKILL_SEEKER_ENHANCE_ACTIVE`). Never make these in-process.\n- **Domain logic lives in `skill_seekers.services/`** (marketplace_manager, marketplace_publisher, config_publisher, source_manager, git_repo) — importable by CLI without the `[mcp]` extra; old `skill_seekers.mcp.*` paths are back-compat shims. No `sys.path` hacks anywhere in `mcp/`.\n\n### Enhancement (AgentClient is the single AI transport)\n\nEvery AI call goes through `AgentClient` (`src/skill_seekers/cli/agent_client.py`): central truncation gate, timeout policy, error classification. `API_PROVIDERS` (provider registry) and `AGENT_PRESETS` (local-agent command templates) live ONLY there. Each `API_PROVIDERS` entry declares its wire `protocol` (`anthropic`/`openai`/`google`) and `supports_images` capability — `_call_api` branches on the resolved protocol, NOT the provider name, so an OpenAI/Anthropic-compatible provider needs no new branch. Adaptors declare provider/endpoint/model/prompt and route through `SkillAdaptor._enhance_skill_md_via_client` (atomic save with backup). Multimodal image input goes through `AgentClient.call_with_image()` (used by `video_visual` frame OCR across all image-capable providers); it no longer bypasses AgentClient with a direct SDK call.\n\n- **API mode** (if API key set): Anthropic, Google Gemini, OpenAI, Moonshot/Kimi, MiniMax — detected in registry order; `SKILL_SEEKER_PROVIDER` forces one. Models: `SKILL_SEEKER_MODEL` (global) or `ANTHROPIC_MODEL`/`GOOGLE_MODEL`/`OPENAI_MODEL`/`MOONSHOT_MODEL`/`MINIMAX_MODEL`; `ANTHROPIC_BASE_URL` for compatible endpoints. MiniMax adds `MINIMAX_API_REGION` (`global_en`/`cn_zh`) and `MINIMAX_API_PROTOCOL` (`openai`/`anthropic`). Vision OCR provider: `SKILL_SEEKER_VISION_PROVIDER` (`auto` picks the first image-capable provider with a key).\n- **LOCAL mode** (fallback): Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents — command built by `build_local_agent_command()`.\n- Control: `--enhance-level 0` (off) / `1` (SKILL.md only) / `2` (default, balanced) / `3` (full)\n- Agent selection: `--agent claude|codex|copilot|opencode|kimi|custom`\n\n## Key Implementation Details\n\n### Smart Categorization (`doc_scraper.py:smart_categorize()`)\n\nScores pages against category keywords: 3 points for URL match, 2 for title, 1 for content. Threshold of 2+ required. Falls back to \"other\".\n\n### Content Extraction (`doc_scraper.py`)\n\n`FALLBACK_MAIN_SELECTORS` constant + `_find_main_content()` helper handle CSS selector fallback. Links are extracted from the full page before early return (not just main content). `body` is deliberately excluded from fallbacks.\n\n### Three-Stream GitHub Architecture (`unified_codebase_analyzer.py`)\n\nStream 1: Code Analysis (AST, patterns, tests, guides). Stream 2: Documentation (README, docs/, wiki). Stream 3: Community (issues, PRs, metadata). Depth control: `basic` (1-2 min) or `c3x` (20-60 min).\n\n## Testing\n\n### Test markers (pytest.ini)\n\n```bash\npytest tests/ -v                                    # Default: fast tests only\npytest tests/ -v -m slow                            # Include slow tests (>5s)\npytest tests/ -v -m integration                     # External services required\npytest tests/ -v -m e2e                             # Resource-intensive\npytest tests/ -v -m \"not slow and not integration\"  # Fastest subset\n```\n\n### Known legitimate skips (~11)\n\n- 2: chromadb incompatible with Python 3.14 (pydantic v1)\n- 2: weaviate-client not installed\n- 2: Qdrant not running (requires docker)\n- 2: langchain/llama_index not installed\n- 3: GITHUB_TOKEN not set\n\n### sys.modules gotcha\n\n`test_swift_detection.py` deletes `skill_seekers.cli` modules from `sys.modules`. It must save and restore both `sys.modules` entries AND parent package attributes (`setattr`). See the test file for the pattern.\n\n## Dependencies\n\nCore deps include `langchain`, `llama-index`, `anthropic`, `httpx`, `PyMuPDF`, `pydantic`. Platform-specific deps are optional:\n\n```bash\npip install -e \".[mcp]\"       # MCP server\npip install -e \".[gemini]\"    # Google Gemini\npip install -e \".[openai]\"    # OpenAI\npip install -e \".[docx]\"      # Word documents\npip install -e \".[epub]\"      # EPUB books\npip install -e \".[video]\"     # Video (lightweight)\npip install -e \".[video-full]\"# Video (Whisper + visual)\npip install -e \".[jupyter]\"   # Jupyter notebooks\npip install -e \".[pptx]\"      # PowerPoint\npip install -e \".[rss]\"       # RSS/Atom feeds\npip install -e \".[confluence]\"# Confluence wiki\npip install -e \".[notion]\"    # Notion pages\npip install -e \".[chroma]\"    # ChromaDB\npip install -e \".[all]\"       # Everything (except video-full)\n```\n\nDev dependencies use PEP 735 `[dependency-groups]` in pyproject.toml.\n\n## Environment Variables\n\n```bash\nANTHROPIC_API_KEY=sk-ant-...          # Claude AI (or compatible endpoint)\nANTHROPIC_BASE_URL=https://...        # Optional: Claude-compatible API endpoint\nGOOGLE_API_KEY=AIza...                # Google Gemini (optional)\nOPENAI_API_KEY=sk-...                 # OpenAI (optional)\nGITHUB_TOKEN=ghp_...                  # Higher GitHub rate limits\n```\n\n## Adding New Features\n\n### New platform adaptor\n1. Create `src/skill_seekers/cli/adaptors/{platform}.py` inheriting `SkillAdaptor` from `base.py`\n2. Register in `adaptors/__init__.py` (add try/except import + add to `ADAPTORS` dict)\n3. Add optional dep to `pyproject.toml`\n4. Add tests in `tests/`\n\n### New source type converter\n1. Create `src/skill_seekers/cli/{type}_scraper.py` — for document-shaped sources inherit `DocumentSkillBuilder` (categorization/references/index/SKILL.md come free; implement `extract()` + hooks), otherwise inherit `SkillConverter` and implement `extract()` and `build_skill()`. Set `SOURCE_TYPE`.\n2. Register in `CONVERTER_REGISTRY` in `skill_converter.py` — this also makes the type work in unified configs automatically (UnifiedScraper engine)\n3. Add source type config building in `create_command.py:_build_config()`\n4. Add auto-detection in `source_detector.py`\n5. Add optional dep if needed\n6. Add tests\n\n### New CLI argument\n- Subcommand flag: define ONLY in the central parser class (`parsers/{cmd}_parser.py`) — module `main()` builds from it; the drift-guard test fails otherwise\n- Universal: `UNIVERSAL_ARGUMENTS` in `arguments/create.py`\n- Source-specific: appropriate dict (`WEB_ARGUMENTS`, `GITHUB_ARGUMENTS`, etc.)\n- Shared across scrapers: `add_all_standard_arguments()` in `arguments/common.py`\n","AGENTS.md":"# AGENTS.md - Skill Seekers\n\nComprehensive reference for AI coding agents. Skill Seekers is a Python CLI tool (v3.6.0) that converts documentation sites, GitHub repos, PDFs, videos, notebooks, wikis, and more into AI-ready skills for 21+ LLM platforms and RAG pipelines.\n\n## Project Overview\n\n**Skill Seekers** is a universal preprocessing layer that transforms raw documentation and code into structured knowledge assets. It supports 17+ source types and exports to 21+ AI platforms including Claude, Gemini, OpenAI, LangChain, LlamaIndex, and various vector databases.\n\n### Key Capabilities\n- **Source Types (17):** Documentation websites, GitHub repos, PDFs, Word docs, EPUBs, videos, local codebases, Jupyter notebooks, HTML, OpenAPI specs, AsciiDoc, PowerPoint, Confluence, Notion, RSS feeds, man pages, chat exports\n- **Export Targets (21):** Claude, Gemini, OpenAI, MiniMax, OpenCode, Kimi, DeepSeek, Qwen, OpenRouter, Together AI, Fireworks AI, Markdown, LangChain, LlamaIndex, Haystack, Weaviate, ChromaDB, FAISS, Qdrant, Pinecone\n- **MCP Server:** FastMCP-based Model Context Protocol server for AI assistant integration\n\n## Setup\n\n```bash\n# REQUIRED before running tests (src/ layout — tests hard-exit if package not installed)\npip install -e .\n\n# With dev tools (pytest, ruff, mypy, coverage)\npip install -e \".[dev]\"\n\n# With specific LLM platform support\npip install -e \".[gemini]\"      # Google Gemini\npip install -e \".[openai]\"      # OpenAI ChatGPT\npip install -e \".[all-llms]\"    # All LLM platforms\n\n# With all optional dependencies (except video-full)\npip install -e \".[all]\"\n\n# Full video processing (heavy dependencies)\npip install -e \".[video-full]\"\n```\n\nNote: `tests/conftest.py` checks that `skill_seekers` is importable and calls `sys.exit(1)` if not. Always install in editable mode first.\n\n### Environment Variables\n\nCreate a `.env` file or export these variables:\n```bash\nANTHROPIC_API_KEY      # For Claude AI enhancement\nGOOGLE_API_KEY         # For Gemini support\nOPENAI_API_KEY         # For OpenAI support\nGITHUB_TOKEN           # For GitHub repo scraping (higher rate limits)\n```\n\n## Build / Test / Lint\n\n```bash\n# Full suite (never skip — all must pass)\npytest tests/ -v\n\n# Fast iteration (skip slow, integration, E2E, network, MCP)\npytest tests/ -m \"not slow and not integration and not e2e and not network and not serial and not mcp_only\" -q\n\n# Fast parallel (install pytest-xdist first)\npytest tests/ -n auto --dist=loadfile -m \"not slow and not integration and not e2e and not network and not serial and not mcp_only\" -q\n\n# 3-phase runner script (recommended for local dev)\nbash scripts/run_tests_fast.sh\n\n# Single test\npytest tests/test_scraper_features.py::test_detect_language -v\n\n# Skip slow/integration\npytest tests/ -v -m \"not slow and not integration\"\n\n# With coverage\npytest tests/ --cov=src/skill_seekers --cov-report=term\n\n# Lint + format check (matches CI)\nruff check src/ tests/\nruff format --check src/ tests/\n\n# Type check (non-blocking — mypy is continue-on-error in CI)\nmypy src/skill_seekers --show-error-codes --pretty\n```\n\n**Pytest config:** `asyncio_mode = \"auto\"`, so `@pytest.mark.asyncio` is implicit. Test markers: `slow`, `integration`, `e2e`, `venv`, `bootstrap`, `benchmark`, `asyncio`, `serial`, `network`, `mcp_only`.\n\n**CI note:** CI pins `ruff==0.15.8` (not the `>=0.14.13` dev dep). If formatting behaves differently locally, check the CI version.\n\n**CI test phases:** Tests are split into 3 parallel jobs:\n- `test-fast` — 3386 unit tests with xdist across OS/Python matrix\n- `test-serial` — 69 serial/integration/E2E/network tests\n- `test-mcp` — 193 MCP tests (requires `[mcp]` extras)\n\n## Code Style\n\n### Formatting Rules (ruff — from pyproject.toml)\n- **Line length:** 100 characters\n- **Target Python:** 3.10+\n- **Enabled lint rules:** E, W, F, I, B, C4, UP, ARG, SIM\n- **Ignored rules:** E501 (line length handled by formatter), F541 (f-string style), ARG002 (unused method args for interface compliance), B007 (intentional unused loop vars), I001 (formatter handles imports), SIM114 (readability preference)\n\n### Imports\n- Sort with isort (via ruff); `skill_seekers` is first-party\n- Standard library → third-party → first-party, separated by blank lines\n- Use `from __future__ import annotations` only if needed for forward refs\n- Guard optional imports with try/except ImportError (see `adaptors/__init__.py` pattern):\n  ```python\n  try:\n      from .claude import ClaudeAdaptor\n      from .minimax import MiniMaxAdaptor\n  except ImportError:\n      ClaudeAdaptor = None\n      MiniMaxAdaptor = None\n  ```\n\n### Naming Conventions\n- **Files:** `snake_case.py` (e.g., `source_detector.py`, `config_validator.py`)\n- **Classes:** `PascalCase` (e.g., `SkillAdaptor`, `ClaudeAdaptor`, `SourceDetector`)\n- **Functions/methods:** `snake_case` (e.g., `get_adaptor()`, `detect_language()`)\n- **Constants:** `UPPER_CASE` (e.g., `ADAPTORS`, `DEFAULT_CHUNK_TOKENS`, `VALID_SOURCE_TYPES`)\n- **Private:** prefix with `_` (e.g., `_read_existing_content()`, `_validate_unified()`)\n\n### Type Hints\n- Gradual typing — add hints where practical, not enforced everywhere\n- Use modern syntax: `str | None` not `Optional[str]`, `list[str]` not `List[str]`\n- MyPy config: `disallow_untyped_defs = false`, `check_untyped_defs = true`, `ignore_missing_imports = true`\n- Tests are excluded from strict type checking (`disallow_untyped_defs = false`, `check_untyped_defs = false` for `tests.*`)\n\n### Docstrings\n- Module-level docstring on every file (triple-quoted, describes purpose)\n- Google-style docstrings for public functions/classes\n- Include `Args:`, `Returns:`, `Raises:` sections where useful\n\n### Error Handling\n- Use specific exceptions, never bare `except:`\n- Provide helpful error messages with context\n- Use `raise ValueError(...)` for invalid arguments, `raise RuntimeError(...)` for state errors\n- Guard optional dependency imports with try/except and give clear install instructions on failure\n- Chain exceptions with `raise ... from e` when wrapping\n\n### Suppressing Lint Warnings\n- Use inline `# noqa: XXXX` comments (e.g., `# noqa: F401` for re-exports, `# noqa: ARG001` for required but unused params)\n\n## Project Layout\n\n```\nsrc/skill_seekers/           # Main package (src/ layout)\n  cli/                       # CLI commands and entry points (100+ files)\n    adaptors/                # Platform adaptors (Strategy pattern, inherit SkillAdaptor)\n    arguments/               # CLI argument definitions (one per source type)\n    parsers/                 # Subcommand parsers (one per source type)\n    storage/                 # Cloud storage (inherit BaseStorageAdaptor)\n    main.py                  # Unified CLI entry point (COMMAND_MODULES dict)\n    source_detector.py       # Auto-detects source type from user input\n    create_command.py        # Unified `create` command routing\n    config_validator.py      # VALID_SOURCE_TYPES set + per-type validation\n    unified_scraper.py       # Multi-source orchestrator (scraped_data + dispatch)\n    unified_skill_builder.py # Pairwise synthesis + generic merge\n  mcp/                       # MCP server (FastMCP + legacy)\n    tools/                   # MCP tool implementations by category (10 files)\n    server_fastmcp.py        # FastMCP server implementation\n    server_legacy.py         # Legacy MCP server\n  sync/                      # Sync monitoring (Pydantic models)\n  benchmark/                 # Benchmarking framework\n  embedding/                 # FastAPI embedding server\n  workflows/                 # 67 YAML workflow presets\n  _version.py                # Reads version from pyproject.toml\ntests/                       # 160 test files (pytest)\n  test_adaptors/             # 22 adaptor-specific test files\n  conftest.py                # Test configuration with package check\nconfigs/                     # Preset JSON scraping configs\ndocs/                        # Documentation (guides, integrations, architecture)\n```\n\n## Key Patterns\n\n**Adaptor (Strategy) pattern** — all platform logic in `cli/adaptors/`. Inherit `SkillAdaptor`, implement `format_skill_md()`, `package()`, `upload()`. Register in `adaptors/__init__.py` ADAPTORS dict.\n\n**Scraper pattern** — each source type has: `cli/<type>_scraper.py` (with `<Type>ToSkillConverter` class + `main()`), `arguments/<type>.py`, `parsers/<type>_parser.py`. Register in `parsers/__init__.py` PARSERS list, `main.py` COMMAND_MODULES dict, `config_validator.py` VALID_SOURCE_TYPES set.\n\n**Unified pipeline** — `unified_scraper.py` dispatches to per-type `_scrape_<type>()` methods. `unified_skill_builder.py` uses pairwise synthesis for docs+github+pdf combos and `_generic_merge()` for all other combinations.\n\n**MCP tools** — grouped in `mcp/tools/` by category. `scrape_generic_tool` handles all new source types.\n\n**CLI subcommands** — git-style in `cli/main.py`. Each delegates to a module's `main()` function.\n\n**Supported source types (17):** documentation (web), github, pdf, local, word, video, epub, jupyter, html, openapi, asciidoc, pptx, confluence, notion, rss, manpage, chat. Each detected automatically by `source_detector.py`.\n\n**Supported platforms (21):** claude, gemini, openai, minimax, opencode, kimi, deepseek, qwen, openrouter, together, fireworks, markdown, langchain, llama-index, haystack, weaviate, chroma, faiss, qdrant, pinecone.\n\n## CLI Commands\n\n```bash\n# Core commands\nskill-seekers create <source>              # Create skill from any source (auto-detects type)\nskill-seekers scan <dir>                   # AI-detect a project's tech stack and emit per-framework configs\nskill-seekers enhance <directory>          # AI-powered enhancement\nskill-seekers package <directory>          # Package skill for target platform\nskill-seekers upload <file>                # Upload skill to target platform\nskill-seekers install <source>             # One-command workflow (scrape + enhance + package + upload)\n\n# Utilities\nskill-seekers estimate <source>            # Estimate page count before scraping\nskill-seekers doctor                       # Health check for dependencies\nskill-seekers config                       # Configure API keys and settings\nskill-seekers workflows                    # List and apply workflow presets\nskill-seekers resume <job_id>              # Resume interrupted scraping\n\n# Advanced\nskill-seekers stream <source>              # Streaming ingestion\nskill-seekers update <directory>           # Incremental update\nskill-seekers multilang <directory>        # Multi-language support\n```\n\n## Testing Instructions\n\n### Test Structure\n- Unit tests: `tests/test_*.py` — test individual modules\n- Adaptor tests: `tests/test_adaptors/test_*_adaptor.py` — test platform adaptors\n- E2E tests: `tests/test_*_e2e.py` — end-to-end integration tests\n\n### Running Tests\n```bash\n# Fast test run (skip slow/integration tests)\npytest tests/ -v -m \"not slow and not integration\"\n\n# Full test suite\npytest tests/ -v\n\n# With coverage report\npytest tests/ --cov=src/skill_seekers --cov-report=term-missing\n\n# Specific test categories\npytest tests/ -v -m \"slow\"           # Only slow tests\npytest tests/ -v -m \"integration\"    # Only integration tests\npytest tests/ -v -m \"e2e\"            # Only E2E tests\n```\n\n### Test Fixtures\nTest fixtures are located in `tests/fixtures/` and include sample configs, HTML files, and mock data.\n\n## Git Workflow\n\n- **`main`** — production, protected\n- **`development`** — default PR target, active dev\n- Feature branches created from `development`\n\n## Pre-commit Checklist\n\n```bash\nruff check src/ tests/\nruff format --check src/ tests/\npytest tests/ -v -x   # stop on first failure\n```\n\nNever commit API keys. Use env vars: `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`, `GITHUB_TOKEN`.\n\n## CI/CD\n\nGitHub Actions (7 workflows in `.github/workflows/`):\n- **tests.yml** — ruff + mypy lint job, then pytest matrix (Ubuntu + macOS, Python 3.10-3.12) with Codecov upload\n- **release.yml** — tag-triggered: tests → version verification → PyPI publish via `uv build`\n- **test-vector-dbs.yml** — tests vector DB adaptors (weaviate, chroma, faiss, qdrant)\n- **docker-publish.yml** — multi-platform Docker builds (amd64, arm64) for CLI + MCP images\n- **quality-metrics.yml** — quality analysis with configurable threshold\n- **scheduled-updates.yml** — weekly skill updates for popular frameworks\n- **vector-db-export.yml** — weekly vector DB exports\n\n## Deployment\n\n### Docker\nMulti-stage Dockerfile with Python 3.12 slim base:\n```bash\n# Build CLI image\ndocker build -t skill-seekers:local -f Dockerfile .\n\n# Run CLI\ndocker run -v $(pwd)/output:/output skill-seekers:local create https://docs.example.com\n\n# Run MCP server\ndocker build -t skill-seekers-mcp:local -f Dockerfile.mcp .\ndocker run -p 8765:8765 skill-seekers-mcp:local\n```\n\n### MCP Server\nThe MCP server provides Model Context Protocol integration:\n```bash\n# Start FastMCP server\nskill-seekers-mcp\n\n# Or use the Python module\npython -m skill_seekers.mcp.server_fastmcp\n```\n\n## Security Considerations\n\n- **API Keys:** Never commit API keys to version control. Use environment variables or `.env` files (already in `.gitignore`)\n- **Docker:** Runs as non-root user (`skillseeker`, UID 1000)\n- **Dependencies:** Regular security updates via `pip audit` or `safety check`\n- **Sandboxing:** Video processing uses optional dependencies that can be heavy; install `[video-full]` only when needed\n\n## Additional Resources\n\n- **Website:** https://skillseekersweb.com/\n- **Documentation:** https://skillseekersweb.com/\n- **PyPI:** https://pypi.org/project/skill-seekers/\n- **Repository:** https://github.com/yusufkaraaslan/Skill_Seekers\n- **Config Browser:** https://skillseekersweb.com/\n- **Project Board:** https://github.com/users/yusufkaraaslan/projects/2\n"},"items":[{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\n**Skill Seekers** converts documentation from 18 source types into production-ready formats for 21+ AI platforms (LLM platforms, RAG frameworks, vector databases, AI coding assistants). Published on PyPI as `skill-seekers`.\n\n**Version:** 3.9.0 (dev; last release 3.8.0) — source of truth is `src/skill_seekers/_version.py` | **Python:** 3.10+ | **Website:** https://skillseekersweb.com/\n\n**Architecture:** See `docs/UML_ARCHITECTURE.md` for UML diagrams and module overview. StarUML project at `docs/UML/skill_seekers.mdj`. Refactor state/history: `docs/UNIFICATION_PLAN.md` (Grand Unification — all 5 phases done; remaining cosmetic items listed there).\n\n## Essential Commands\n\n```bash\n# REQUIRED before running tests or CLI (src/ layout)\npip install -e .\n\n# Run all tests (NEVER skip - all must pass before commits)\npytest tests/ -v\n\n# Fast iteration (skip slow MCP tests ~20min)\npytest tests/ --ignore=tests/test_mcp_fastmcp.py --ignore=tests/test_mcp_server.py --ignore=tests/test_install_skill_e2e.py -q\n\n# Single test\npytest tests/test_scraper_features.py::test_detect_language -vv -s\n\n# Code quality (must pass before push - matches CI)\nuvx ruff check src/ tests/\nuvx ruff format --check src/ tests/\nmypy src/skill_seekers  # continue-on-error in CI\n\n# Auto-fix lint/format issues\nuvx ruff check --fix --unsafe-fixes src/ tests/\nuvx ruff format src/ tests/\n\n# Build & publish\nuv build\nuv publish\n```\n\n## CI Matrix\n\nRuns on push/PR to `main` or `development`. Lint job (Python 3.12, Ubuntu) + Test job (Ubuntu + macOS, Python 3.10/3.11/3.12, excludes macOS+3.10). Both must pass for merge.\n\n## Git Workflow\n\n- **Main branch:** `main` (requires tests + 1 review)\n- **Development branch:** `development` (default PR target, requires tests)\n- **Feature branches:** `feature/{task-id}-{description}` from `development`\n- PRs always target `development`, never `main` directly\n\n## Architecture\n\n### CLI: Unified create command\n\nEntry point `src/skill_seekers/cli/main.py`. The `create` command is the **primary** entry point for skill creation — it auto-detects source type and routes to the appropriate `SkillConverter`. The `scan` command (added in #327) is a separate discovery step for projects with multiple frameworks; it emits one config file per detected framework and you then run `create` on each.\n\n```\nskill-seekers create <source>     # Auto-detect: URL, owner/repo, ./path, file.pdf, etc.\nskill-seekers scan <dir>          # AI-driven discovery → emits one config per detected framework + <project>-codebase.json\nskill-seekers package <dir>       # Package for platform (--target claude/gemini/openai/markdown/minimax/opencode/kimi/deepseek/qwen/openrouter/together/fireworks/atlas/langchain/llama-index/haystack/chroma/faiss/weaviate/qdrant/pinecone/ibm-bob)\n```\n\n### Scan command (issue #327)\n\n`skill-seekers scan <dir>` is an AI-driven project knowledge-base bootstrapper. Pipeline in `src/skill_seekers/cli/scan_command.py`:\n\n1. `collect_signals()` in `signal_collectors.py` — deterministic, bounded gathering of manifests + README + Dockerfile/CI + sampled source files + git remote. **Per-kind byte budgets** (24 KB manifest / 6 KB README / 6 KB CI / 28 KB samples, total 64 KB) so a fat package.json can't crowd out other kinds. `_SOURCE_DIRS` covers ~14 layouts (Go `cmd/`, Rust `crates/`, JS monorepo `apps/packages/`, Maven `source/`, Django at root); also walks root one level deep for flat-layout Python.\n2. `detect_with_ai(bundle, AgentClient)` — one LLM call, structured JSON output. **Source signals are first-2-KB of each file** (whole-file sampling, no regex parsing — added in WS4 because regex missed Go multi-line imports + Rust `mod`/`extern crate`). Canonical-slug prompt + the canonical-name resolver are coupled — change one, update the other.\n3. `resolve_or_generate_with_status()` — for each detection: try `out_dir/<slug>.json` (cache from prior run), then `resolve_config_path` from `config_fetcher` with multiple canonical name candidates (`_canonical_name_candidates` handles `\"Godot Engine\"` → `\"godot\"`, plus CJK / European suffixes like `\"Godot 引擎\"`, `\"React フレームワーク\"`, `\"Lodash Bibliothek\"`), then `generate_config_with_ai` as the last resort. Always appends `.json` to lookup names so local-disk and user-dir resolution actually finds files. Always stamps `metadata.detected_version` (nested, not top-level — `metadata.version` already exists and means config-schema version).\n4. `emit_codebase_config()` — always writes `<project>-codebase.json` (a `type: local` source pointed at the project root).\n5. `diff_against_existing()` — keyed by **filename slug** (not internal `data[\"name\"]`) so re-scans don't churn when the AI returns a display name vs the registry canonical slug.\n6. `_archive_removed()` — when a config disappears from detections, MOVE (not delete — user may have hand-edited) to `out_dir/.archived/<UTC-timestamp>/`. Runs after diff, before fresh writes.\n7. `maybe_publish()` — **native async** (WS11). Opt-in submission of freshly AI-generated configs to the community registry. Pre-checks `GITHUB_TOKEN`. Idempotency guard: `_find_existing_issue` queries GitHub Search API for an existing open issue with the same config name before submitting. Retries transient failures (rate limit, 5xx) with 0s/5s/15s backoff. `_prompt_async` wraps `input()` via `asyncio.to_thread` so the event loop isn't blocked.\n\n**CLI dispatch** uses the `COMMAND_CLASSES` table in `main.py` (added in WS1). `scan` and `doctor` are dispatched as `Cls(args).execute()` consuming the parsed argparse namespace directly — no `_reconstruct_argv` hack, no duplicate argparse. `ScanCommand.execute()` is the single `asyncio.run` boundary wrapping `run_scan` (sync) + `maybe_publish` (async). Remaining ~14 commands still use the legacy `COMMAND_MODULES` dispatch; they're flagged for migration.\n\n**Cost guardrails**: `--max-ai-generations N` (default 10) caps unbounded AI generation; `--dry-run` previews without writing or invoking AI; `--probe-urls` HEAD-checks AI-generated URLs with retry-on-404 and stamps `metadata._url_unverified` on confirmed-bad URLs.\n\n**Safety**: All writes use `_atomic_write_json` (`os.replace` after writing to `.tmp`) so a `KeyboardInterrupt` mid-write can't corrupt configs. `_safe_size` guards `stat()` so broken symlinks don't crash the scan. `ScanCommand.execute` calls `logging.basicConfig` so `logger.warning`/`error` is visible; exit code is non-zero when no configs and no codebase config were emitted.\n\n**Public constant**: `SourceDetector.CODE_PROJECT_MARKERS` (was `_CODE_PROJECT_MARKERS`) — shared between source_detector + signal_collectors. ~50 manifest types now (Pipfile, environment.yml, deno.json, flake.nix, Chart.yaml, deps.edn, dune-project, BUILD.bazel, …). Public so cross-module access doesn't reach into a private attribute.\n\n### SkillConverter Pattern (Template Method + Factory)\n\nAll 18 source types implement the `SkillConverter` base class (`skill_converter.py`):\n\n```python\nconverter = get_converter(\"web\", config)  # Factory lookup\nconverter.run()  # Template: extract() → build_skill()\n```\n\nRegistry in `CONVERTER_REGISTRY` maps source type → (module, class). `create_command.py` builds config from `ExecutionContext`, calls `get_converter()`, then runs centralized enhancement. `get_converter(\"config\", {...})` constructs `UnifiedScraper` from the same factory-shaped dict (no special cases in create_command/MCP). The base resolves `skill_dir` once (strips trailing separators) and derives `data_file` via `data_file_for()` — subclasses must not re-derive paths.\n\n### DocumentSkillBuilder (build side of 9 document scrapers)\n\n`cli/document_skill_builder.py:DocumentSkillBuilder` sits between `SkillConverter` and the 9 document scrapers (epub, word, pptx, html, pdf, jupyter, man, rss, chat). It owns `categorize_content`, reference-file writing (tables, truncation, image guard), `index.md` + `SKILL.md` generation, and `load_extracted_data`. Variation points are class attrs (`DOC_NOUN`, `SOURCE_LABEL`, `LOAD_TOTAL_KEY`, `PATTERN_KEYWORDS`, `RANGE_LABEL`, …) and small hook methods (`category_stem`, `_write_reference_section`, `_write_skill_md_metadata`). Output is pinned **byte-identical** by golden trees in `tests/golden/phase2/` — `UPDATE_GOLDENS=1` rewrites them, only do that deliberately. Surviving full-method overrides are domain-shaped and commented per scraper.\n\n### UnifiedScraper (multi-source configs)\n\n`unified_scraper.py` dispatches via the class-level `SOURCE_DISPATCH` table; `_scrape_with_converter()` is the shared engine for the 13 mechanical source types (`get_converter()` + public `converter.extract()` + cache copy + sub-skill build), so **new types registered in `CONVERTER_REGISTRY` work in unified configs automatically**. documentation/github/local stay bespoke (commented why). `run()` deliberately does NOT follow the base template (TestRunOrchestration pins that run() triggers workflows).\n\n### Data Flow (5 phases)\n\n1. **Scrape** - Source-specific scraper extracts content to `output/{name}_data/pages/*.json`\n2. **Build** - `build_skill()` categorizes pages, extracts patterns, generates `output/{name}/SKILL.md`\n3. **Enhance** (optional) - LLM rewrites SKILL.md (`--enhance-level 0-3`, auto-detects API vs LOCAL mode)\n4. **Package** - Platform adaptor formats output (`.zip`, `.tar.gz`, JSON, vector index)\n5. **Upload** (optional) - Platform API upload\n\n### Platform Adaptor Pattern (Strategy + Factory)\n\nFactory: `get_adaptor(platform, config)` in `adaptors/__init__.py` returns a `SkillAdaptor` instance. Base class `SkillAdaptor` + `SkillMetadata` in `adaptors/base.py`.\n\n```\nsrc/skill_seekers/cli/adaptors/\n├── __init__.py              # Factory: get_adaptor(platform, config), ADAPTORS registry\n├── base.py                  # Abstract base: SkillAdaptor, SkillMetadata\n├── openai_compatible.py     # Shared base for OpenAI-compatible platforms\n├── claude.py                # --target claude\n├── gemini.py                # --target gemini\n├── openai.py                # --target openai\n├── markdown.py              # --target markdown\n├── minimax.py               # --target minimax\n├── opencode.py              # --target opencode\n├── kimi.py                  # --target kimi\n├── deepseek.py              # --target deepseek\n├── qwen.py                  # --target qwen\n├── openrouter.py            # --target openrouter\n├── together.py              # --target together\n├── fireworks.py             # --target fireworks\n├── langchain.py             # --target langchain\n├── llama_index.py           # --target llama-index\n├── haystack.py              # --target haystack\n├── chroma.py                # --target chroma\n├── faiss_helpers.py         # --target faiss\n├── qdrant.py                # --target qdrant\n├── weaviate.py              # --target weaviate\n├── pinecone_adaptor.py      # --target pinecone\n└── streaming_adaptor.py     # --target streaming\n```\n\nAll adaptors use `--target`. All adaptors are imported with `try/except ImportError` so missing optional deps don't break the registry.\n\n### 18 Source Type Converters\n\nEach in `src/skill_seekers/cli/{type}_scraper.py` as a `SkillConverter` subclass (no `main()`). The `create_command.py` uses `source_detector.py` to auto-detect, then calls `get_converter()`. Converters: web (doc_scraper), github, pdf, word, epub, video, local (codebase_scraper), jupyter, html, openapi, asciidoc, pptx, rss, manpage, confluence, notion, chat, config (unified_scraper).\n\n### CLI Argument System (single-definition parsers)\n\n```\nsrc/skill_seekers/cli/\n├── parsers/              # Central SubcommandParser classes — the ONLY definition of each command's flags\n│   └── create_parser.py  # Progressive help disclosure (--help-web, --help-github, etc.)\n├── arguments/            # Argument definitions\n│   ├── common.py         # add_all_standard_arguments() - shared across all scrapers\n│   └── create.py         # UNIVERSAL_ARGUMENTS, WEB_ARGUMENTS, GITHUB_ARGUMENTS, etc.\n├── exit_codes.py         # EXIT_SUCCESS/ERROR/VALIDATION/INTERRUPT\n└── source_detector.py    # Auto-detect source type from input string\n```\n\nCommand modules' standalone `main(args=None)` paths build their parser FROM the central `SubcommandParser` class — **add/change a flag in `parsers/*.py` only**. Drift guards (`tests/test_cli_parsers.py::TestCentralModuleParserSync` and `TestCentralParserSingleSource`) fail CI on any divergence of dests/defaults/option strings.\n\n`ExecutionContext.override()` is context-local (a `ContextVar` layered over the unchanged base singleton) — thread/async safe for the MCP server; propagate to worker threads via `copy_context`.\n\n### Standalone subsystems (outside `cli/`)\n\nFour top-level packages sit beside `cli/` and are largely independent of the scrape→build→package flow:\n\n- `embedding/` — FastAPI embedding-generation server (`python -m skill_seekers.embedding.server`) with a caching layer (`cache.py`) and multi-backend generators (OpenAI, sentence-transformers, Anthropic). Feeds the vector-DB adaptors.\n- `sync/` — real-time doc-sync system: `detector.py` (content-hash / last-modified change detection), `monitor.py` (scheduled incremental re-scrapes), `notifier.py` (email/Slack/webhook). Keeps generated skills fresh as upstream docs change.\n- `benchmark/` — performance suite (`runner.py`, `framework.py`) measuring scrape/embedding/storage/e2e timing, memory, and CPU; emits comparison + optimization reports.\n- `workflows/` — bundled default enhancement-workflow presets consumed by the enhancement step.\n\n### C3.x Codebase Analysis Pipeline\n\nLocal codebase analysis features, all opt-out (`--skip-*` flags):\n- C3.1 `pattern_recognizer.py` - Design pattern detection (10 GoF patterns, 9 languages)\n- C3.2 `test_example_extractor.py` - Usage examples from tests\n- C3.3 `how_to_guide_builder.py` - AI-enhanced educational guides\n- C3.4 `config_extractor.py` - Configuration pattern extraction\n- C3.5 `generate_router.py` - Architecture overview generation\n- C3.10 `signal_flow_analyzer.py` - Godot signal flow analysis\n\n### MCP Server\n\n`src/skill_seekers/mcp/server_fastmcp.py` - 40 tools via FastMCP. Transport: stdio (Claude Code) or HTTP (Cursor/Windsurf). Optional dependency: `pip install -e \".[mcp]\"`\n\n- **Tools run in-process** via `run_cli_main()` in `mcp/tools/_common.py`: same argv parsed by the command's REAL parser (sys.argv patch under a lock), stdout/stderr capture + contextvar log capture, identical `(stdout, stderr, returncode)` contract. No subprocess startup; old hard timeouts are advisory.\n- **Exceptions BY DESIGN**: `enhance_skill` (LOCAL agent) and `install_skill`'s enhancement step stay subprocess — the agent must be a real child process for the fork-bomb-guard env semantics (`SKILL_SEEKER_ENHANCE_ACTIVE`). Never make these in-process.\n- **Domain logic lives in `skill_seekers.services/`** (marketplace_manager, marketplace_publisher, config_publisher, source_manager, git_repo) — importable by CLI without the `[mcp]` extra; old `skill_seekers.mcp.*` paths are back-compat shims. No `sys.path` hacks anywhere in `mcp/`.\n\n### Enhancement (AgentClient is the single AI transport)\n\nEvery AI call goes through `AgentClient` (`src/skill_seekers/cli/agent_client.py`): central truncation gate, timeout policy, error classification. `API_PROVIDERS` (provider registry) and `AGENT_PRESETS` (local-agent command templates) live ONLY there. Each `API_PROVIDERS` entry declares its wire `protocol` (`anthropic`/`openai`/`google`) and `supports_images` capability — `_call_api` branches on the resolved protocol, NOT the provider name, so an OpenAI/Anthropic-compatible provider needs no new branch. Adaptors declare provider/endpoint/model/prompt and route through `SkillAdaptor._enhance_skill_md_via_client` (atomic save with backup). Multimodal image input goes through `AgentClient.call_with_image()` (used by `video_visual` frame OCR across all image-capable providers); it no longer bypasses AgentClient with a direct SDK call.\n\n- **API mode** (if API key set): Anthropic, Google Gemini, OpenAI, Moonshot/Kimi, MiniMax — detected in registry order; `SKILL_SEEKER_PROVIDER` forces one. Models: `SKILL_SEEKER_MODEL` (global) or `ANTHROPIC_MODEL`/`GOOGLE_MODEL`/`OPENAI_MODEL`/`MOONSHOT_MODEL`/`MINIMAX_MODEL`; `ANTHROPIC_BASE_URL` for compatible endpoints. MiniMax adds `MINIMAX_API_REGION` (`global_en`/`cn_zh`) and `MINIMAX_API_PROTOCOL` (`openai`/`anthropic`). Vision OCR provider: `SKILL_SEEKER_VISION_PROVIDER` (`auto` picks the first image-capable provider with a key).\n- **LOCAL mode** (fallback): Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents — command built by `build_local_agent_command()`.\n- Control: `--enhance-level 0` (off) / `1` (SKILL.md only) / `2` (default, balanced) / `3` (full)\n- Agent selection: `--agent claude|codex|copilot|opencode|kimi|custom`\n\n## Key Implementation Details\n\n### Smart Categorization (`doc_scraper.py:smart_categorize()`)\n\nScores pages against category keywords: 3 points for URL match, 2 for title, 1 for content. Threshold of 2+ required. Falls back to \"other\".\n\n### Content Extraction (`doc_scraper.py`)\n\n`FALLBACK_MAIN_SELECTORS` constant + `_find_main_content()` helper handle CSS selector fallback. Links are extracted from the full page before early return (not just main content). `body` is deliberately excluded from fallbacks.\n\n### Three-Stream GitHub Architecture (`unified_codebase_analyzer.py`)\n\nStream 1: Code Analysis (AST, patterns, tests, guides). Stream 2: Documentation (README, docs/, wiki). Stream 3: Community (issues, PRs, metadata). Depth control: `basic` (1-2 min) or `c3x` (20-60 min).\n\n## Testing\n\n### Test markers (pytest.ini)\n\n```bash\npytest tests/ -v                                    # Default: fast tests only\npytest tests/ -v -m slow                            # Include slow tests (>5s)\npytest tests/ -v -m integration                     # External services required\npytest tests/ -v -m e2e                             # Resource-intensive\npytest tests/ -v -m \"not slow and not integration\"  # Fastest subset\n```\n\n### Known legitimate skips (~11)\n\n- 2: chromadb incompatible with Python 3.14 (pydantic v1)\n- 2: weaviate-client not installed\n- 2: Qdrant not running (requires docker)\n- 2: langchain/llama_index not installed\n- 3: GITHUB_TOKEN not set\n\n### sys.modules gotcha\n\n`test_swift_detection.py` deletes `skill_seekers.cli` modules from `sys.modules`. It must save and restore both `sys.modules` entries AND parent package attributes (`setattr`). See the test file for the pattern.\n\n## Dependencies\n\nCore deps include `langchain`, `llama-index`, `anthropic`, `httpx`, `PyMuPDF`, `pydantic`. Platform-specific deps are optional:\n\n```bash\npip install -e \".[mcp]\"       # MCP server\npip install -e \".[gemini]\"    # Google Gemini\npip install -e \".[openai]\"    # OpenAI\npip install -e \".[docx]\"      # Word documents\npip install -e \".[epub]\"      # EPUB books\npip install -e \".[video]\"     # Video (lightweight)\npip install -e \".[video-full]\"# Video (Whisper + visual)\npip install -e \".[jupyter]\"   # Jupyter notebooks\npip install -e \".[pptx]\"      # PowerPoint\npip install -e \".[rss]\"       # RSS/Atom feeds\npip install -e \".[confluence]\"# Confluence wiki\npip install -e \".[notion]\"    # Notion pages\npip install -e \".[chroma]\"    # ChromaDB\npip install -e \".[all]\"       # Everything (except video-full)\n```\n\nDev dependencies use PEP 735 `[dependency-groups]` in pyproject.toml.\n\n## Environment Variables\n\n```bash\nANTHROPIC_API_KEY=sk-ant-...          # Claude AI (or compatible endpoint)\nANTHROPIC_BASE_URL=https://...        # Optional: Claude-compatible API endpoint\nGOOGLE_API_KEY=AIza...                # Google Gemini (optional)\nOPENAI_API_KEY=sk-...                 # OpenAI (optional)\nGITHUB_TOKEN=ghp_...                  # Higher GitHub rate limits\n```\n\n## Adding New Features\n\n### New platform adaptor\n1. Create `src/skill_seekers/cli/adaptors/{platform}.py` inheriting `SkillAdaptor` from `base.py`\n2. Register in `adaptors/__init__.py` (add try/except import + add to `ADAPTORS` dict)\n3. Add optional dep to `pyproject.toml`\n4. Add tests in `tests/`\n\n### New source type converter\n1. Create `src/skill_seekers/cli/{type}_scraper.py` — for document-shaped sources inherit `DocumentSkillBuilder` (categorization/references/index/SKILL.md come free; implement `extract()` + hooks), otherwise inherit `SkillConverter` and implement `extract()` and `build_skill()`. Set `SOURCE_TYPE`.\n2. Register in `CONVERTER_REGISTRY` in `skill_converter.py` — this also makes the type work in unified configs automatically (UnifiedScraper engine)\n3. Add source type config building in `create_command.py:_build_config()`\n4. Add auto-detection in `source_detector.py`\n5. Add optional dep if needed\n6. Add tests\n\n### New CLI argument\n- Subcommand flag: define ONLY in the central parser class (`parsers/{cmd}_parser.py`) — module `main()` builds from it; the drift-guard test fails otherwise\n- Universal: `UNIVERSAL_ARGUMENTS` in `arguments/create.py`\n- Source-specific: appropriate dict (`WEB_ARGUMENTS`, `GITHUB_ARGUMENTS`, etc.)\n- Shared across scrapers: `add_all_standard_arguments()` in `arguments/common.py`\n","category":"root","tokens":5323},{"name":"AGENTS.md","path":"AGENTS.md","title":"AGENTS.md","content":"# AGENTS.md - Skill Seekers\n\nComprehensive reference for AI coding agents. Skill Seekers is a Python CLI tool (v3.6.0) that converts documentation sites, GitHub repos, PDFs, videos, notebooks, wikis, and more into AI-ready skills for 21+ LLM platforms and RAG pipelines.\n\n## Project Overview\n\n**Skill Seekers** is a universal preprocessing layer that transforms raw documentation and code into structured knowledge assets. It supports 17+ source types and exports to 21+ AI platforms including Claude, Gemini, OpenAI, LangChain, LlamaIndex, and various vector databases.\n\n### Key Capabilities\n- **Source Types (17):** Documentation websites, GitHub repos, PDFs, Word docs, EPUBs, videos, local codebases, Jupyter notebooks, HTML, OpenAPI specs, AsciiDoc, PowerPoint, Confluence, Notion, RSS feeds, man pages, chat exports\n- **Export Targets (21):** Claude, Gemini, OpenAI, MiniMax, OpenCode, Kimi, DeepSeek, Qwen, OpenRouter, Together AI, Fireworks AI, Markdown, LangChain, LlamaIndex, Haystack, Weaviate, ChromaDB, FAISS, Qdrant, Pinecone\n- **MCP Server:** FastMCP-based Model Context Protocol server for AI assistant integration\n\n## Setup\n\n```bash\n# REQUIRED before running tests (src/ layout — tests hard-exit if package not installed)\npip install -e .\n\n# With dev tools (pytest, ruff, mypy, coverage)\npip install -e \".[dev]\"\n\n# With specific LLM platform support\npip install -e \".[gemini]\"      # Google Gemini\npip install -e \".[openai]\"      # OpenAI ChatGPT\npip install -e \".[all-llms]\"    # All LLM platforms\n\n# With all optional dependencies (except video-full)\npip install -e \".[all]\"\n\n# Full video processing (heavy dependencies)\npip install -e \".[video-full]\"\n```\n\nNote: `tests/conftest.py` checks that `skill_seekers` is importable and calls `sys.exit(1)` if not. Always install in editable mode first.\n\n### Environment Variables\n\nCreate a `.env` file or export these variables:\n```bash\nANTHROPIC_API_KEY      # For Claude AI enhancement\nGOOGLE_API_KEY         # For Gemini support\nOPENAI_API_KEY         # For OpenAI support\nGITHUB_TOKEN           # For GitHub repo scraping (higher rate limits)\n```\n\n## Build / Test / Lint\n\n```bash\n# Full suite (never skip — all must pass)\npytest tests/ -v\n\n# Fast iteration (skip slow, integration, E2E, network, MCP)\npytest tests/ -m \"not slow and not integration and not e2e and not network and not serial and not mcp_only\" -q\n\n# Fast parallel (install pytest-xdist first)\npytest tests/ -n auto --dist=loadfile -m \"not slow and not integration and not e2e and not network and not serial and not mcp_only\" -q\n\n# 3-phase runner script (recommended for local dev)\nbash scripts/run_tests_fast.sh\n\n# Single test\npytest tests/test_scraper_features.py::test_detect_language -v\n\n# Skip slow/integration\npytest tests/ -v -m \"not slow and not integration\"\n\n# With coverage\npytest tests/ --cov=src/skill_seekers --cov-report=term\n\n# Lint + format check (matches CI)\nruff check src/ tests/\nruff format --check src/ tests/\n\n# Type check (non-blocking — mypy is continue-on-error in CI)\nmypy src/skill_seekers --show-error-codes --pretty\n```\n\n**Pytest config:** `asyncio_mode = \"auto\"`, so `@pytest.mark.asyncio` is implicit. Test markers: `slow`, `integration`, `e2e`, `venv`, `bootstrap`, `benchmark`, `asyncio`, `serial`, `network`, `mcp_only`.\n\n**CI note:** CI pins `ruff==0.15.8` (not the `>=0.14.13` dev dep). If formatting behaves differently locally, check the CI version.\n\n**CI test phases:** Tests are split into 3 parallel jobs:\n- `test-fast` — 3386 unit tests with xdist across OS/Python matrix\n- `test-serial` — 69 serial/integration/E2E/network tests\n- `test-mcp` — 193 MCP tests (requires `[mcp]` extras)\n\n## Code Style\n\n### Formatting Rules (ruff — from pyproject.toml)\n- **Line length:** 100 characters\n- **Target Python:** 3.10+\n- **Enabled lint rules:** E, W, F, I, B, C4, UP, ARG, SIM\n- **Ignored rules:** E501 (line length handled by formatter), F541 (f-string style), ARG002 (unused method args for interface compliance), B007 (intentional unused loop vars), I001 (formatter handles imports), SIM114 (readability preference)\n\n### Imports\n- Sort with isort (via ruff); `skill_seekers` is first-party\n- Standard library → third-party → first-party, separated by blank lines\n- Use `from __future__ import annotations` only if needed for forward refs\n- Guard optional imports with try/except ImportError (see `adaptors/__init__.py` pattern):\n  ```python\n  try:\n      from .claude import ClaudeAdaptor\n      from .minimax import MiniMaxAdaptor\n  except ImportError:\n      ClaudeAdaptor = None\n      MiniMaxAdaptor = None\n  ```\n\n### Naming Conventions\n- **Files:** `snake_case.py` (e.g., `source_detector.py`, `config_validator.py`)\n- **Classes:** `PascalCase` (e.g., `SkillAdaptor`, `ClaudeAdaptor`, `SourceDetector`)\n- **Functions/methods:** `snake_case` (e.g., `get_adaptor()`, `detect_language()`)\n- **Constants:** `UPPER_CASE` (e.g., `ADAPTORS`, `DEFAULT_CHUNK_TOKENS`, `VALID_SOURCE_TYPES`)\n- **Private:** prefix with `_` (e.g., `_read_existing_content()`, `_validate_unified()`)\n\n### Type Hints\n- Gradual typing — add hints where practical, not enforced everywhere\n- Use modern syntax: `str | None` not `Optional[str]`, `list[str]` not `List[str]`\n- MyPy config: `disallow_untyped_defs = false`, `check_untyped_defs = true`, `ignore_missing_imports = true`\n- Tests are excluded from strict type checking (`disallow_untyped_defs = false`, `check_untyped_defs = false` for `tests.*`)\n\n### Docstrings\n- Module-level docstring on every file (triple-quoted, describes purpose)\n- Google-style docstrings for public functions/classes\n- Include `Args:`, `Returns:`, `Raises:` sections where useful\n\n### Error Handling\n- Use specific exceptions, never bare `except:`\n- Provide helpful error messages with context\n- Use `raise ValueError(...)` for invalid arguments, `raise RuntimeError(...)` for state errors\n- Guard optional dependency imports with try/except and give clear install instructions on failure\n- Chain exceptions with `raise ... from e` when wrapping\n\n### Suppressing Lint Warnings\n- Use inline `# noqa: XXXX` comments (e.g., `# noqa: F401` for re-exports, `# noqa: ARG001` for required but unused params)\n\n## Project Layout\n\n```\nsrc/skill_seekers/           # Main package (src/ layout)\n  cli/                       # CLI commands and entry points (100+ files)\n    adaptors/                # Platform adaptors (Strategy pattern, inherit SkillAdaptor)\n    arguments/               # CLI argument definitions (one per source type)\n    parsers/                 # Subcommand parsers (one per source type)\n    storage/                 # Cloud storage (inherit BaseStorageAdaptor)\n    main.py                  # Unified CLI entry point (COMMAND_MODULES dict)\n    source_detector.py       # Auto-detects source type from user input\n    create_command.py        # Unified `create` command routing\n    config_validator.py      # VALID_SOURCE_TYPES set + per-type validation\n    unified_scraper.py       # Multi-source orchestrator (scraped_data + dispatch)\n    unified_skill_builder.py # Pairwise synthesis + generic merge\n  mcp/                       # MCP server (FastMCP + legacy)\n    tools/                   # MCP tool implementations by category (10 files)\n    server_fastmcp.py        # FastMCP server implementation\n    server_legacy.py         # Legacy MCP server\n  sync/                      # Sync monitoring (Pydantic models)\n  benchmark/                 # Benchmarking framework\n  embedding/                 # FastAPI embedding server\n  workflows/                 # 67 YAML workflow presets\n  _version.py                # Reads version from pyproject.toml\ntests/                       # 160 test files (pytest)\n  test_adaptors/             # 22 adaptor-specific test files\n  conftest.py                # Test configuration with package check\nconfigs/                     # Preset JSON scraping configs\ndocs/                        # Documentation (guides, integrations, architecture)\n```\n\n## Key Patterns\n\n**Adaptor (Strategy) pattern** — all platform logic in `cli/adaptors/`. Inherit `SkillAdaptor`, implement `format_skill_md()`, `package()`, `upload()`. Register in `adaptors/__init__.py` ADAPTORS dict.\n\n**Scraper pattern** — each source type has: `cli/<type>_scraper.py` (with `<Type>ToSkillConverter` class + `main()`), `arguments/<type>.py`, `parsers/<type>_parser.py`. Register in `parsers/__init__.py` PARSERS list, `main.py` COMMAND_MODULES dict, `config_validator.py` VALID_SOURCE_TYPES set.\n\n**Unified pipeline** — `unified_scraper.py` dispatches to per-type `_scrape_<type>()` methods. `unified_skill_builder.py` uses pairwise synthesis for docs+github+pdf combos and `_generic_merge()` for all other combinations.\n\n**MCP tools** — grouped in `mcp/tools/` by category. `scrape_generic_tool` handles all new source types.\n\n**CLI subcommands** — git-style in `cli/main.py`. Each delegates to a module's `main()` function.\n\n**Supported source types (17):** documentation (web), github, pdf, local, word, video, epub, jupyter, html, openapi, asciidoc, pptx, confluence, notion, rss, manpage, chat. Each detected automatically by `source_detector.py`.\n\n**Supported platforms (21):** claude, gemini, openai, minimax, opencode, kimi, deepseek, qwen, openrouter, together, fireworks, markdown, langchain, llama-index, haystack, weaviate, chroma, faiss, qdrant, pinecone.\n\n## CLI Commands\n\n```bash\n# Core commands\nskill-seekers create <source>              # Create skill from any source (auto-detects type)\nskill-seekers scan <dir>                   # AI-detect a project's tech stack and emit per-framework configs\nskill-seekers enhance <directory>          # AI-powered enhancement\nskill-seekers package <directory>          # Package skill for target platform\nskill-seekers upload <file>                # Upload skill to target platform\nskill-seekers install <source>             # One-command workflow (scrape + enhance + package + upload)\n\n# Utilities\nskill-seekers estimate <source>            # Estimate page count before scraping\nskill-seekers doctor                       # Health check for dependencies\nskill-seekers config                       # Configure API keys and settings\nskill-seekers workflows                    # List and apply workflow presets\nskill-seekers resume <job_id>              # Resume interrupted scraping\n\n# Advanced\nskill-seekers stream <source>              # Streaming ingestion\nskill-seekers update <directory>           # Incremental update\nskill-seekers multilang <directory>        # Multi-language support\n```\n\n## Testing Instructions\n\n### Test Structure\n- Unit tests: `tests/test_*.py` — test individual modules\n- Adaptor tests: `tests/test_adaptors/test_*_adaptor.py` — test platform adaptors\n- E2E tests: `tests/test_*_e2e.py` — end-to-end integration tests\n\n### Running Tests\n```bash\n# Fast test run (skip slow/integration tests)\npytest tests/ -v -m \"not slow and not integration\"\n\n# Full test suite\npytest tests/ -v\n\n# With coverage report\npytest tests/ --cov=src/skill_seekers --cov-report=term-missing\n\n# Specific test categories\npytest tests/ -v -m \"slow\"           # Only slow tests\npytest tests/ -v -m \"integration\"    # Only integration tests\npytest tests/ -v -m \"e2e\"            # Only E2E tests\n```\n\n### Test Fixtures\nTest fixtures are located in `tests/fixtures/` and include sample configs, HTML files, and mock data.\n\n## Git Workflow\n\n- **`main`** — production, protected\n- **`development`** — default PR target, active dev\n- Feature branches created from `development`\n\n## Pre-commit Checklist\n\n```bash\nruff check src/ tests/\nruff format --check src/ tests/\npytest tests/ -v -x   # stop on first failure\n```\n\nNever commit API keys. Use env vars: `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`, `GITHUB_TOKEN`.\n\n## CI/CD\n\nGitHub Actions (7 workflows in `.github/workflows/`):\n- **tests.yml** — ruff + mypy lint job, then pytest matrix (Ubuntu + macOS, Python 3.10-3.12) with Codecov upload\n- **release.yml** — tag-triggered: tests → version verification → PyPI publish via `uv build`\n- **test-vector-dbs.yml** — tests vector DB adaptors (weaviate, chroma, faiss, qdrant)\n- **docker-publish.yml** — multi-platform Docker builds (amd64, arm64) for CLI + MCP images\n- **quality-metrics.yml** — quality analysis with configurable threshold\n- **scheduled-updates.yml** — weekly skill updates for popular frameworks\n- **vector-db-export.yml** — weekly vector DB exports\n\n## Deployment\n\n### Docker\nMulti-stage Dockerfile with Python 3.12 slim base:\n```bash\n# Build CLI image\ndocker build -t skill-seekers:local -f Dockerfile .\n\n# Run CLI\ndocker run -v $(pwd)/output:/output skill-seekers:local create https://docs.example.com\n\n# Run MCP server\ndocker build -t skill-seekers-mcp:local -f Dockerfile.mcp .\ndocker run -p 8765:8765 skill-seekers-mcp:local\n```\n\n### MCP Server\nThe MCP server provides Model Context Protocol integration:\n```bash\n# Start FastMCP server\nskill-seekers-mcp\n\n# Or use the Python module\npython -m skill_seekers.mcp.server_fastmcp\n```\n\n## Security Considerations\n\n- **API Keys:** Never commit API keys to version control. Use environment variables or `.env` files (already in `.gitignore`)\n- **Docker:** Runs as non-root user (`skillseeker`, UID 1000)\n- **Dependencies:** Regular security updates via `pip audit` or `safety check`\n- **Sandboxing:** Video processing uses optional dependencies that can be heavy; install `[video-full]` only when needed\n\n## Additional Resources\n\n- **Website:** https://skillseekersweb.com/\n- **Documentation:** https://skillseekersweb.com/\n- **PyPI:** https://pypi.org/project/skill-seekers/\n- **Repository:** https://github.com/yusufkaraaslan/Skill_Seekers\n- **Config Browser:** https://skillseekersweb.com/\n- **Project Board:** https://github.com/users/yusufkaraaslan/projects/2\n","category":"root","tokens":3443}]}