## File: README.md # Skill Seekers English | [简体中文](README.zh-CN.md) | [日本語](README.ja.md) | [한국어](README.ko.md) | [Español](README.es.md) | [Français](README.fr.md) | [Deutsch](README.de.md) | [Português](README.pt-BR.md) | [Türkçe](README.tr.md) | [العربية](README.ar.md) | [हिन्दी](README.hi.md) | [Русский](README.ru.md) [](https://github.com/yusufkaraaslan/Skill_Seekers/releases) [](https://opensource.org/licenses/MIT) [](https://www.python.org/downloads/) [](https://modelcontextprotocol.io) [](tests/) [](https://pypi.org/project/skill-seekers/) [](https://pypi.org/project/skill-seekers/) [](https://skillseekersweb.com/) [](https://github.com/yusufkaraaslan/Skill_Seekers) [](https://pepy.tech/projects/skill-seekers) [](https://trendshift.io/repositories/18329) **🧠 The data layer for AI systems.** Skill Seekers turns documentation sites, GitHub repos, PDFs, videos, notebooks, wikis, and more — **18 source types** — into structured knowledge assets, ready to power AI Skills (Claude, Gemini, OpenAI), RAG pipelines (LangChain, LlamaIndex, Pinecone), and AI coding assistants (Cursor, Windsurf, Cline). Prepare once, export to **22 targets**. ## 💛 Sponsors ### Launch Partner [Atlas Cloud](https://www.atlascloud.ai/) — A full-modal, OpenAI-compatible AI inference platform. Skill Seekers supports it as a packaging/enhancement target via `--target atlas` with `ATLAS_API_KEY`. ### Silver Sponsors **[Become a sponsor](SPONSORSHIP.md)** · [GitHub Sponsors](https://github.com/sponsors/yusufkaraaslan) --- ## 🚀 Quick Start ```bash # 1. Install pip install skill-seekers # 2. Create a skill from any source skill-seekers create https://docs.djangoproject.com/ # 3. Package it for your AI platform skill-seekers package output/django --target claude ``` You now have `output/django-claude.zip`, ready to use. ```bash # Pick a different AI agent for enhancement (default: claude) skill-seekers create https://docs.djangoproject.com/ --agent kimi skill-seekers create https://docs.djangoproject.com/ --agent-cmd "my-custom-agent run" ``` ### 🛰️ AI-driven project scan Point `scan` at a project and an AI agent reads its manifests, README, Dockerfile/CI and sampled source imports — then emits one config per detected framework, plus a `-codebase.json` for your own code: ```bash skill-seekers scan ./my-react-app --out ./configs/scanned/ # → react.json, vite.json, tailwind.json, jest.json, my-react-app-codebase.json skill-seekers create ./configs/scanned/react.json ``` If a detection has no existing preset, the AI generates a fresh config; on exit you can optionally publish it back to the [community registry](https://github.com/yusufkaraaslan/skill-seekers-configs). ### All 18 source types ```bash skill-seekers create facebook/react # GitHub repository skill-seekers create ./my-project # Local codebase skill-seekers create manual.pdf # PDF skill-seekers create report.docx # Word skill-seekers create book.epub # EPUB skill-seekers create notebook.ipynb # Jupyter skill-seekers create openapi.yaml # OpenAPI/Swagger skill-seekers create presentation.pptx # PowerPoint skill-seekers create guide.adoc # AsciiDoc skill-seekers create page.html # Local HTML (or a whole dir) skill-seekers create feed.rss # RSS/Atom skill-seekers create curl.1 # Man page # Video (YouTube, Vimeo, or local — needs skill-seekers[video]) skill-seekers create --video-url https://www.youtube.com/watch?v=... --name mytutorial skill-seekers create --setup # auto-install GPU-aware visual deps skill-seekers create --space-key TEAM --name wiki # Confluence skill-seekers create --database-id ... --name docs # Notion skill-seekers create --chat-export-path ./slack-export --name team-chat # Slack/Discord ``` See the [Scraping Guide](docs/user-guide/02-scraping.md) for every source type and its options. --- ## 📦 Installation ```bash pip install skill-seekers # Core: scraping, GitHub, PDF, packaging pip install skill-seekers[all-llms] # + every LLM platform pip install skill-seekers[mcp] # + MCP server pip install skill-seekers[all] # Everything ``` **Not sure what you need?** Run the wizard: `skill-seekers-setup` **All installation extras** | Install | Adds | |---------|------| | `skill-seekers[gemini]` | Google Gemini support | | `skill-seekers[openai]` | OpenAI ChatGPT support | | `skill-seekers[all-llms]` | All LLM platforms | | `skill-seekers[mcp]` | MCP server for Claude Code, Cursor, etc. | | `skill-seekers[video]` | YouTube/Vimeo transcript & metadata extraction | | `skill-seekers[video-full]` | + Whisper transcription & visual frame extraction | | `skill-seekers[jupyter]` | Jupyter Notebook support | | `skill-seekers[pptx]` | PowerPoint support | | `skill-seekers[confluence]` | Confluence wiki support | | `skill-seekers[notion]` | Notion pages support | | `skill-seekers[rss]` | RSS/Atom feed support | | `skill-seekers[chat]` | Slack/Discord chat export support | | `skill-seekers[asciidoc]` | AsciiDoc support | | `skill-seekers[all]` | Everything | > **Video visual deps (GPU-aware):** after installing `skill-seekers[video-full]`, run `skill-seekers create --setup` to auto-detect your GPU and install the matching PyTorch variant + easyocr. **Prerequisites:** Python 3.10+, Git. New here? → **[Bulletproof Quick Start](docs/getting-started/BULLETPROOF_QUICKSTART.md)** 🎯 --- ## 📚 Documentation | I want to... | Read this | |--------------|-----------| | **Get started quickly** | [Quick Start](docs/getting-started/02-quick-start.md) — 3 commands to your first skill | | **Understand the concepts** | [Core Concepts](docs/user-guide/01-core-concepts.md) | | **Scrape sources** | [Scraping Guide](docs/user-guide/02-scraping.md) — all 18 source types | | **Enhance skills with AI** | [Enhancement Guide](docs/user-guide/03-enhancement.md) · [Enhancement Modes](docs/features/ENHANCEMENT_MODES.md) | | **Export skills** | [Packaging Guide](docs/user-guide/04-packaging.md) | | **Build workflows** | [Workflows](docs/user-guide/05-workflows.md) | | **Look up a command** | [CLI Reference](docs/reference/CLI_REFERENCE.md) — all 19 commands | | **Configure** | [Config Format](docs/reference/CONFIG_FORMAT.md) · [Environment Variables](docs/reference/ENVIRONMENT_VARIABLES.md) | | **Set up MCP** | [MCP Setup](docs/guides/MCP_SETUP.md) · [MCP Reference](docs/reference/MCP_REFERENCE.md) | | **Integrate with RAG / IDEs** | [LangChain](docs/integrations/LANGCHAIN.md) · [RAG Pipelines](docs/integrations/RAG_PIPELINES.md) · [Cursor](docs/integrations/CURSOR.md) · [Windsurf](docs/integrations/WINDSURF.md) · [Cline](docs/integrations/CLINE.md) | | **Handle huge doc sets** | [Large Documentation](docs/reference/LARGE_DOCUMENTATION.md) — 10K–40K+ pages | | **Understand the architecture** | [UML Architecture](docs/UML_ARCHITECTURE.md) — 14 diagrams | | **Fix a problem** | [Troubleshooting](docs/user-guide/06-troubleshooting.md) | **Complete documentation index:** [docs/README.md](docs/README.md) --- ## 🎯 What you get | Use case | Output | Powers | |----------|--------|--------| | **AI Skills** | Comprehensive `SKILL.md` + reference files | Claude Code, Gemini, GPT | | **RAG pipelines** | Chunked documents with rich metadata | LangChain, LlamaIndex, Haystack | | **Vector databases** | Pre-formatted data ready for upsert | Pinecone, Chroma, Weaviate, FAISS, Qdrant | | **AI coding assistants** | Context files your IDE AI reads automatically | Cursor, Windsurf, Cline, Continue.dev | ### Export targets (22) ```bash skill-seekers package output/react --target claude # → Claude Skill (ZIP + YAML) skill-seekers package output/react --target langchain # → LangChain Documents skill-seekers package output/react --target llama-index # → LlamaIndex TextNodes skill-seekers package output/react --target ibm-bob # → IBM Bob skill directory ``` **LLM platforms (12):** `claude` · `gemini` · `openai` · `minimax` · `opencode` · `kimi` · `deepseek` · `qwen` · `openrouter` · `together` · `fireworks` · `markdown` **RAG & vector (8):** `langchain` · `llama-index` · `haystack` · `chroma` · `faiss` · `weaviate` · `qdrant` · `pinecone` **Other (2):** `atlas` · `ibm-bob` See the [Feature Matrix](docs/reference/FEATURE_MATRIX.md) for per-platform support details. ### Why it matters - ⚡ **99% faster** — days of manual data prep → 15–45 minutes - 🎯 **Real skill quality** — 500+ line `SKILL.md` files with examples, patterns, and guides - 📊 **RAG-ready chunks** — smart chunking preserves code blocks and context - 🔄 **Multi-source** — combine docs + GitHub + PDFs + videos into one knowledge asset - 🌐 **One prep, every target** — export to 22 targets without re-scraping - ✅ **Battle-tested** — 3,900+ tests, 68 workflow presets, production-ready --- ## ✨ Key capabilities **Documentation scraping** — SPA discovery, llms.txt, smart categorization Three-layer discovery for JavaScript SPA sites (`sitemap.xml` → `llms.txt` → headless browser rendering), automatic `llms.txt` detection (10× faster when present), smart topic categorization, and a lenient HTML parser fallback so broken markup still scrapes. → [Scraping Guide](docs/user-guide/02-scraping.md) · [llms.txt Support](docs/reference/LLMS_TXT_SUPPORT.md) **GitHub & codebase analysis (C3.x)** — AST parsing, pattern detection, how-to guides Three-stream architecture: code analysis (AST, design patterns, tests), documentation (README, `docs/`, wiki), and community (issues, PRs, metadata). The C3.x pipeline adds 10 GoF pattern detectors across 9 languages, usage examples extracted from tests, AI-written how-to guides, config extraction, and architecture overviews. ```bash skill-seekers create ./my-project --preset quick # 1–2 min, surface level skill-seekers create ./my-project --preset standard # balanced (default) skill-seekers create ./my-project --preset comprehensive # deep, exhaustive ``` → [Pattern Detection](docs/features/PATTERN_DETECTION.md) · [How-To Guides](docs/features/HOW_TO_GUIDES.md) · [Test Example Extraction](docs/features/TEST_EXAMPLE_EXTRACTION.md) **AI enhancement** — API or local agents, 68 workflow presets Every AI call runs through one transport, in **API mode** (Anthropic, Google Gemini, OpenAI, Moonshot/Kimi, MiniMax) or **LOCAL mode** (Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents — no API costs). Control depth with `--enhance-level 0-3` and pick an agent with `--agent`. → [Enhancement Guide](docs/user-guide/03-enhancement.md) · [Enhancement Modes](docs/features/ENHANCEMENT_MODES.md) · [Multi-Agent Setup](docs/guides/MULTI_AGENT_SETUP.md) **Unified multi-source scraping** — combine many sources into one skill One config can pull documentation, GitHub, PDFs, videos, and more into a single knowledge asset, with conflict detection and pairwise synthesis across sources. → [Unified Scraping](docs/features/UNIFIED_SCRAPING.md) **Video extraction** — transcripts, frames, on-screen code YouTube, Vimeo, and local files. Three-tier transcript fallback (subtitles → YouTube transcript API → local Whisper), plus optional visual extraction that OCRs on-screen code from sampled frames. → [Video Guide](docs/VIDEO_GUIDE.md) **Quality, sync & scale** Quality scoring with a gate (`skill-seekers quality output/react/ --threshold 7`), provisional English readability metrics (informational — they never affect the score), doc-change detection with scheduled re-scrapes and notifications, streaming ingestion for very large doc sets, and incremental updates. → [Large Documentation](docs/reference/LARGE_DOCUMENTATION.md) · [Code Quality](docs/reference/CODE_QUALITY.md) --- ## 🔌 MCP Integration (40 tools) Skill Seekers ships an MCP server for Claude Code, Cursor, Windsurf, VS Code + Cline, and IntelliJ IDEA. ```bash # stdio mode (Claude Code, VS Code + Cline) python -m skill_seekers.mcp.server_fastmcp # HTTP mode (Cursor, Windsurf, IntelliJ) python -m skill_seekers.mcp.server_fastmcp --transport http --port 8765 ``` Then just ask your assistant: *"Package and upload the React skill."* → [MCP Setup](docs/guides/MCP_SETUP.md) · [MCP Reference](docs/reference/MCP_REFERENCE.md) · [HTTP Transport](docs/guides/HTTP_TRANSPORT.md) --- ## 🤖 Installing to AI agents Skills install automatically into **19 AI coding agents**: ```bash skill-seekers install-agent output/react/ --agent cursor skill-seekers install-agent output/react/ --agent all # every detected agent skill-seekers install-agent output/react/ --agent cursor --dry-run ``` | Agent | Path | Scope | |-------|------|-------| | Claude Code | `~/.claude/skills/` | Global | | Cursor | `.cursor/skills/` | Project | | VS Code / Copilot | `.github/skills/` | Project | | Amp | `~/.amp/skills/` | Global | | Goose | `~/.config/goose/skills/` | Global | | OpenCode | `~/.opencode/skills/` | Global | | Letta | `~/.letta/skills/` | Global | | Aide | `~/.aide/skills/` | Global | | Windsurf | `~/.windsurf/skills/` | Global | | Neovate | `~/.neovate/skills/` | Global | | Roo Code | `.roo/skills/` | Project | | Cline | `.cline/skills/` | Project | | Aider | `~/.aider/skills/` | Global | | Bolt | `.bolt/skills/` | Project | | Kilo Code | `.kilo/skills/` | Project | | Continue | `~/.continue/skills/` | Global | | Kimi Code | `~/.kimi/skills/` | Global | | IBM Bob | `.bob/skills/` | Project | ### Uploading to Claude ```bash export ANTHROPIC_API_KEY=sk-ant-... skill-seekers package output/react/ --upload # package + upload skill-seekers upload output/react.zip # upload an existing zip ``` No API key? Package it and upload `output/react.zip` manually at [claude.ai/skills](https://claude.ai/skills). → [Upload Guide](docs/guides/UPLOAD_GUIDE.md) --- ## ⚙️ How it works ```mermaid graph LR A[Documentation Website] --> B[Skill Seekers] B --> C[Scraper] B --> D[AI Enhancement] B --> E[Packager] C --> F[Organized References] D --> F F --> E E --> G[AI Skill .zip] G --> H[Upload to AI Platform] ``` 1. **Scrape** — extract every page (checking `llms.txt` first) 2. **Categorize** — organize content into topics (API, guides, tutorials, …) 3. **Enhance** — AI writes a comprehensive `SKILL.md` with examples 4. **Package** — bundle into a platform-ready artifact 5. **Upload** — ship it to your AI platform (optional) ### Architecture **8 core modules + 5 utility modules** (~200 classes): | Module | Purpose | |--------|---------| | **CLICore** | Git-style command dispatcher, source auto-detection | | **Scrapers** | 18 source-type extractors on a shared build layer | | **Adaptors** | 22 output platform formats behind one `SkillAdaptor` ABC | | **Analysis** | C3.x codebase pipeline, 10 GoF pattern detectors | | **Enhancement** | AI improvement via a single `AgentClient` transport | | **Packaging** | Package, upload, and install skills | | **MCP** | FastMCP server (40 tools, 10 tool modules) | | **Sync** | Doc change detection and notification | → [UML Architecture](docs/UML_ARCHITECTURE.md) · [API Reference](docs/reference/API_REFERENCE.md) · [Skill Architecture](docs/reference/SKILL_ARCHITECTURE.md) --- ## 🆕 New in v3.9.0 - **HTML parser fallback for broken markup** (#96) — severely malformed pages no longer scrape as empty; well-formed pages are byte-identical. - **Transient-failure retries** — the doc scraper (#97) and MCP `fetch_config` (#92) now retry connection blips and 5xx with backoff; 4xx still fails fast. - **Whisper transcription fallback** (#420) — local videos without subtitles finally get a real transcript. - **MiniMax image OCR + registry-driven multimodal providers** (#423) — providers declare their wire protocol and image capability; China-issued keys work against the right endpoint. - **Token-lean GitHub issue defaults** (#169) — GitHub skills no longer bundle full closed-issue history by default. - **Env-driven CORS across all three servers** (#422, #424) — no more wildcard origins with credentials. Full history: **[CHANGELOG.md](CHANGELOG.md)** --- ## 📈 Performance | Documentation size | Time | Output | |---|---|---| | Small (< 100 pages) | 5–10 min | ~2 MB | | Medium (100–500 pages) | 15–30 min | ~10 MB | | Large (500–2,000 pages) | 30–60 min | ~40 MB | | Huge (10K–40K+ pages) | Use `stream` | See [Large Documentation](docs/reference/LARGE_DOCUMENTATION.md) | --- ## 🐛 Troubleshooting ```bash skill-seekers doctor # diagnose installation & environment skill-seekers sync-config # detect config drift ``` Common issues and fixes: **[Troubleshooting Guide](docs/user-guide/06-troubleshooting.md)** · [TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) --- ## 🤝 Contributing Contributions are welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. - 📋 **[Development Roadmap & Tasks](https://github.com/users/yusufkaraaslan/projects/2)** — pick any task - 💬 **[Discussions](https://github.com/yusufkaraaslan/Skill_Seekers/discussions)** — questions and ideas - 🐛 **[Issues](https://github.com/yusufkaraaslan/Skill_Seekers/issues)** — bugs and feature requests --- ## 📝 License MIT — see [LICENSE](LICENSE). ## 🔒 Security [](https://mseep.ai/app/yusufkaraaslan-skill-seekers) --- ## 🌐 Ecosystem Skill Seekers is a multi-repo project: | Repository | Description | Links | |-----------|-------------|-------| | **[Skill_Seekers](https://github.com/yusufkaraaslan/Skill_Seekers)** | Core CLI & MCP server (this repo) | [PyPI](https://pypi.org/project/skill-seekers/) | | **[skillseekersweb](https://github.com/yusufkaraaslan/skillseekersweb)** | Website & documentation | [Live](https://skillseekersweb.com/) | | **[skill-seekers-configs](https://github.com/yusufkaraaslan/skill-seekers-configs)** | Community config repository | | | **[skill-seekers-action](https://github.com/yusufkaraaslan/skill-seekers-action)** | GitHub Action for CI/CD | | | **[skill-seekers-plugin](https://github.com/yusufkaraaslan/skill-seekers-plugin)** | Claude Code plugin | | | **[homebrew-skill-seekers](https://github.com/yusufkaraaslan/homebrew-skill-seekers)** | Homebrew tap for macOS | | > **Want to contribute?** The website and configs repos are great starting points for new contributors! --- ## File: api/README.md # Skill Seekers Config API FastAPI backend for discovering and downloading Skill Seekers configuration files. ## 🚀 Endpoints ### Base URL - **Production**: `https://skillseekersweb.com` - **Local**: `http://localhost:8000` ### Available Endpoints #### 1. **GET /** - API Information Returns API metadata and available endpoints. ```bash curl https://skillseekersweb.com/ ``` **Response:** ```json { "name": "Skill Seekers Config API", "version": "1.0.0", "endpoints": { "/api/configs": "List all available configs", "/api/configs/{name}": "Get specific config details", "/api/categories": "List all categories", "/docs": "API documentation" }, "repository": "https://github.com/yusufkaraaslan/Skill_Seekers", "website": "https://skillseekersweb.com" } ``` --- #### 2. **GET /api/configs** - List All Configs Returns list of all available configs with metadata. **Query Parameters:** - `category` (optional) - Filter by category (e.g., `web-frameworks`) - `tag` (optional) - Filter by tag (e.g., `javascript`) - `type` (optional) - Filter by type (`single-source` or `unified`) ```bash # Get all configs curl https://skillseekersweb.com/api/configs # Filter by category curl https://skillseekersweb.com/api/configs?category=web-frameworks # Filter by tag curl https://skillseekersweb.com/api/configs?tag=javascript # Filter by type curl https://skillseekersweb.com/api/configs?type=unified ``` **Response:** ```json { "version": "1.0.0", "total": 24, "filters": null, "configs": [ { "name": "react", "description": "React framework for building user interfaces...", "type": "single-source", "category": "web-frameworks", "tags": ["javascript", "frontend", "documentation"], "primary_source": "https://react.dev/", "max_pages": 300, "file_size": 1055, "last_updated": "2025-11-30T09:26:07+00:00", "download_url": "https://skillseekersweb.com/api/download/react.json", "config_file": "react.json" } ] } ``` --- #### 3. **GET /api/configs/{name}** - Get Specific Config Returns detailed information about a specific config. ```bash curl https://skillseekersweb.com/api/configs/react ``` **Response:** ```json { "name": "react", "description": "React framework for building user interfaces...", "type": "single-source", "category": "web-frameworks", "tags": ["javascript", "frontend", "documentation"], "primary_source": "https://react.dev/", "max_pages": 300, "file_size": 1055, "last_updated": "2025-11-30T09:26:07+00:00", "download_url": "https://skillseekersweb.com/api/download/react.json", "config_file": "react.json" } ``` --- #### 4. **GET /api/categories** - List Categories Returns all available categories with config counts. ```bash curl https://skillseekersweb.com/api/categories ``` **Response:** ```json { "total_categories": 5, "categories": { "web-frameworks": 7, "game-engines": 2, "devops": 2, "css-frameworks": 1, "uncategorized": 12 } } ``` --- #### 5. **GET /api/download/{config_name}** - Download Config File Downloads the actual config JSON file. ```bash # Download react config curl -O https://skillseekersweb.com/api/download/react.json # Download with just name (auto-adds .json) curl -O https://skillseekersweb.com/api/download/react ``` --- #### 6. **GET /health** - Health Check Health check endpoint for monitoring. ```bash curl https://skillseekersweb.com/health ``` **Response:** ```json { "status": "healthy", "service": "skill-seekers-api" } ``` --- #### 7. **GET /docs** - API Documentation Interactive OpenAPI documentation (Swagger UI). Visit: `https://skillseekersweb.com/docs` --- ## 📦 Metadata Fields Each config includes the following metadata: | Field | Type | Description | |-------|------|-------------| | `name` | string | Config identifier (e.g., "react") | | `description` | string | What the config is used for | | `type` | string | "single-source" or "unified" | | `category` | string | Auto-categorized (e.g., "web-frameworks") | | `tags` | array | Relevant tags (e.g., ["javascript", "frontend"]) | | `primary_source` | string | Main documentation URL or repo | | `max_pages` | int | Estimated page count for scraping | | `file_size` | int | Config file size in bytes | | `last_updated` | string | ISO 8601 date of last update | | `download_url` | string | Direct download link | | `config_file` | string | Filename (e.g., "react.json") | --- ## 🏗️ Categories Configs are auto-categorized into: - **web-frameworks** - Web development frameworks (React, Django, FastAPI, etc.) - **game-engines** - Game development engines (Godot, Unity, etc.) - **devops** - DevOps tools (Kubernetes, Ansible, etc.) - **css-frameworks** - CSS frameworks (Tailwind, etc.) - **development-tools** - Dev tools (Claude Code, etc.) - **gaming** - Gaming platforms (Steam, etc.) - **uncategorized** - Other configs --- ## 🏷️ Tags Common tags include: - **Language**: `javascript`, `python`, `php` - **Domain**: `frontend`, `backend`, `devops`, `game-development` - **Type**: `documentation`, `github`, `pdf`, `multi-source` - **Tech**: `css`, `testing`, `api` --- ## 🚀 Local Development ### Setup ```bash # Install dependencies cd api pip install -r requirements.txt # Run server python main.py ``` API will be available at `http://localhost:8000` ### Testing ```bash # Test health check curl http://localhost:8000/health # List all configs curl http://localhost:8000/api/configs # Get specific config curl http://localhost:8000/api/configs/react # Download config curl -O http://localhost:8000/api/download/react.json ``` --- ## 📝 Deployment ### Render This API is configured for Render deployment via `render.yaml`. 1. Push to GitHub 2. Connect repository to Render 3. Render auto-deploys from `render.yaml` 4. Configure custom domain: `skillseekersweb.com` --- ## 🔗 Links - **API Documentation**: https://skillseekersweb.com/docs - **GitHub Repository**: https://github.com/yusufkaraaslan/Skill_Seekers - **Main Project**: https://github.com/yusufkaraaslan/Skill_Seekers#readme --- ## File: distribution/smithery/README.md # Skill Seekers — Smithery MCP Registry Publishing guide for the Skill Seekers MCP server on [Smithery](https://smithery.ai). ## Status - **Namespace created:** `yusufkaraaslan` - **Server created:** `yusufkaraaslan/skill-seekers` - **Server page:** https://smithery.ai/servers/yusufkaraaslan/skill-seekers - **Release status:** Needs re-publish (initial release failed — Smithery couldn't scan GitHub URL as MCP endpoint) ## Publishing Smithery requires a live, scannable MCP HTTP endpoint for URL-based publishing. Two options: ### Option A: Publish via Web UI (Recommended) 1. Go to https://smithery.ai/servers/yusufkaraaslan/skill-seekers/releases 2. The server already exists — create a new release 3. For the "Local" tab: follow the prompts to publish as a stdio server 4. For the "URL" tab: provide a hosted HTTP endpoint URL ### Option B: Deploy HTTP endpoint first, then publish via CLI 1. Deploy the MCP server on Render/Railway/Fly.io: ```bash # Using existing Dockerfile.mcp docker build -f Dockerfile.mcp -t skill-seekers-mcp . # Deploy to your hosting provider ``` 2. Publish the live URL: ```bash npx @smithery/cli@latest auth login npx @smithery/cli@latest mcp publish "https://your-deployed-url/mcp" \ -n yusufkaraaslan/skill-seekers ``` ### CLI Authentication (already done) ```bash # Install via npx (no global install needed) npx @smithery/cli@latest auth login npx @smithery/cli@latest namespace show # Should show: yusufkaraaslan ``` ### After Publishing Update the server page with metadata: **Display name:** Skill Seekers — AI Skill & RAG Toolkit **Description:** > Transform 18 source types into AI-ready skills and RAG knowledge. Ingest documentation sites, GitHub repos, PDFs, Jupyter notebooks, videos, Confluence, Notion, Slack/Discord exports, and more. Package for 21+ LLM platforms including Claude, GPT, Gemini, LangChain, LlamaIndex, and vector databases. **Tags:** `ai`, `rag`, `documentation`, `skills`, `preprocessing`, `mcp`, `knowledge-base`, `vector-database` ## User Installation Once published, users can add the server to their MCP client: ```bash # Via Smithery CLI (adds to Claude Desktop, Cursor, etc.) smithery mcp add yusufkaraaslan/skill-seekers --client claude # Or configure manually — users need skill-seekers installed: pip install skill-seekers[mcp] ``` ### Manual MCP Configuration For clients that use JSON config (Claude Desktop, Claude Code, Cursor): ```json { "mcpServers": { "skill-seekers": { "command": "python", "args": ["-m", "skill_seekers.mcp.server_fastmcp"] } } } ``` ## Available Tools (40) | Category | Tools | Description | |----------|-------|-------------| | Config | 3 | Generate, list, validate scraping configs | | Sync | 1 | Sync config URLs against live docs | | Scraping | 11 | Scrape docs, GitHub, PDF, video, codebase, generic (10 types) | | Packaging | 4 | Package, upload, enhance, install skills | | Splitting | 2 | Split large configs, generate routers | | Sources | 5 | Fetch, submit, manage config sources | | Vector DB | 4 | Export to Weaviate, Chroma, FAISS, Qdrant | | Workflows | 5 | List, get, create, update, delete workflows | | Quality | 3 | Quality checks, estimate scope, resume operations | | Agent Install | 2 | Install agent, extract test examples | ## Maintenance - Update description/tags on major releases - No code changes needed — users always get the latest via `pip install` ## Notes - Smithery CLI v4.7.0 removed the `--transport stdio` flag from the docs - The CLI `publish` command only supports URL-based (external) publishing - For local/stdio servers, use the web UI at smithery.ai/servers/new - The namespace and server entity are already created; only the release needs to succeed --- ## File: distribution/github-action/README.md # Skill Seekers GitHub Action Transform documentation, GitHub repos, PDFs, videos, and 13 other source types into AI-ready skills and RAG knowledge — directly in your CI/CD pipeline. ## Quick Start ```yaml - uses: yusufkaraaslan/skill-seekers-action@v3 with: source: 'https://react.dev' ``` ## Inputs | Input | Required | Default | Description | |-------|----------|---------|-------------| | `source` | Yes | — | Source URL, file path, or `owner/repo` | | `command` | No | `create` | Command: `create`, `scan`, `doctor`, `enhance`, `enhance-status`, `package`, `upload`, `install`, `install-agent`, `estimate`, `extract-test-examples`, `resume`, `quality`, `config`, `workflows`, `sync-config`, `stream`, `update`, `multilang` | | `target` | No | `claude` | Target platform: `claude`, `openai`, `gemini`, `langchain`, `llama-index`, `markdown` | | `config` | No | — | Path to JSON config file | | `output-dir` | No | `output` | Output directory | | `extra-args` | No | — | Additional CLI arguments | ## Outputs | Output | Description | |--------|-------------| | `skill-dir` | Path to the generated skill directory | | `skill-name` | Name of the generated skill | ## Examples ### Auto-update documentation skill weekly ```yaml name: Update AI Skills on: schedule: - cron: '0 6 * * 1' # Every Monday 6am UTC workflow_dispatch: jobs: update-skills: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: yusufkaraaslan/skill-seekers-action@v3 with: source: 'https://react.dev' target: 'langchain' - uses: actions/upload-artifact@v4 with: name: react-skill path: output/ ``` ### Generate skill from GitHub repo ```yaml - uses: yusufkaraaslan/skill-seekers-action@v3 with: source: 'pallets/flask' command: 'create' target: 'claude' ``` ### Process PDF documentation ```yaml - uses: actions/checkout@v4 - uses: yusufkaraaslan/skill-seekers-action@v3 with: source: 'docs/api-reference.pdf' command: 'create' ``` ### Unified multi-source build with config ```yaml - uses: actions/checkout@v4 - uses: yusufkaraaslan/skill-seekers-action@v3 with: config: 'configs/my-project.json' command: 'create' target: 'openai' ``` ### Commit generated skill back to repo ```yaml - uses: actions/checkout@v4 - uses: yusufkaraaslan/skill-seekers-action@v3 id: generate with: source: 'https://fastapi.tiangolo.com' - name: Commit skill run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" git add output/ git diff --staged --quiet || git commit -m "Update AI skill: ${{ steps.generate.outputs.skill-name }}" git push ``` ## Environment Variables Pass API keys as environment variables for AI-enhanced skills: ```yaml env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` ## Supported Source Types | Type | Example Source | |------|---------------| | Documentation (web) | `https://react.dev` | | GitHub repo | `pallets/flask` or `https://github.com/pallets/flask` | | PDF | `docs/manual.pdf` | | Video | `https://youtube.com/watch?v=...` | | Local codebase | `./src` | | Jupyter Notebook | `analysis.ipynb` | | OpenAPI/Swagger | `openapi.yaml` | | Word (.docx) | `docs/guide.docx` | | EPUB | `book.epub` | | PowerPoint | `slides.pptx` | | AsciiDoc | `docs/guide.adoc` | | HTML | `page.html` | | RSS/Atom | `feed.rss` | | Man pages | `tool.1` | | Confluence | Via config file | | Notion | Via config file | | Chat (Slack/Discord) | Via config file | ## License MIT --- ## File: distribution/claude-plugin/README.md # Skill Seekers — Claude Code Plugin Transform 18 source types into AI-ready skills and RAG knowledge, directly from Claude Code. ## Installation ### From the Official Plugin Directory ``` /plugin install skill-seekers@claude-plugin-directory ``` Or browse for it in `/plugin > Discover`. ### Local Installation (for development) ```bash claude --plugin-dir ./path/to/skill-seekers-plugin ``` ### Prerequisites The plugin requires `skill-seekers` to be installed: ```bash pip install skill-seekers[mcp] ``` ## What's Included ### MCP Server (40 tools) The plugin bundles the Skill Seekers MCP server providing tools for: - Scraping documentation, GitHub repos, PDFs, videos, and 13 other source types - Packaging skills for 21+ LLM platforms - Exporting to vector databases (Weaviate, Chroma, FAISS, Qdrant) - Managing configs, workflows, and sources ### Slash Commands | Command | Description | |---------|-------------| | `/skill-seekers:create-skill ` | Create a skill from any source (auto-detects type) | | `/skill-seekers:sync-config ` | Sync config URLs against live docs | | `/skill-seekers:install-skill ` | End-to-end: fetch, scrape, enhance, package, install | ### Agent Skill The **skill-builder** skill is automatically available to Claude. It detects source types and uses the appropriate MCP tools to build skills autonomously. ## Usage Examples ``` # Create a skill from a documentation site /skill-seekers:create-skill https://react.dev # Create from a GitHub repo, targeting LangChain /skill-seekers:create-skill pallets/flask --target langchain # Full install workflow with AI enhancement /skill-seekers:install-skill https://fastapi.tiangolo.com --enhance # Sync an existing config /skill-seekers:sync-config react ``` Or just ask Claude naturally: > "Create an AI skill from the React documentation" > "Scrape the Flask GitHub repo and package it for OpenAI" > "Export my skill to a Chroma vector database" The skill-builder agent skill will automatically detect the intent and use the right tools. ## Remote MCP Alternative By default, the plugin runs the MCP server locally via `python -m skill_seekers.mcp.server_fastmcp`. To use a remote server instead, edit `.mcp.json`: ```json { "skill-seekers": { "type": "http", "url": "https://your-hosted-server.com/mcp" } } ``` ## Supported Source Types Documentation (web), GitHub repos, PDFs, Word docs, EPUBs, videos, local codebases, Jupyter notebooks, HTML files, OpenAPI specs, AsciiDoc, PowerPoint, RSS/Atom feeds, man pages, Confluence, Notion, Slack/Discord exports. ## License MIT — https://github.com/yusufkaraaslan/Skill_Seekers --- ## File: docs/agents/research/2026-03-14-epub-input-support-affected-files.md --- date: 2026-03-14T12:54:24.700367+00:00 git_commit: 7c90a4b9c9bccac8341b0769550d77aae3b4e524 branch: development topic: "What files would be affected to add .epub support for input" tags: [research, codebase, epub, input-format, scraper] status: complete --- > 🗄️ **Historical — completed** — This research informed the EPUB implementation that is now live in Skill Seekers v3.6.0. # Research: What files would be affected to add .epub support for input ## Research Question What files would be affected to add .epub support for input. ## Summary Adding `.epub` input support follows an established pattern already used for PDF and Word (.docx) formats. The codebase has a consistent multi-layer architecture for document input formats: source detection, argument definitions, parser registration, create command routing, standalone scraper module, and tests. Based on analysis of the existing PDF and Word implementations, **16 existing files would need modification** and **4 new files would need to be created**. ## Detailed Findings ### New Files to Create (4 files) | File | Purpose | |------|---------| | `src/skill_seekers/cli/epub_scraper.py` | Core EPUB extraction and skill building logic (analog: `word_scraper.py` at ~750 lines) | | `src/skill_seekers/cli/arguments/epub.py` | EPUB-specific argument definitions (analog: `arguments/word.py`) | | `src/skill_seekers/cli/parsers/epub_parser.py` | Subcommand parser class (analog: `parsers/word_parser.py`) | | `tests/test_epub_scraper.py` | Test suite (analog: `test_word_scraper.py` at ~750 lines, 130+ tests) | ### Existing Files to Modify (16 files) #### 1. Source Detection Layer **`src/skill_seekers/cli/source_detector.py`** (3 locations) - **`SourceDetector.detect()`** (line ~60): Add `.epub` extension check, following the `.docx` pattern at line 63-64: ```python if source.endswith(".epub"): return cls._detect_epub(source) ``` - **New method `_detect_epub()`**: Add detection method (following `_detect_word()` at lines 124-129): ```python @classmethod def _detect_epub(cls, source: str) -> SourceInfo: name = os.path.splitext(os.path.basename(source))[0] return SourceInfo( type="epub", parsed={"file_path": source}, suggested_name=name, raw_input=source ) ``` - **`validate_source()`** (line ~250): Add epub validation block (following the word block at lines 273-278) - **Error message** (line ~94): Add EPUB example to the `ValueError` help text #### 2. CLI Dispatcher **`src/skill_seekers/cli/main.py`** (2 locations) - **`COMMAND_MODULES` dict** (line ~46): Add epub entry: ```python "epub": "skill_seekers.cli.epub_scraper", ``` - **Module docstring** (line ~1): Add `epub` to the commands list #### 3. Create Command Routing **`src/skill_seekers/cli/create_command.py`** (3 locations) - **`_route_to_scraper()`** (line ~121): Add `elif self.source_info.type == "epub":` routing case - **New `_route_epub()` method**: Following the `_route_word()` pattern at lines 331-352: ```python def _route_epub(self) -> int: from skill_seekers.cli import epub_scraper argv = ["epub_scraper"] file_path = self.source_info.parsed["file_path"] argv.extend(["--epub", file_path]) self._add_common_args(argv) # epub-specific args here ... ``` - **`main()` epilog** (line ~537): Add EPUB example and source auto-detection entry - **Progressive help** (line ~590): Add `--help-epub` flag and handler block #### 4. Argument Definitions **`src/skill_seekers/cli/arguments/create.py`** (4 locations) - **New `EPUB_ARGUMENTS` dict** (~line 401): Define epub-specific arguments (e.g., `--epub` file path flag), following the `WORD_ARGUMENTS` pattern at lines 402-411 - **`get_source_specific_arguments()`** (line 595): Add `"epub": EPUB_ARGUMENTS` to the `source_args` dict - **`add_create_arguments()`** (line 676): Add epub mode block: ```python if mode in ["epub", "all"]: for arg_name, arg_def in EPUB_ARGUMENTS.items(): parser.add_argument(*arg_def["flags"], **arg_def["kwargs"]) ``` #### 5. Parser Registration **`src/skill_seekers/cli/parsers/__init__.py`** (2 locations) - **Import** (line ~15): Add `from .epub_parser import EpubParser` - **`PARSERS` list** (line ~46): Add `EpubParser()` entry (near `WordParser()` and `PDFParser()`) #### 6. Package Configuration **`pyproject.toml`** (3 locations) - **`[project.optional-dependencies]`** (line ~111): Add `epub` optional dependency group: ```toml epub = [ "ebooklib>=0.18", ] ``` - **`all` optional dependency group** (line ~178): Add epub dependency to the combined `all` group - **`[project.scripts]`** (line ~224): Add standalone entry point: ```toml skill-seekers-epub = "skill_seekers.cli.epub_scraper:main" ``` #### 7. Argument Commons **`src/skill_seekers/cli/arguments/common.py`** - No changes strictly required, but `add_all_standard_arguments()` is called by the new `arguments/epub.py` (no modification needed — it's used as-is) #### 8. Documentation / Configuration **`CLAUDE.md`** (2 locations) - **Commands section**: Add `epub` to the list of subcommands - **Key source files table**: Add `epub_scraper.py` entry **`CONTRIBUTING.md`** — Potentially update with epub format mention **`CHANGELOG.md`** — New feature entry ### Files NOT Affected These files do **not** need changes: - **`unified_scraper.py`** — Multi-source configs could add epub support later but it's not required for basic input support - **Platform adaptors** (`adaptors/*.py`) — Adaptors work on the output side (packaging), not input - **Enhancement system** (`enhance_skill.py`, `enhance_skill_local.py`) — Works generically on SKILL.md - **MCP server** (`mcp/server_fastmcp.py`) — Operates on completed skills - **`pdf_extractor_poc.py`** — PDF-specific extraction; epub needs its own extractor ## Code References ### Pattern to Follow (Word .docx implementation) - `src/skill_seekers/cli/word_scraper.py:1-750` — Full scraper with `WordToSkillConverter` class - `src/skill_seekers/cli/arguments/word.py:1-75` — Argument definitions with `add_word_arguments()` - `src/skill_seekers/cli/parsers/word_parser.py:1-33` — Parser class extending `SubcommandParser` - `tests/test_word_scraper.py:1-750` — Comprehensive test suite with 130+ tests ### Key Integration Points - `src/skill_seekers/cli/source_detector.py:57-65` — File extension detection order - `src/skill_seekers/cli/source_detector.py:124-129` — `_detect_word()` method (template for `_detect_epub()`) - `src/skill_seekers/cli/create_command.py:121-143` — `_route_to_scraper()` dispatch - `src/skill_seekers/cli/create_command.py:331-352` — `_route_word()` (template for `_route_epub()`) - `src/skill_seekers/cli/arguments/create.py:401-411` — `WORD_ARGUMENTS` dict (template) - `src/skill_seekers/cli/arguments/create.py:595-604` — `get_source_specific_arguments()` mapping - `src/skill_seekers/cli/arguments/create.py:676-678` — `add_create_arguments()` mode handling - `src/skill_seekers/cli/parsers/__init__.py:35-59` — `PARSERS` registry list - `src/skill_seekers/cli/main.py:46-70` — `COMMAND_MODULES` dict - `pyproject.toml:111-115` — Optional dependency group pattern (docx) - `pyproject.toml:213-246` — Script entry points ### Data Flow Architecture The epub scraper would follow the same three-step pipeline as Word/PDF: 1. **Extract** — Parse `.epub` file → sections with text, headings, code, images → save to `output/{name}_extracted.json` 2. **Categorize** — Group sections by chapters/keywords 3. **Build** — Generate `SKILL.md`, `references/*.md`, `references/index.md`, `assets/` The intermediate JSON format uses the same structure as Word/PDF: ```python { "source_file": str, "metadata": {"title", "author", "created", ...}, "total_sections": int, "total_code_blocks": int, "total_images": int, "languages_detected": {str: int}, "pages": [ # sections { "section_number": int, "heading": str, "text": str, "code_samples": [...], "images": [...], "headings": [...] } ] } ``` ## Architecture Documentation ### Document Input Format Pattern Each input format follows a consistent architecture: ``` [source_detector.py] → detect type by extension ↓ [create_command.py] → route to scraper ↓ [{format}_scraper.py] → extract → categorize → build skill ↓ [output/{name}/] → SKILL.md + references/ + assets/ ``` Supporting files per format: - `arguments/{format}.py` — CLI argument definitions - `parsers/{format}_parser.py` — Subcommand parser class - `tests/test_{format}_scraper.py` — Test suite ### Dependency Guard Pattern The Word scraper uses an optional dependency guard that epub should replicate: ```python try: import ebooklib from ebooklib import epub EPUB_AVAILABLE = True except ImportError: EPUB_AVAILABLE = False def _check_epub_deps(): if not EPUB_AVAILABLE: raise RuntimeError( "ebooklib is required for EPUB support.\n" 'Install with: pip install "skill-seekers[epub]"\n' "Or: pip install ebooklib" ) ``` ## Summary Table | Category | Files | Action | |----------|-------|--------| | New files | 4 | Create from scratch | | Source detection | 1 | Add epub detection + validation | | CLI dispatcher | 1 | Add command module mapping | | Create command | 1 | Add routing + help + examples | | Arguments | 1 | Add EPUB_ARGUMENTS + register in helpers | | Parser registry | 1 | Import + register EpubParser | | Package config | 1 | Add deps + entry point | | Documentation | 2+ | Update CLAUDE.md, CHANGELOG | | **Total** | **12+ modified, 4 new** | | ## Open Questions - Should epub support reuse any of the existing HTML parsing from `word_scraper.py` (which uses mammoth to convert to HTML then parses with BeautifulSoup)? EPUB internally contains XHTML files, so BeautifulSoup parsing would be directly applicable. - Should the epub scraper support DRM-protected files, or only DRM-free epub files? - Should epub-specific arguments include options like `--chapter-range` (similar to PDF's `--pages`)? --- ## File: docs/agents/plans/2026-03-14-epub-input-support.md --- date: 2026-03-14T19:30:35.172407+00:00 git_commit: 7c90a4b9c9bccac8341b0769550d77aae3b4e524 branch: development topic: "Add EPUB Input Support" tags: [plan, epub, scraper, input-format] status: complete --- > 🗄️ **Historical — completed** — This plan was fully executed. EPUB input support is live in Skill Seekers v3.6.0 (`epub_scraper.py`, `arguments/epub.py`, `parsers/epub_parser.py`, 107+ tests). # Add EPUB Input Support — Implementation Plan ## Overview Add `.epub` as an input format for Skill Seekers, enabling `skill-seekers create book.epub` and `skill-seekers epub --epub book.epub`. Follows the established Word/PDF scraper pattern: source detection → routing → extraction → categorize → build skill. **Authoritative reference**: [W3C EPUB 3.3 Specification](https://www.w3.org/TR/epub-33/) (also covers EPUB 2 backward compatibility). ## Current State Analysis The codebase has a consistent multi-layer architecture for document input formats. PDF and Word (.docx) serve as direct analogs. The Word scraper (`word_scraper.py`) is the closest pattern match since both Word and EPUB produce HTML/XHTML that is parsed with BeautifulSoup. ### Key Discoveries: - Word scraper converts `.docx` → HTML (via mammoth) → BeautifulSoup parse → intermediate JSON → SKILL.md (`word_scraper.py:96-235`) - EPUB files contain XHTML natively (per W3C spec §5), so the mammoth conversion step is unnecessary — BeautifulSoup can parse EPUB XHTML content directly - Source detection uses file extension matching (`source_detector.py:57-65`) - Optional dependencies use a guard pattern with `try/except ImportError` and a `_check_*_deps()` function (`word_scraper.py:21-40`) - The `ebooklib` library (v0.18+) provides `epub.read_epub()` returning an `EpubBook` with spine iteration, metadata access via `get_metadata('DC', key)`, and item content via `get_content()`/`get_body_content()` - ebooklib has a known bug: EPUB 3 files read TOC from NCX instead of NAV (issue #200); workaround: `options={"ignore_ncx": True}` - ebooklib loads entire EPUB into memory — acceptable for typical books but relevant for edge cases ## Desired End State Running `skill-seekers create book.epub` produces: ``` output/book/ ├── SKILL.md # Main skill file with metadata, concepts, code examples ├── references/ │ ├── index.md # Category index with statistics │ └── book.md # Chapter content (or multiple files if categorized) ├── scripts/ └── assets/ └── *.png|*.jpg # Extracted images ``` ### CLI Output Mockup ``` $ skill-seekers create programming-rust.epub ℹ️ Detected source type: epub ℹ️ Routing to epub scraper... 🔍 Extracting from EPUB: programming-rust.epub Title: Programming Rust, 2nd Edition Author: Jim Blandy, Jason Orendorff Language: en Chapters: 23 (spine items) 📄 Processing chapters... Chapter 1/23: Why Rust? (2 sections, 1 code block) Chapter 2/23: A Tour of Rust (5 sections, 12 code blocks) ... Chapter 23/23: Macros (4 sections, 8 code blocks) 📊 Extraction complete: Sections: 142 Code blocks: 287 (Rust: 245, Shell: 28, TOML: 14) Images: 34 Tables: 12 💾 Saved extracted data to: output/programming-rust_extracted.json 📋 Categorizing content... ✅ Created 1 category (single EPUB source) - programming-rust: 142 sections 📝 Generating reference files... Generated: output/programming-rust/references/programming-rust.md Generated: output/programming-rust/references/index.md ✅ Skill built successfully: output/programming-rust/ 📦 Next step: Package with: skill-seekers package output/programming-rust/ ``` ### Verification: - [x] `skill-seekers create book.epub` produces valid output directory - [x] `skill-seekers create book.epub --name mybook` works standalone - [x] `skill-seekers create book.epub --dry-run` shows config without processing - [x] All ~2,540+ existing tests still pass (982 passed, 1 pre-existing failure) - [x] New test suite has 100+ tests covering happy path, errors, and edge cases (107 tests, 14 classes) ## What We're NOT Doing - DRM decryption (detect and error gracefully with clear message) - EPUB writing/creation (read-only) - Media overlay / audio / video extraction (ignore gracefully) - Fixed-layout OCR (detect and warn; extract whatever text exists in XHTML) - `--chapter-range` flag (can be added later) - Unified scraper (`unified_scraper.py`) EPUB support (separate future task) - MCP tool for EPUB (separate future task) ## Implementation Approach Follow the Word scraper pattern exactly, with EPUB-specific extraction logic: 1. **Phase 1**: Core `epub_scraper.py` — the `EpubToSkillConverter` class 2. **Phase 2**: CLI integration — source detection, arguments, parser, routing, entry points 3. **Phase 3**: Comprehensive test suite — 100+ tests across 11 test classes 4. **Phase 4**: Documentation updates --- ## Phase 1: Core EPUB Scraper ### Overview Create `epub_scraper.py` with `EpubToSkillConverter` class following the Word scraper pattern. This is the bulk of new code. ### Changes Required: #### [x] 1. Optional dependency in pyproject.toml **File**: `pyproject.toml` **Changes**: Add `epub` optional dependency group and include in `all` group ```toml # After the docx group (~line 115) # EPUB (.epub) support epub = [ "ebooklib>=0.18", ] ``` Add `"ebooklib>=0.18",` to the `all` group (~line 178). #### [x] 2. Create `src/skill_seekers/cli/epub_scraper.py` **File**: `src/skill_seekers/cli/epub_scraper.py` (new) **Changes**: Full EPUB scraper module **Structure** (following `word_scraper.py` pattern): ```python """ EPUB Documentation to Skill Converter Converts EPUB e-books into skills. Uses ebooklib for EPUB parsing, BeautifulSoup for XHTML content extraction. Usage: skill-seekers create book.epub --name myskill skill-seekers create book_extracted.json """ import argparse import json import logging import os import re import sys from pathlib import Path # Optional dependency guard try: import ebooklib from ebooklib import epub EPUB_AVAILABLE = True except ImportError: EPUB_AVAILABLE = False # BeautifulSoup is a core dependency (always available) from bs4 import BeautifulSoup, Comment logger = logging.getLogger(__name__) def _check_epub_deps(): """Raise RuntimeError if ebooklib is not installed.""" if not EPUB_AVAILABLE: raise RuntimeError( "ebooklib is required for EPUB support.\n" 'Install with: pip install "skill-seekers[epub]"\n' "Or: pip install ebooklib" ) def infer_description_from_epub(metadata: dict | None = None, name: str = "") -> str: """Infer skill description from EPUB metadata.""" if metadata: if metadata.get("description") and len(metadata["description"]) > 20: desc = metadata["description"].strip() if len(desc) > 150: desc = desc[:147] + "..." return f"Use when {desc.lower()}" if metadata.get("title") and len(metadata["title"]) > 10: return f"Use when working with {metadata['title'].lower()}" return ( f"Use when referencing {name} documentation" if name else "Use when referencing this documentation" ) ``` **`EpubToSkillConverter` class methods:** ```python class EpubToSkillConverter: def __init__(self, config: dict): self.config = config self.name = config["name"] self.epub_path = config.get("epub_path", "") self.description = config.get( "description", f"Use when referencing {self.name} documentation" ) self.skill_dir = f"output/{self.name}" self.data_file = f"output/{self.name}_extracted.json" self.categories = config.get("categories", {}) self.extracted_data = None def extract_epub(self) -> bool: """Extract content from EPUB file. Workflow: 1. Check dependencies (ebooklib) 2. Detect DRM via META-INF/encryption.xml (fail fast) 3. Read EPUB via ebooklib with ignore_ncx=True (EPUB 3 TOC bug workaround) 4. Extract Dublin Core metadata (title, creator, language, publisher, date, description, subject) 5. Iterate spine items in reading order 6. For each ITEM_DOCUMENT: parse XHTML with BeautifulSoup 7. Split by h1/h2 heading boundaries into sections 8. Extract code blocks from / elements 9. Extract images from EpubImage items 10. Detect code languages via LanguageDetector 11. Save intermediate JSON to {name}_extracted.json Returns True on success. Raises RuntimeError for DRM-protected files. Raises FileNotFoundError for missing files. Raises ValueError for invalid EPUB files. """ ``` **DRM detection** (per W3C spec §4.2.6.3.2): ```python def _detect_drm(self, book) -> bool: """Detect DRM by checking for encryption.xml with non-font-obfuscation entries. Per W3C EPUB 3.3 spec: encryption.xml is present when resources are encrypted. Font obfuscation (IDPF algorithm http://www.idpf.org/2008/embedding or Adobe algorithm http://ns.adobe.com/pdf/enc#RC) is NOT DRM — only font mangling. Actual DRM uses algorithms like: - Adobe ADEPT: http://ns.adobe.com/adept namespace - Apple FairPlay: http://itunes.apple.com/dataenc - Readium LCP: http://readium.org/2014/01/lcp """ ``` **Metadata extraction** (per W3C spec §5.2, Dublin Core): ```python def _extract_metadata(self, book) -> dict: """Extract Dublin Core metadata from EPUB. Per W3C EPUB 3.3 spec: required elements are dc:identifier, dc:title, dc:language. Optional: dc:creator, dc:contributor, dc:date, dc:description, dc:publisher, dc:subject, dc:rights, dc:type, dc:coverage, dc:source, dc:relation, dc:format. ebooklib API: book.get_metadata('DC', key) returns list of (value, attrs) tuples. """ def _get_one(key): data = book.get_metadata('DC', key) return data[0][0] if data else None def _get_list(key): data = book.get_metadata('DC', key) return [x[0] for x in data] if data else [] return { "title": _get_one('title') or "Untitled", "author": ", ".join(_get_list('creator')) or None, "language": _get_one('language') or "en", "publisher": _get_one('publisher'), "date": _get_one('date'), "description": _get_one('description'), "subject": ", ".join(_get_list('subject')) or None, "rights": _get_one('rights'), "identifier": _get_one('identifier'), } ``` **Content extraction** (per W3C spec §5 — XHTML Content Documents use XML serialization of HTML5): ```python def _extract_spine_content(self, book) -> list[dict]: """Extract content from spine items in reading order. Per W3C EPUB 3.3 spec §3.4.8: spine defines ordered list of content documents. Linear="yes" (default) items form the primary reading order. Linear="no" items are auxiliary (footnotes, glossary). Per spec §5: XHTML content documents use XML syntax of HTML5. Parse with BeautifulSoup, split by h1/h2 heading boundaries. """ sections = [] section_number = 0 for item_id, linear in book.spine: item = book.get_item_with_id(item_id) if not item or item.get_type() != ebooklib.ITEM_DOCUMENT: continue soup = BeautifulSoup(item.get_content(), 'html.parser') # Remove scripts, styles, comments (not useful for text extraction) for tag in soup(['script', 'style']): tag.decompose() for comment in soup.find_all(string=lambda t: isinstance(t, Comment)): comment.extract() body = soup.find('body') if not body: continue # Split by h1/h2 heading boundaries (same as word_scraper) # Each heading starts a new section ... ``` **Image extraction** (per W3C spec §3.3 — core media types include JPEG, PNG, GIF, WebP, SVG): ```python def _extract_images(self, book) -> list[dict]: """Extract images from EPUB manifest. Per W3C EPUB 3.3 spec §3.3: core image media types are image/gif, image/jpeg, image/png, image/svg+xml, image/webp. ebooklib API: book.get_items_of_type(ebooklib.ITEM_IMAGE) returns EpubImage items with get_content() (bytes) and media_type. SVG images (ITEM_VECTOR) handled separately. """ ``` **The remaining methods** (`categorize_content`, `build_skill`, `_generate_reference_file`, `_generate_index`, `_generate_skill_md`, `_format_key_concepts`, `_format_patterns_from_content`, `_sanitize_filename`) follow the Word scraper pattern exactly — they operate on the same intermediate JSON structure. **`main()` function** (following `word_scraper.py:923-1059`): ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Success Criteria: #### Automated Verification: - [x] `ruff check src/skill_seekers/cli/epub_scraper.py` passes - [x] `ruff format --check src/skill_seekers/cli/epub_scraper.py` passes - [ ] `mypy src/skill_seekers/cli/epub_scraper.py` passes (continue-on-error) - [x] `pip install -e ".[epub]"` installs successfully #### Manual Verification: - [x] Verify `import ebooklib` works after install - [x] Review epub_scraper.py structure matches word_scraper.py pattern **Implementation Note**: After completing this phase and all automated verification passes, pause here for manual confirmation from the human before proceeding to the next phase. --- ## Phase 2: CLI Integration ### Overview Wire the EPUB scraper into the CLI: source detection, argument definitions, parser registration, create command routing, and entry points. ### Changes Required: #### [x] 1. Source detection **File**: `src/skill_seekers/cli/source_detector.py` **Changes**: Add `.epub` extension detection, `_detect_epub()` method, validation, and error message update Add after the `.docx` check (line 64): ```python if source.endswith(".epub"): return cls._detect_epub(source) ``` Add `_detect_epub()` method (following `_detect_word()` at line 124): ```python @classmethod def _detect_epub(cls, source: str) -> SourceInfo: """Detect EPUB file source.""" name = os.path.splitext(os.path.basename(source))[0] return SourceInfo( type="epub", parsed={"file_path": source}, suggested_name=name, raw_input=source ) ``` Add epub validation in `validate_source()` (after word block at line 278): ```python elif source_info.type == "epub": file_path = source_info.parsed["file_path"] if not os.path.exists(file_path): raise ValueError(f"EPUB file does not exist: {file_path}") if not os.path.isfile(file_path): raise ValueError(f"Path is not a file: {file_path}") ``` Add EPUB example to the ValueError message (line 94): ```python " EPUB: skill-seekers create ebook.epub\n" ``` #### [x] 2. Argument definitions **File**: `src/skill_seekers/cli/arguments/epub.py` (new) **Changes**: EPUB-specific argument definitions ```python """EPUB-specific CLI arguments.""" import argparse from typing import Any from .common import add_all_standard_arguments EPUB_ARGUMENTS: dict[str, dict[str, Any]] = { "epub": { "flags": ("--epub",), "kwargs": { "type": str, "help": "Direct EPUB file path", "metavar": "PATH", }, }, "from_json": { "flags": ("--from-json",), "kwargs": { "type": str, "help": "Build skill from extracted JSON", "metavar": "FILE", }, }, } def add_epub_arguments(parser: argparse.ArgumentParser) -> None: """Add EPUB-specific arguments to parser.""" add_all_standard_arguments(parser) # Override enhance-level default to 0 for EPUB for action in parser._actions: if hasattr(action, "dest") and action.dest == "enhance_level": action.default = 0 action.help = ( "AI enhancement level (auto-detects API vs LOCAL mode): " "0=disabled (default for EPUB), 1=SKILL.md only, " "2=+architecture/config, 3=full enhancement. " "Mode selection: uses API if ANTHROPIC_API_KEY is set, " "otherwise LOCAL (Claude Code)" ) for arg_name, arg_def in EPUB_ARGUMENTS.items(): parser.add_argument(*arg_def["flags"], **arg_def["kwargs"]) ``` #### [x] 3. Create command argument integration **File**: `src/skill_seekers/cli/arguments/create.py` **Changes**: Add EPUB_ARGUMENTS dict, register in helper functions, add mode handling Add after WORD_ARGUMENTS (~line 411): ```python # EPUB specific (from epub.py) EPUB_ARGUMENTS: dict[str, dict[str, Any]] = { "epub": { "flags": ("--epub",), "kwargs": { "type": str, "help": "EPUB file path", "metavar": "PATH", }, }, } ``` Add to `get_source_specific_arguments()` (line 595): ```python "epub": EPUB_ARGUMENTS, ``` Add to `add_create_arguments()` (after word block at line 678): ```python if mode in ["epub", "all"]: for arg_name, arg_def in EPUB_ARGUMENTS.items(): parser.add_argument(*arg_def["flags"], **arg_def["kwargs"]) ``` #### [x] 4. Parser class **File**: `src/skill_seekers/cli/parsers/epub_parser.py` (new) **Changes**: Subcommand parser for standalone `skill-seekers epub` command ```python """Parser for epub subcommand.""" from .base import SubcommandParser from skill_seekers.cli.arguments.epub import add_epub_arguments class EpubParser(SubcommandParser): """Parser for EPUB extraction command.""" @property def name(self) -> str: return "epub" @property def help(self) -> str: return "Extract from EPUB e-book (.epub)" @property def description(self) -> str: return "Extract content from EPUB e-book (.epub) and generate skill" def add_arguments(self, parser): add_epub_arguments(parser) ``` #### [x] 5. Parser registration **File**: `src/skill_seekers/cli/parsers/__init__.py` **Changes**: Import and register EpubParser Add import (after WordParser import, line 15): ```python from .epub_parser import EpubParser ``` Add to PARSERS list (after `WordParser()`, line 46): ```python EpubParser(), ``` #### [x] 6. CLI dispatcher **File**: `src/skill_seekers/cli/main.py` **Changes**: Add epub to COMMAND_MODULES dict and module docstring Add to COMMAND_MODULES (after "word" entry, line 52): ```python "epub": "skill_seekers.cli.epub_scraper", ``` Add to module docstring (after "word" line, line 15): ```python # epub Extract from EPUB e-book (.epub) ``` #### [x] 7. Create command routing **File**: `src/skill_seekers/cli/create_command.py` **Changes**: Add `_route_epub()` method, routing case, help flag, and epilog example Add to `_route_to_scraper()` (after word case, line 136): ```python elif self.source_info.type == "epub": return self._route_epub() ``` Add `_route_epub()` method (after `_route_word()`, line 352): ```python def _route_epub(self) -> int: """Route to EPUB scraper (epub_scraper.py).""" from skill_seekers.cli import epub_scraper argv = ["epub_scraper"] file_path = self.source_info.parsed["file_path"] argv.extend(["--epub", file_path]) self._add_common_args(argv) logger.debug(f"Calling epub_scraper with argv: {argv}") original_argv = sys.argv try: sys.argv = argv return epub_scraper.main() finally: sys.argv = original_argv ``` Add to epilog (line 543, after DOCX example): ```python EPUB: skill-seekers create ebook.epub ``` Add to Source Auto-Detection section: ```python • file.epub → EPUB extraction ``` Add `--help-epub` flag and handler (after `--help-word` at line 592): ```python parser.add_argument( "--help-epub", action="store_true", help=argparse.SUPPRESS, dest="_help_epub" ) ``` Add handler block (after `_help_word` block at line 654): ```python elif args._help_epub: parser_epub = argparse.ArgumentParser( prog="skill-seekers create", description="Create skill from EPUB e-book (.epub)", formatter_class=argparse.RawDescriptionHelpFormatter, ) add_create_arguments(parser_epub, mode="epub") parser_epub.print_help() return 0 ``` #### [x] 8. Entry point **File**: `pyproject.toml` **Changes**: Add standalone entry point Add after `skill-seekers-word` (line 224): ```toml skill-seekers-epub = "skill_seekers.cli.epub_scraper:main" ``` #### [x] 9. Positional argument handling in main.py **File**: `src/skill_seekers/cli/main.py` **Changes**: Add "input_file" is already in the positional list at line 153, so no change needed. Verify `_reconstruct_argv` handles epub correctly through the standard delegation path. ### Success Criteria: #### Automated Verification: - [x] `ruff check src/skill_seekers/cli/source_detector.py src/skill_seekers/cli/arguments/epub.py src/skill_seekers/cli/parsers/epub_parser.py src/skill_seekers/cli/create_command.py` passes - [x] `ruff format --check src/skill_seekers/cli/` passes - [x] `pip install -e ".[epub]"` installs with all entry points - [x] `skill-seekers create --help` shows EPUB-specific help - [x] `skill-seekers create --help-epub` shows EPUB arguments (via standalone entry point `skill-seekers-create`) - [x] `skill-seekers create nonexistent.epub` gives clear error about missing file - [x] Existing tests still pass: `pytest tests/ -v -x -m "not slow and not integration"` (875 passed, 1 pre-existing unrelated failure in test_git_sources_e2e) #### Manual Verification: - [x] `skill-seekers --help` lists `epub` command - [x] `skill-seekers create book.epub --dry-run` shows dry run output **Implementation Note**: After completing this phase and all automated verification passes, pause here for manual confirmation from the human before proceeding to the next phase. --- ## Phase 3: Comprehensive Test Suite ### Overview Create `tests/test_epub_scraper.py` with 100+ tests across 11 test classes, covering happy path, negative cases, edge cases, and CLI integration. ### Changes Required: #### [x] 1. Create test file **File**: `tests/test_epub_scraper.py` (new) **Changes**: Comprehensive test suite following `test_word_scraper.py` patterns ```python """ Tests for EPUB scraper (epub_scraper.py). Covers: initialization, extraction, categorization, skill building, code blocks, tables, images, error handling, JSON workflow, CLI arguments, helper functions, source detection, DRM detection, and edge cases. Tests use mock data and do not require actual EPUB files or ebooklib installed. """ import json import os import shutil import tempfile import unittest from pathlib import Path from unittest.mock import MagicMock, patch, PropertyMock # Conditional import (same pattern as test_word_scraper.py) try: from skill_seekers.cli.epub_scraper import ( EpubToSkillConverter, infer_description_from_epub, _score_code_quality, _check_epub_deps, EPUB_AVAILABLE, ) IMPORT_OK = True except ImportError: IMPORT_OK = False ``` **Helper factory function:** ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Test Classes and Methods: #### [x] Class 1: `TestEpubToSkillConverterInit` (8 tests) **Happy path:** - `test_init_with_name_and_epub_path` — basic config with name + epub_path - `test_init_with_full_config` — config with all fields (name, epub_path, description, categories) - `test_default_description_uses_name` — description defaults to "Use when referencing {name} documentation" - `test_skill_dir_uses_name` — skill_dir is `output/{name}` - `test_data_file_uses_name` — data_file is `output/{name}_extracted.json` **Negative:** - `test_init_requires_name` — missing "name" key raises KeyError - `test_init_empty_name` — empty string name still works (no crash) **Edge case:** - `test_init_with_special_characters_in_name` — name with spaces/dashes sanitized for paths #### [x] Class 2: `TestEpubExtraction` (12 tests) **Happy path:** - `test_extract_basic_epub` — mock ebooklib, verify sections extracted in spine order - `test_extract_metadata` — verify Dublin Core metadata extraction (title, creator, language, etc.) - `test_extract_multiple_chapters` — multiple spine items produce multiple sections - `test_extract_code_blocks` — `` elements extracted with language detection - `test_extract_images` — ITEM_IMAGE items extracted with correct content - `test_heading_boundary_splitting` — h1/h2 boundaries create new sections **Negative:** - `test_extract_missing_file_raises_error` — FileNotFoundError for nonexistent path - `test_extract_invalid_epub_raises_error` — ValueError for corrupted/non-EPUB file - `test_extract_deps_not_installed` — RuntimeError with install instructions when ebooklib missing **Edge cases:** - `test_extract_empty_spine` — EPUB with no spine items produces empty sections list - `test_extract_spine_item_no_body` — XHTML without `` tag skipped gracefully - `test_extract_non_linear_spine_items` — linear="no" items still extracted (included but flagged) #### [x] Class 3: `TestEpubDrmDetection` (6 tests) **Happy path:** - `test_no_drm_detected` — normal EPUB without encryption.xml returns False **Negative:** - `test_drm_detected_adobe_adept` — encryption.xml with Adobe namespace raises RuntimeError - `test_drm_detected_apple_fairplay` — encryption.xml with Apple namespace raises RuntimeError - `test_drm_detected_readium_lcp` — encryption.xml with Readium namespace raises RuntimeError **Edge cases:** - `test_font_obfuscation_not_drm` — encryption.xml with only IDPF font obfuscation algorithm (`http://www.idpf.org/2008/embedding`) is NOT DRM, extraction proceeds - `test_drm_error_message_is_clear` — error message mentions DRM and suggests removing protection #### [x] Class 4: `TestEpubCategorization` (8 tests) **Happy path:** - `test_single_source_creates_one_category` — single EPUB creates category named after file - `test_keyword_categorization` — sections matched to categories by keyword scoring - `test_no_categories_uses_default` — no category config creates single "content" category **Negative:** - `test_categorize_empty_sections` — empty sections list produces empty categories - `test_categorize_no_keyword_matches` — unmatched sections go to "other" category **Edge cases:** - `test_categorize_single_section` — one section creates one category - `test_categorize_many_sections` — 50+ sections categorized correctly - `test_categorize_preserves_section_order` — sections maintain original order within categories #### [x] Class 5: `TestEpubSkillBuilding` (10 tests) **Happy path:** - `test_build_creates_directory_structure` — output/{name}/, references/, scripts/, assets/ created - `test_build_generates_skill_md` — SKILL.md created with YAML frontmatter - `test_build_generates_reference_files` — reference markdown files created per category - `test_build_generates_index` — references/index.md created with category links - `test_skill_md_contains_metadata` — SKILL.md includes title, author, language from metadata - `test_skill_md_yaml_frontmatter` — frontmatter has name and description fields **Negative:** - `test_build_without_extracted_data_fails` — calling build_skill() before extraction raises error **Edge cases:** - `test_build_overwrites_existing_output` — re-running build overwrites existing files - `test_build_with_long_name` — name > 64 chars truncated in YAML frontmatter - `test_build_with_unicode_content` — Unicode text (CJK, Arabic, emoji) preserved correctly #### [x] Class 6: `TestEpubCodeBlocks` (8 tests) **Happy path:** - `test_code_blocks_included_in_reference_files` — code samples appear in reference markdown - `test_code_blocks_in_skill_md_top_15` — SKILL.md shows top 15 code examples by quality - `test_code_language_grouped` — code examples grouped by language in SKILL.md **Edge cases:** - `test_empty_code_block` — ``` /* Detailed source-code truncated for AI context efficiency. */ ``` skill-seekers create book.epub --name myskill ``` Add to "Unified create" examples: ``` skill-seekers create book.epub ``` Add to Key source files table: ``` | Core scraping | `cli/epub_scraper.py` | ``` Add to "Adding things → New create command flags" section: ``` - Source-specific → `EPUB_ARGUMENTS` ``` #### [x] 2. Update CHANGELOG.md **File**: `CHANGELOG.md` **Changes**: Add entry for EPUB support under next version ```markdown ### Added - EPUB (.epub) input support via `skill-seekers create book.epub` or `skill-seekers epub --epub book.epub` - Extracts chapters, metadata, code blocks, images, and tables from EPUB 2 and EPUB 3 files - DRM detection with clear error messages - Optional dependency: `pip install "skill-seekers[epub]"` ``` ### Success Criteria: #### Automated Verification: - [x] `ruff check` passes on any modified files - [x] `pytest tests/ -v -x -m "not slow and not integration"` — all tests still pass (982 passed, 1 pre-existing failure) #### Manual Verification: - [x] CLAUDE.md accurately reflects new commands - [x] CHANGELOG.md entry is clear and complete **Implementation Note**: After completing this phase and all automated verification passes, pause here for manual confirmation from the human before proceeding. --- ## Testing Strategy ### Unit Tests (Phase 3 — ~108 tests): **By category:** | Category | Count | What's tested | |----------|-------|---------------| | Initialization | 8 | Config parsing, defaults, edge cases | | Extraction | 12 | Spine iteration, metadata, headings, code, images | | DRM detection | 6 | Adobe, Apple, Readium, font obfuscation (not DRM) | | Categorization | 8 | Single/multi category, keywords, empty, ordering | | Skill building | 10 | Directory structure, SKILL.md, references, index | | Code blocks | 8 | Extraction, quality, language detection, HTML entities | | Tables | 5 | Headers, no-thead fallback, empty, colspan | | Images | 7 | Save, references, SVG, conflicts, cover, many | | Error handling | 10 | Missing file, corrupt, DRM, no deps, malformed XHTML | | JSON workflow | 6 | Load, build, round-trip, invalid, missing fields | | CLI arguments | 8 | Flags, defaults, dry-run, verbose/quiet | | Helper functions | 6 | Description inference, quality scoring, filename sanitization | | Source detection | 6 | Detection, validation, regression | | Edge cases | 8 | EPUB 2/3, no TOC, empty chapters, Unicode, fixed-layout | ### Integration Tests: - Full extract → categorize → build workflow with mock ebooklib - JSON round-trip (extract → save → load → build) ### Manual Testing Steps: 1. `pip install -e ".[epub]"` — verify install 2. `skill-seekers create book.epub` with a real EPUB file — verify output directory structure 3. `skill-seekers create book.epub --dry-run` — verify dry run output 4. `skill-seekers create drm-book.epub` — verify DRM error message 5. `skill-seekers create nonexistent.epub` — verify file-not-found error 6. Open generated `SKILL.md` — verify content quality and structure ## Performance Considerations - ebooklib loads entire EPUB into memory. For typical books (<50MB), this is fine - For very large EPUBs (100MB+), memory usage may spike. No mitigation needed for v1 — document as known limitation - BeautifulSoup parsing of XHTML is fast. No performance concerns expected ## Migration Notes - No migration needed — this is a new feature with no existing data to migrate - Optional dependency (`ebooklib`) means existing installs are unaffected - No breaking changes to any existing commands or APIs ## References - [W3C EPUB 3.3 Specification](https://www.w3.org/TR/epub-33/) — authoritative source of truth - [W3C EPUB Reading Systems 3.3](https://www.w3.org/TR/epub-rs-33/) — reading system requirements - [ebooklib GitHub](https://github.com/aerkalov/ebooklib) — Python EPUB library - [ebooklib PyPI](https://pypi.org/project/EbookLib/) — v0.20, Python 3.9-3.13 - [Research document](../research/2026-03-14-epub-input-support-affected-files.md) — affected files analysis - Similar implementation: `src/skill_seekers/cli/word_scraper.py` — closest analog - Similar tests: `tests/test_word_scraper.py` — test pattern template --- ## File: docs/advanced/custom-workflows.md # Custom Workflows Guide > **Skill Seekers v3.9.0** > **Create custom AI enhancement workflows** --- ## What are Custom Workflows? Workflows are YAML-defined, multi-stage AI enhancement pipelines: ```yaml my-workflow.yaml ├── name ├── description ├── variables (optional) └── stages (1-10) ├── name ├── type (builtin/custom) ├── target (skill_md/references/) ├── prompt └── uses_history (optional) ``` --- ## Basic Workflow Structure ```yaml name: my-custom stages: - name: stage-one type: builtin target: skill_md prompt: | Improve the SKILL.md by adding... - name: stage-two type: custom target: references prompt: | Enhance the references by... ``` --- ## Workflow Fields ### Top Level | Field | Required | Description | |-------|----------|-------------| | `name` | Yes | Workflow identifier | | `description` | No | Human-readable description | | `variables` | No | Configurable variables | | `stages` | Yes | Array of stage definitions | ### Stage Fields | Field | Required | Description | |-------|----------|-------------| | `name` | Yes | Stage identifier | | `type` | Yes | `builtin` or `custom` | | `target` | Yes | `skill_md` or `references` | | `prompt` | Yes | AI prompt text | | `uses_history` | No | Access previous stage results | --- ## Creating Your First Workflow ### Example: Performance Analysis ```yaml # performance.yaml name: performance-focus variables: target_latency: "100ms" target_throughput: "1000 req/s" stages: - name: performance-overview type: builtin target: skill_md prompt: | Add a "Performance" section to SKILL.md covering: - Benchmark results - Performance characteristics - Resource requirements - name: optimization-guide type: custom target: references uses_history: true prompt: | Create an optimization guide with: - Target latency: {target_latency} - Target throughput: {target_throughput} - Common bottlenecks - Optimization techniques ``` ### Install and Use ```bash # Add workflow skill-seekers workflows add performance.yaml # Use it skill-seekers create --enhance-workflow performance-focus # With custom variables skill-seekers create \ --enhance-workflow performance-focus \ --var target_latency=50ms \ --var target_throughput=5000req/s ``` --- ## Stage Types ### builtin Uses built-in enhancement logic: ```yaml stages: - name: structure-improvement type: builtin target: skill_md prompt: "Improve document structure" ``` ### custom Full custom prompt control: ```yaml stages: - name: custom-analysis type: custom target: skill_md prompt: | Your detailed custom prompt here... Can use {variables} and {history} ``` --- ## Targets ### skill_md Enhances the main SKILL.md file: ```yaml stages: - name: improve-skill target: skill_md prompt: "Add comprehensive overview section" ``` ### references Enhances reference files: ```yaml stages: - name: improve-refs target: references prompt: "Add cross-references between files" ``` --- ## Variables ### Defining Variables ```yaml variables: audience: "beginners" focus_area: "security" include_examples: true ``` ### Using Variables ```yaml stages: - name: customize prompt: | Tailor content for {audience}. Focus on {focus_area}. Include examples: {include_examples} ``` ### Overriding at Runtime ```bash skill-seekers create \ --enhance-workflow my-workflow \ --var audience=experts \ --var focus_area=performance ``` --- ## History Passing Access results from previous stages: ```yaml stages: - name: analyze type: custom target: skill_md prompt: "Analyze security features" - name: document type: custom target: skill_md uses_history: true prompt: | Based on previous analysis: {previous_results} Create documentation... ``` --- ## Advanced Example: Security Review ```yaml name: comprehensive-security variables: compliance_framework: "OWASP Top 10" risk_level: "high" stages: - name: asset-inventory type: builtin target: skill_md prompt: | Document all security-sensitive components: - Authentication mechanisms - Authorization checks - Data validation - Encryption usage - name: threat-analysis type: custom target: skill_md uses_history: true prompt: | Based on assets: {all_history} Analyze threats for {compliance_framework}: - Threat vectors - Attack scenarios - Risk ratings ({risk_level} focus) - name: mitigation-guide type: custom target: references uses_history: true prompt: | Create mitigation guide: - Countermeasures - Best practices - Code examples - Testing strategies ``` --- ## Validation ### Validate Before Installing ```bash skill-seekers workflows validate ./my-workflow.yaml ``` ### Common Errors | Error | Cause | Fix | |-------|-------|-----| | `Missing 'stages'` | No stages array | Add stages: | | `Invalid type` | Not builtin/custom | Check type field | | `Undefined variable` | Used but not defined | Add to variables: | --- ## Best Practices ### 1. Start Simple ```yaml # Start with 1-2 stages name: simple stages: - name: improve type: builtin target: skill_md prompt: "Improve SKILL.md" ``` ### 2. Use Clear Stage Names ```yaml # Good stages: - name: security-overview - name: vulnerability-analysis # Bad stages: - name: stage1 - name: step2 ``` ### 3. Document Variables ```yaml variables: # Target audience level: beginner, intermediate, expert audience: "intermediate" # Security focus area: owasp, pci, hipaa compliance: "owasp" ``` ### 4. Test Incrementally ```bash # Test with dry run skill-seekers create \ --enhance-workflow my-workflow \ --workflow-dry-run # Then actually run skill-seekers create \ --enhance-workflow my-workflow ``` ### 5. Chain for Complex Analysis ```bash # Use multiple workflows skill-seekers create \ --enhance-workflow security-focus \ --enhance-workflow performance-focus ``` --- ## Sharing Workflows ### Export Workflow ```bash # Get workflow content skill-seekers workflows show my-workflow > my-workflow.yaml ``` ### Share with Team ```bash # Add to version control git add my-workflow.yaml git commit -m "Add custom security workflow" # Team members install skill-seekers workflows add my-workflow.yaml ``` ### Publish Submit to Skill Seekers community: - GitHub Discussions - Skill Seekers website - Documentation contributions --- ## See Also - [Workflows Guide](../user-guide/05-workflows.md) - Using workflows - [MCP Reference](../reference/MCP_REFERENCE.md) - Workflows via MCP - [Enhancement Guide](../user-guide/03-enhancement.md) - Enhancement fundamentals --- ## File: docs/advanced/mcp-server.md # MCP Server Setup Guide > **Skill Seekers v3.9.0** > **Integrate with AI agents via Model Context Protocol** --- ## What is MCP? MCP (Model Context Protocol) lets AI agents like Claude Code control Skill Seekers through natural language: ``` You: "Scrape the React documentation" Claude: ▶️ scrape_docs({"url": "https://react.dev/"}) ✅ Done! Created output/react/ ``` --- ## Installation ```bash # Install with MCP support pip install skill-seekers[mcp] # Verify skill-seekers-mcp --version ``` --- ## Transport Modes ### stdio Mode (Default) For Claude Code, VS Code + Cline: ```bash skill-seekers-mcp ``` **Use when:** - Running in Claude Code - Direct integration with terminal-based agents - Simple local setup --- ### HTTP Mode For Cursor, Windsurf, HTTP clients: ```bash # Start HTTP server skill-seekers-mcp --transport http --port 8765 # Custom host skill-seekers-mcp --transport http --host 0.0.0.0 --port 8765 ``` **Use when:** - IDE integration (Cursor, Windsurf) - Remote access needed - Multiple clients --- ## Claude Code Integration ### Automatic Setup ```bash # In Claude Code, run: /claude add-mcp-server skill-seekers ``` Or manually add to `~/.claude/mcp.json`: ```json { "mcpServers": { "skill-seekers": { "command": "skill-seekers-mcp", "env": { "ANTHROPIC_API_KEY": "sk-ant-...", "GITHUB_TOKEN": "ghp_..." } } } } ``` ### Usage Once connected, ask Claude: ``` "List available configs" "Scrape the Django documentation" "Package output/react for Gemini" "Enhance output/my-skill with security-focus workflow" ``` --- ## Cursor IDE Integration ### Setup 1. Start MCP server: ```bash skill-seekers-mcp --transport http --port 8765 ``` 2. In Cursor Settings → MCP: - Name: `skill-seekers` - URL: `http://localhost:8765` ### Usage In Cursor chat: ``` "Create a skill from the current project" "Analyze this codebase and generate a cursorrules file" ``` --- ## Windsurf Integration ### Setup 1. Start MCP server: ```bash skill-seekers-mcp --transport http --port 8765 ``` 2. In Windsurf Settings: - Add MCP server endpoint: `http://localhost:8765` --- ## Available Tools 40 tools organized by category: ### Core Tools (9) - `list_configs` - List presets - `generate_config` - Create config from URL - `validate_config` - Check config - `estimate_pages` - Page estimation - `scrape_docs` - Scrape documentation - `package_skill` - Package skill - `upload_skill` - Upload to platform - `enhance_skill` - AI enhancement - `install_skill` - Complete workflow ### Extended Tools (10) - `scrape_github` - GitHub repo - `scrape_pdf` - PDF extraction - `scrape_generic` - Generic scraper for 10 new source types (see below) - `scrape_codebase` - Local code - `unified_scrape` - Multi-source - `detect_patterns` - Pattern detection - `extract_test_examples` - Test examples - `build_how_to_guides` - How-to guides - `extract_config_patterns` - Config patterns - `detect_conflicts` - Doc/code conflicts ### Config Sources (5) - `add_config_source` - Register git source - `list_config_sources` - List sources - `remove_config_source` - Remove source - `fetch_config` - Fetch configs - `submit_config` - Submit configs ### Vector DB (4) - `export_to_weaviate` - `export_to_chroma` - `export_to_faiss` - `export_to_qdrant` ### scrape_generic Tool The `scrape_generic` tool is the generic entry point for 10 new source types added in v3.2.0. It delegates to the appropriate CLI scraper module. **Supported source types:** `jupyter`, `html`, `openapi`, `asciidoc`, `pptx`, `rss`, `manpage`, `confluence`, `notion`, `chat` **Parameters:** | Name | Type | Required | Description | |------|------|----------|-------------| | `source_type` | string | Yes | One of the 10 supported source types | | `name` | string | Yes | Skill name for the output | | `path` | string | No | File or directory path (for file-based sources) | | `url` | string | No | URL (for URL-based sources like confluence, notion, rss) | **Usage examples:** ``` "Scrape the Jupyter notebook analysis.ipynb" → scrape_generic(source_type="jupyter", name="analysis", path="analysis.ipynb") "Extract content from the API spec" → scrape_generic(source_type="openapi", name="my-api", path="api-spec.yaml") "Process the PowerPoint slides" → scrape_generic(source_type="pptx", name="slides", path="presentation.pptx") "Scrape the Confluence wiki" → scrape_generic(source_type="confluence", name="wiki", url="https://wiki.example.com") ``` See [MCP Reference](../reference/MCP_REFERENCE.md) for full details. --- ## Common Workflows ### Workflow 1: Documentation Skill ``` User: "Create a skill from React docs" Claude: ▶️ scrape_docs({"url": "https://react.dev/"}) ⏳ Scraping... ✅ Created output/react/ ▶️ package_skill({"skill_directory": "output/react/", "target": "claude"}) ✅ Created output/react-claude.zip Skill ready! Upload to Claude? ``` ### Workflow 2: GitHub Analysis ``` User: "Analyze the facebook/react repo" Claude: ▶️ scrape_github({"repo": "facebook/react"}) ⏳ Analyzing... ✅ Created output/react/ ▶️ enhance_skill({"skill_directory": "output/react/", "workflow": "architecture-comprehensive"}) ✅ Enhanced with architecture analysis ``` ### Workflow 3: Multi-Platform Export ``` User: "Create Django skill for all platforms" Claude: ▶️ scrape_docs({"config": "django"}) ✅ Created output/django/ ▶️ package_skill({"skill_directory": "output/django/", "target": "claude"}) ▶️ package_skill({"skill_directory": "output/django/", "target": "gemini"}) ▶️ package_skill({"skill_directory": "output/django/", "target": "openai"}) ✅ Created packages for all platforms ``` --- ## Configuration ### Environment Variables Set in `~/.claude/mcp.json` or before starting server: ```bash export ANTHROPIC_API_KEY=sk-ant-... export GOOGLE_API_KEY=AIza... export OPENAI_API_KEY=sk-... export GITHUB_TOKEN=ghp_... ``` ### Server Options ```bash # Debug mode skill-seekers-mcp --verbose # Custom port skill-seekers-mcp --port 8080 # Allow all origins (CORS) skill-seekers-mcp --cors ``` --- ## Security ### Local Only (stdio) ```bash # Only accessible by local Claude Code skill-seekers-mcp ``` ### HTTP with Auth ```bash # Use reverse proxy with auth # nginx, traefik, etc. ``` ### API Key Protection ```bash # Don't hardcode keys # Use environment variables # Or secret management ``` --- ## Troubleshooting ### "Server not found" ```bash # Check if running curl http://localhost:8765/health # Restart skill-seekers-mcp --transport http --port 8765 ``` ### "Tool not available" ```bash # Check version skill-seekers-mcp --version # Update pip install --upgrade skill-seekers[mcp] ``` ### "Connection refused" ```bash # Check port lsof -i :8765 # Use different port skill-seekers-mcp --port 8766 ``` --- ## See Also - [MCP Reference](../reference/MCP_REFERENCE.md) - Complete tool reference - MCP Tools Deep Dive - Advanced usage - [MCP Protocol](https://modelcontextprotocol.io/) - Official MCP docs --- ## File: docs/advanced/multi-source.md # Multi-Source Scraping Guide > **Skill Seekers v3.9.0** > **Combine 18 source types into one unified skill** --- ## What is Multi-Source Scraping? Combine multiple sources into a single, comprehensive skill. Skill Seekers supports **18 source types** that can be freely mixed and matched: ``` ┌──────────────┐ │ Documentation│──┐ │ (Web docs) │ │ ├──────────────┤ │ │ GitHub Repo │ │ │ (Source code) │ │ ├──────────────┤ │ ┌──────────────────┐ │ PDF / Word / │ │ │ Unified Skill │ │ EPUB / PPTX │──┼────▶│ (Single source │ ├──────────────┤ │ │ of truth) │ │ Video / │ │ └──────────────────┘ │ Jupyter / HTML│ │ ├──────────────┤ │ │ OpenAPI / │ │ │ AsciiDoc / │ │ │ RSS / Man │ │ ├──────────────┤ │ │ Confluence / │──┘ │ Notion / Chat│ └──────────────┘ ``` --- ## When to Use Multi-Source ### Use Cases | Scenario | Sources | Benefit | |----------|---------|---------| | Framework + Examples | Docs + GitHub repo | Theory + practice | | Product + API | Docs + OpenAPI spec | Usage + reference | | Legacy + Current | PDF + Web docs | Complete history | | Internal + External | Local code + Public docs | Full context | | Data Science Project | Jupyter + GitHub + Docs | Code + notebooks + docs | | Enterprise Wiki | Confluence + GitHub + Video | Wiki + code + tutorials | | API-First Product | OpenAPI + Docs + Jupyter | Spec + docs + examples | | CLI Tool | Man pages + GitHub + AsciiDoc | Reference + code + docs | | Team Knowledge | Notion + Slack/Discord + Docs | Notes + discussions + docs | | Book + Code | EPUB + GitHub + PDF | Theory + implementation | | Presentations + Code | PowerPoint + GitHub + Docs | Slides + code + reference | | Content Feed | RSS/Atom + Docs + GitHub | Updates + docs + code | ### Benefits - **Single source of truth** - One skill with all context - **Conflict detection** - Find doc/code discrepancies - **Cross-references** - Link between sources - **Comprehensive** - No gaps in knowledge --- ## Creating Unified Configs ### Basic Structure ```json { "name": "my-framework-complete", "description": "Complete documentation and code", "merge_mode": "claude-enhanced", "sources": [ { "type": "docs", "name": "documentation", "base_url": "https://docs.example.com/" }, { "type": "github", "name": "source-code", "repo": "owner/repo" } ] } ``` --- ## Source Types (17 Supported) ### 1. Documentation (Web) ```json { "type": "docs", "name": "official-docs", "base_url": "https://docs.framework.com/", "max_pages": 500, "categories": { "getting_started": ["intro", "quickstart"], "api": ["reference", "api"] } } ``` ### 2. GitHub Repository ```json { "type": "github", "name": "source-code", "repo": "facebook/react", "fetch_issues": true, "max_issues": 100, "enable_codebase_analysis": true } ``` ### 3. PDF Document ```json { "type": "pdf", "name": "legacy-manual", "pdf_path": "docs/legacy-manual.pdf", "enable_ocr": false } ``` ### 4. Local Codebase ```json { "type": "local", "name": "internal-tools", "directory": "./internal-lib", "languages": ["Python", "JavaScript"] } ``` ### 5. Word Document (.docx) ```json { "type": "word", "name": "product-spec", "path": "docs/specification.docx" } ``` ### 6. Video (YouTube/Vimeo/Local) ```json { "type": "video", "name": "tutorial-video", "url": "https://www.youtube.com/watch?v=example", "language": "en" } ``` ### 7. EPUB ```json { "type": "epub", "name": "programming-book", "path": "books/python-guide.epub" } ``` ### 8. Jupyter Notebook ```json { "type": "jupyter", "name": "analysis-notebooks", "path": "notebooks/data-analysis.ipynb" } ``` ### 9. Local HTML ```json { "type": "html", "name": "exported-docs", "path": "exports/documentation.html" } ``` ### 10. OpenAPI/Swagger ```json { "type": "openapi", "name": "api-spec", "path": "specs/openapi.yaml" } ``` ### 11. AsciiDoc ```json { "type": "asciidoc", "name": "technical-docs", "path": "docs/manual.adoc" } ``` ### 12. PowerPoint (.pptx) ```json { "type": "pptx", "name": "architecture-deck", "path": "presentations/architecture.pptx" } ``` ### 13. RSS/Atom Feed ```json { "type": "rss", "name": "release-feed", "url": "https://blog.example.com/releases.xml" } ``` ### 14. Man Pages ```json { "type": "manpage", "name": "cli-reference", "path": "man/mytool.1" } ``` ### 15. Confluence ```json { "type": "confluence", "name": "team-wiki", "base_url": "https://company.atlassian.net/wiki", "space_key": "ENGINEERING" } ``` ### 16. Notion ```json { "type": "notion", "name": "project-docs", "workspace": "my-workspace", "root_page_id": "abc123def456" } ``` ### 17. Slack/Discord (Chat) ```json { "type": "chat", "name": "team-discussions", "path": "exports/slack-export/" } ``` --- ## Complete Example ### React Complete Skill ```json { "name": "react-complete", "description": "React - docs, source, and guides", "merge_mode": "claude-enhanced", "sources": [ { "type": "docs", "name": "react-docs", "base_url": "https://react.dev/", "max_pages": 300, "categories": { "getting_started": ["learn", "tutorial"], "api": ["reference", "hooks"], "advanced": ["concurrent", "suspense"] } }, { "type": "github", "name": "react-source", "repo": "facebook/react", "fetch_issues": true, "max_issues": 50, "enable_codebase_analysis": true, "code_analysis_depth": "deep" }, { "type": "pdf", "name": "react-patterns", "pdf_path": "downloads/react-patterns.pdf" } ], "conflict_detection": { "enabled": true, "rules": [ { "field": "api_signature", "action": "flag_mismatch" }, { "field": "version", "action": "warn_outdated" } ] }, "output_structure": { "group_by_source": false, "cross_reference": true } } ``` --- ## Running Unified Scraping ### Basic Command ```bash skill-seekers create --config react-complete.json ``` ### With Options ```bash # Fresh start (ignore cache) skill-seekers create --config react-complete.json --fresh # Dry run skill-seekers create --config react-complete.json --dry-run # Rule-based merging skill-seekers create --config react-complete.json --merge-mode rule-based ``` --- ## Merge Modes ### claude-enhanced (Default) Uses AI to intelligently merge sources: - Detects relationships between content - Resolves conflicts intelligently - Creates cross-references - Best quality, slower ```bash skill-seekers create --config my-config.json --merge-mode claude-enhanced ``` ### rule-based Uses defined rules for merging: - Faster - Deterministic - Less sophisticated ```bash skill-seekers create --config my-config.json --merge-mode rule-based ``` ### Generic Merge System When combining source types beyond the standard docs+github+pdf trio, the **generic merge system** (`_generic_merge()` in `unified_skill_builder.py`) handles any combination automatically. It uses pairwise synthesis for known combos (docs+github, docs+pdf, github+pdf) and falls back to a generic merging strategy for all other source type combinations. ### AI-Powered Multi-Source Merging For complex multi-source projects, use the `complex-merge.yaml` workflow preset to apply AI-powered merging: ```bash skill-seekers create --config my-config.json \ --enhance-workflow complex-merge ``` This workflow uses Claude to intelligently reconcile content from disparate source types, resolving conflicts and creating coherent cross-references between sources that would otherwise be difficult to merge deterministically. --- ## Conflict Detection ### Automatic Detection Finds discrepancies between sources: ```json { "conflict_detection": { "enabled": true, "rules": [ { "field": "api_signature", "action": "flag_mismatch" }, { "field": "version", "action": "warn_outdated" }, { "field": "deprecation", "action": "highlight" } ] } } ``` ### Conflict Report After scraping, check for conflicts: ```bash # Conflicts are reported in output ls output/react-complete/conflicts.json # Or use MCP tool detect_conflicts({ "docs_source": "output/react-docs", "code_source": "output/react-source" }) ``` --- ## Output Structure ### Merged Output ``` output/react-complete/ ├── SKILL.md # Combined skill ├── references/ │ ├── index.md # Master index │ ├── getting_started.md # From docs │ ├── api_reference.md # From docs │ ├── source_overview.md # From GitHub │ ├── code_examples.md # From GitHub │ └── patterns.md # From PDF ├── .skill-seekers/ │ ├── manifest.json # Metadata │ ├── sources.json # Source list │ └── conflicts.json # Detected conflicts └── cross-references.json # Links between sources ``` --- ## Best Practices ### 1. Name Sources Clearly ```json { "sources": [ {"type": "docs", "name": "official-docs"}, {"type": "github", "name": "source-code"}, {"type": "pdf", "name": "legacy-reference"}, {"type": "openapi", "name": "api-spec"}, {"type": "confluence", "name": "team-wiki"} ] } ``` ### 2. Limit Source Scope ```json { "type": "github", "name": "core-source", "repo": "owner/repo", "file_patterns": ["src/**/*.py"], // Only core files "exclude_patterns": ["tests/**", "docs/**"] } ``` ### 3. Enable Conflict Detection ```json { "conflict_detection": { "enabled": true } } ``` ### 4. Use Appropriate Merge Mode - **claude-enhanced** - Best quality, for important skills - **rule-based** - Faster, for testing or large datasets ### 5. Test Incrementally ```bash # Test with one source first skill-seekers create # Then add sources skill-seekers create --config my-config.json --dry-run ``` --- ## Troubleshooting ### "Source not found" ```bash # Check all sources exist curl -I https://docs.example.com/ ls downloads/manual.pdf ``` ### "Merge conflicts" ```bash # Check conflicts report cat output/my-skill/conflicts.json # Adjust merge_mode skill-seekers create --config my-config.json --merge-mode rule-based ``` ### "Out of memory" ```bash # Process sources separately # Then merge manually ``` --- ## Examples ### Framework + Examples ```json { "name": "django-complete", "sources": [ {"type": "docs", "base_url": "https://docs.djangoproject.com/"}, {"type": "github", "repo": "django/django", "fetch_issues": false} ] } ``` ### Docs + OpenAPI Spec ```json { "name": "stripe-complete", "sources": [ {"type": "docs", "base_url": "https://stripe.com/docs"}, {"type": "openapi", "path": "specs/stripe-openapi.yaml"} ] } ``` ### Code + Jupyter Notebooks ```json { "name": "ml-project", "sources": [ {"type": "github", "repo": "org/ml-pipeline"}, {"type": "jupyter", "path": "notebooks/training.ipynb"}, {"type": "jupyter", "path": "notebooks/evaluation.ipynb"} ] } ``` ### Confluence + GitHub ```json { "name": "internal-platform", "sources": [ {"type": "confluence", "base_url": "https://company.atlassian.net/wiki", "space_key": "PLATFORM"}, {"type": "github", "repo": "company/platform-core"}, {"type": "openapi", "path": "specs/platform-api.yaml"} ] } ``` ### Legacy + Current ```json { "name": "product-docs", "sources": [ {"type": "docs", "base_url": "https://docs.example.com/v2/"}, {"type": "pdf", "pdf_path": "v1-legacy-manual.pdf"} ] } ``` ### CLI Tool (Man Pages + GitHub + AsciiDoc) ```json { "name": "mytool-complete", "sources": [ {"type": "manpage", "path": "man/mytool.1"}, {"type": "github", "repo": "org/mytool"}, {"type": "asciidoc", "path": "docs/user-guide.adoc"} ] } ``` ### Team Knowledge (Notion + Chat + Video) ```json { "name": "onboarding-knowledge", "sources": [ {"type": "notion", "workspace": "engineering", "root_page_id": "abc123"}, {"type": "chat", "path": "exports/slack-engineering/"}, {"type": "video", "url": "https://www.youtube.com/playlist?list=PLonboarding"} ] } ``` --- ## See Also - [Config Format](../reference/CONFIG_FORMAT.md) - Full JSON specification - [Scraping Guide](../user-guide/02-scraping.md) - Individual source options - [MCP Reference](../reference/MCP_REFERENCE.md) - `scrape_docs` tool (handles unified multi-source configs)