## File: README.md

book-to-skill

**How it works, in 3 steps:** 1. **Point** it at a file, folder, or glob β€” `/book-to-skill ./my-book.pdf` 2. **It distills** the book into a skill β€” frameworks, decision rules, anti-patterns, and per-chapter files. Structure, not a summary. 3. **Your agent loads it on demand** β€” ask `/my-book replication` and it reads the right chapter and answers from the real content, no hallucination. --- ## πŸ€” Why You buy a great technical book. You read it once. Three months later you can't remember chapter 7 existed. The usual workarounds don't help: - πŸ“„ "Let me just search the PDF" β†’ you get a list of pages, not answers - 🧠 "I'll ask the agent about this book" β†’ it either hallucinates or says it doesn't have the content - πŸ“ "I'll take notes as I read" β†’ you end up with a 200-line doc you never open again **book-to-skill solves this by turning the book into a structured skill your agent loads on demand.** Once installed, you just type `/your-book-slug replication` and the agent reads the right chapter and answers from the actual content. No hallucination. No digging through PDFs. The book becomes part of your workflow. Works with any host that supports the open [Agent Skills](https://github.com/agentskills/agentskills) standard β€” GitHub Copilot CLI, Amp, and Claude Code all read the same `SKILL.md` format. --- ## πŸ“¦ What it generates Running `/book-to-skill your-book.pdf` (or a folder, glob, or list of files) creates a full skill in your agent's skills directory (`~/.copilot/skills//` for Copilot CLI, `~/.agents/skills//` for Amp or cross-agent, `~/.claude/skills//` for Claude Code): | File | Purpose | Size | |------|---------|------| | `SKILL.md` | Core mental models + chapter index | ~4,000 tokens | | `chapters/ch01-*.md` … | One file per chapter, loaded on-demand | ~1,000 tokens each | | `glossary.md` | Every key term, alphabetically sorted with chapter refs | ~1,500 tokens | | `patterns.md` | All techniques, algorithms, and design patterns | ~2,000 tokens | | `cheatsheet.md` | Decision tables and quick-reference rules | ~1,000 tokens | **Chapter files are loaded on-demand** β€” they don't count against the skill budget until you ask about that topic. --- ## 🏒 Beyond books The name says "book", but the input is any structured prose. The same extraction works on knowledge you own and re-read constantly: - **Internal documentation** β€” architecture decision records, runbooks, onboarding guides. Fold a whole `docs/` folder into one skill and ask it while you code. - **Brand & design systems** β€” voice guidelines, tone-of-voice docs, component principles. Turn a brand book into a skill your team queries instead of skimming a 60-page PDF. - **Research clusters** β€” a stack of papers plus your own notes, merged into a single unified skill and updated as new material lands (see [Update / fold-in](#-usage)). - **Specs & standards** β€” RFCs, API contracts, compliance docs you reference but never memorize. If you re-open a document often enough to wish you'd memorized it, it's a candidate. --- ## 🧾 The Discovery Loop Tax A PDF-reading agent doesn't just read β€” it *navigates*: it re-fetches the ToC, backtracks, and re-processes all of it on every turn. book-to-skill pays that structuring cost **once**, at conversion, so queries stay proportional to the answer β€” **24×–51Γ— fewer tokens** than dumping the book into context, measured on real books. πŸ“Š **Full methodology, numbers, and per-book tables β†’ [docs/performance.md](docs/performance.md#the-discovery-loop-tax)** --- ## βš™οΈ How it works Two halves: a deterministic Python **extractor** (document β†’ clean text + metadata) and a spec-driven **generator** (your agent follows `SKILL.md` to turn that into a structured skill). On-demand chapter files keep the loaded skill small. πŸ”§ **Full walkthrough (Steps 0–10, extraction modes, token budgets) β†’ [docs/how-it-works.md](docs/how-it-works.md)** --- ## πŸš€ Usage `/book-to-skill [skill-name]` β€” plus analyze-only, generate-from-analysis, and update/fold-in modes. After a conversion, the converter can publish the skill to GitHub (private by default) so any host installs it with `npx skills add`. ▢️ **All modes and examples β†’ [docs/usage.md](docs/usage.md)** πŸ’¬ **In practice β†’ [use cases](https://github.com/virgiliojr94/book-to-skill-use-cases)** β€” a DevEx book became a survey of 300+ engineers; a scanned PDF that stalled became [#130](https://github.com/virgiliojr94/book-to-skill/pull/130). Add yours: the account lives in your own Gist, the index takes a one-line PR. --- ## πŸ“₯ Install ```bash # One command, any host β€” via the cross-agent skills CLI: npx skills add virgiliojr94/book-to-skill # Or manually β€” clone into your skills folder (registers /book-to-skill): git clone https://github.com/virgiliojr94/book-to-skill.git ~/.claude/skills/book-to-skill # (Copilot CLI: ~/.copilot/skills/ Β· Amp/cross-agent: ~/.agents/skills/) ``` πŸ“₯ **All hosts, optional extractors, and the standalone CLI β†’ [docs/install.md](docs/install.md)** --- ## ❓ FAQ Common questions β€” "why not just dump the PDF?", cost, privacy, non-book inputs, multi-file books. ❓ **Answers β†’ [docs/faq.md](docs/faq.md)** --- πŸ”§ **Requirements** The extractor tries tools in order per format and uses the first available. If nothing is installed, it tells you which command to run. Plain text, Markdown, reStructuredText and AsciiDoc need no extra deps. > **Check your setup in one command:** `python3 scripts/extract.py --check` prints which extractors are installed for every format and the exact command to install anything missing β€” no file needed. **PDF β€” choose by book type:** | Book type | Tool | Install | Speed | |-----------|------|---------|-------| | Text-heavy (prose, few tables) | `pdftotext` (poppler) | `sudo apt install poppler-utils` | ⚑ instant | | Text-heavy fallback | `pypdf` | `pip3 install pypdf` | ⚑ instant | | Text-heavy fallback | `pdfminer.six` | `pip3 install pdfminer.six` | ⚑ instant | | **Technical (code, tables, formulas)** | **`docling`** | `pip3 install docling` | ~1.5s/page | > Before extraction begins, the skill asks you whether the book is **technical** or **text-heavy** and picks the right tool automatically. Docling preserves markdown tables and code blocks; pdftotext is faster for prose-only books. > **Scanned PDFs need OCR first.** A PDF that is page images with no text layer β€” a photographed or scanned book β€” has nothing for these tools to extract. The extractor checks the first pages and stops immediately with an explanation, rather than working through the whole book to produce an empty skill. Run OCR yourself, then convert the result: > > ```bash > ocrmypdf input.pdf output.pdf > ``` **EPUB:** | Tool | Install | Quality | |------|---------|---------| | `ebooklib` + `beautifulsoup4` | `pip3 install ebooklib beautifulsoup4` | ⭐⭐⭐ Best | | stdlib `zipfile` | built-in β€” no install needed | ⭐⭐ Always available | **Other formats:** | Format | Tool | Install | |--------|------|---------| | DOCX | `python-docx` (fallback: stdlib ZIP/XML) | `pip3 install python-docx` | | HTML | `beautifulsoup4` (fallback: stdlib `html.parser`) | `pip3 install beautifulsoup4` | | RTF | `striprtf` (fallback: regex) | `pip3 install striprtf` | | MOBI / AZW / AZW3 | Calibre `ebook-convert` (external app, not pip) | https://calibre-ebook.com/download | | TXT / Markdown / reStructuredText / AsciiDoc | built-in | β€” | --- πŸ“ **Repository structure** ``` book-to-skill/ β”œβ”€β”€ SKILL.md # Skill definition + step-by-step instructions (the generator spec) β”œβ”€β”€ scripts/ β”‚ β”œβ”€β”€ extract.py # Thin entrypoint wrapper β”‚ └── extractor/ # Modular extraction package β”‚ β”œβ”€β”€ config.py # Extensions, paths, dependency constants β”‚ β”œβ”€β”€ dependencies.py # optional-dep probing + --check β”‚ β”œβ”€β”€ exceptions.py # ExtractionError (per-source failures, batch-safe) β”‚ β”œβ”€β”€ utils.py # CLI parsing, multi-source resolution, chapter detection, runner β”‚ └── parsers/ # Format-specific parsers (pdf, epub, docx, html, rtf, calibre, text) β”œβ”€β”€ tools/ β”‚ β”œβ”€β”€ discovery_tax.py # measures token cost vs context-dump / discovery loop β”‚ └── validate_skill.py # checks a generated SKILL.md against host rules (--lens claude|copilot|amp) β”œβ”€β”€ tests/ # pytest suite (extraction, detection, discovery tax) β”œβ”€β”€ docs/ β”‚ β”œβ”€β”€ performance.md # measured benchmarks, discovery tax, cost β”‚ └── architecture.md # pipeline + component map β”œβ”€β”€ CHANGELOG.md # release history (semver) β”œβ”€β”€ CONTRIBUTING.md # dev setup, PR conventions, release process β”œβ”€β”€ SECURITY.md # vulnerability reporting └── README.md # This file ``` --- --- ## βš–οΈ Copyright & fair use book-to-skill ships **no book content** β€” not a single page. It's a converter you point at files you already own. - **Processing is local.** Extraction and analysis run on your machine. Your files are never uploaded by this tool. (If your agent's model runs in the cloud, the text you feed it follows that provider's normal data terms β€” same as any prompt.) - **You use your own copy.** Bring a book you bought, docs your company owns, or papers you have the right to read. - **The output is your notes.** A generated skill is a structured, synthesized derivative β€” framework names, definitions, takeaways β€” not a reproduction of the text. The skill explicitly never copies raw passages (see Quality Rule #7). Treat it like handwritten study notes: yours, for personal use. - **Don't redistribute.** Publishing or sharing a generated skill of a copyrighted work can infringe the rights holder. Keep skills of third-party books private. Internal docs, your own writing, and openly-licensed material are fine to share within the bounds of their license. When in doubt, follow the license or terms of the source document. This project is a tool; how you use it is on you. --- ## πŸ’– Sponsors book-to-skill is free and MIT-licensed, maintained on personal time. If it saves you tokens or study hours, consider sponsoring its upkeep: PR reviews, multilingual fixes, releases, and docs. **[Become a sponsor β†’ github.com/sponsors/virgiliojr94](https://github.com/sponsors/virgiliojr94)** Every sponsor is listed in [BACKERS.md](BACKERS.md). Thank you for keeping open, privacy-first tooling alive. ✨ ## License MIT β€” applies to the converter (code + skill definition) in this repository, **not** to any book or document you process with it. --- ## File: docs/ARCHITECTURE.md --- seo_title: "Architecture - How book-to-skill Extracts and Generates" --- # Architecture book-to-skill has two halves: a **deterministic extractor** (Python) and a **spec-driven generator** (the agent following `SKILL.md`). The extractor turns any document into clean text + metadata; the agent turns that into a structured skill. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Design principles 1. **Extract structure, not summaries** β€” named frameworks, decision rules, anti-patterns; never raw passages. 2. **Compile-time over runtime** β€” pay navigation/structuring once; at query time load only the relevant chapter. See [Performance & Cost](performance.md). 3. **On-demand chapters** β€” `SKILL.md` stays small; chapter files cost tokens only when read. 4. **Front-loaded `SKILL.md`** β€” most important content first (compaction truncates from the end). 5. **Graceful degradation** β€” every format has a stdlib fallback; one bad source is skipped, not fatal. ## Key components | Path | Responsibility | |------|----------------| | `scripts/extract.py` | thin entrypoint shim β†’ `book_to_skill.cli` (kept so old invocations keep working) | | `book_to_skill/cli.py`, `utils.py` | CLI parsing, multi-source resolution, chapter/ToC detection, runner | | `book_to_skill/parsers/` | one module per format (`pdf`, `epub`, `docx`, `html`, `rtf`, `calibre`, `text`) | | `book_to_skill/config.py` | supported extensions, output paths, dependency map | | `book_to_skill/dependencies.py` | optional-dependency probing + `--check` | | `book_to_skill/sanitize.py` | strips zero-width / Unicode-tag-block characters from extracted text (see Security) | | `tools/discovery_tax.py` | measures token cost vs context-dump / discovery loop | | `tools/validate_skill.py` | checks a generated SKILL.md against host rules (`--lens claude\|copilot\|amp`) | | `tools/scan_generated_skill.py` | advisory prompt-injection scan of a generated skill (see Security) | | `SKILL.md` | the generator spec (Steps 0–10 + fold-in workflow) | ## Security Untrusted documents flow into an agent's context and then into a generated skill that later loads into other agents β€” a documentβ†’context supply chain. The hardening is layered: - **Extraction sanitization** (`book_to_skill/sanitize.py`) β€” strips zero-width (`U+200B/200C/200D/2060/FEFF`) and the Unicode tag block (`U+E0000–E007F`) from every parser's output before metrics or `full_text.txt`, so invisible document-borne instructions never reach the agent. Reports the removal count; rejects a source with no visible content left. - **DOCX XXE / Billion-Laughs guard** (`parsers/docx.py`) β€” rejects any XML part declaring a DTD or entities before parsing. - **Subprocess argument-injection** β€” file paths are absolutised before reaching `pdftotext` / `pdfinfo` / `ebook-convert`, so a `-`-leading filename can't be read as a flag. - **Generated-skill scan** (`tools/scan_generated_skill.py`) β€” an advisory step in the generator (Step 9.5) that flags instruction-override phrases, model-control tags, residual invisible Unicode, authority-widening frontmatter, and exfiltration-shaped content across the generated `SKILL.md`, `chapters/*.md`, `glossary.md`, `patterns.md`, and `cheatsheet.md`. Findings name only the rule and file location β€” never the matched text. - **CI** β€” CodeQL, Bandit (gate on HIGH), Zizmor, and dependency CVE review on PRs. ## Extending - **New format** β†’ add `book_to_skill/parsers/.py`, register its extension in `config.py`, wire dependency probing in `dependencies.py`, branch in `utils.extract_single_file`. - **New generation behavior** β†’ edit the relevant Step in `SKILL.md`; keep it lean and back the change with evidence (see CONTRIBUTING.md). --- ## File: docs/index.md --- seo_title: "book-to-skill - Turn Any Book Into an AI Agent Skill" hide: - navigation - toc --- # book-to-skill Turn any book or document into a structured, on-demand agent skill β€” named frameworks, decision rules, and anti-patterns. **Structure, not a summary.** [Get started](guide.md){ .md-button .md-button--primary } [Skill reference](skill-reference.md){ .md-button } [GitHub](https://github.com/virgiliojr94/book-to-skill){ .md-button } --- ## Why book-to-skill - :material-file-document-multiple:{ .lg .middle } __Multi-format__ --- PDF, EPUB, DOCX, HTML, Markdown, RTF, MOBI/AZW (via Calibre). Extraction runs locally with graceful stdlib fallbacks β€” no upload, no lock-in. - :material-brain:{ .lg .middle } __Structure, not summaries__ --- Named frameworks, mental models, decision rules, and anti-patterns β€” the author's toolkit, captured with their exact terms, not a book report. - :material-flash:{ .lg .middle } __On-demand chapters__ --- Per-chapter files load only when the topic is relevant, so a 200-page book costs tokens proportional to the question, not the page count. - :material-robot-happy:{ .lg .middle } __Multi-agent__ --- One `SKILL.md` runs on Claude Code, GitHub Copilot CLI, and Amp through the open Agent Skills standard. ## Install **As an agent skill** (gives you the `/book-to-skill` command in Claude Code, Copilot CLI, Amp): ```bash npx skills add virgiliojr94/book-to-skill # or manually: git clone https://github.com/virgiliojr94/book-to-skill.git ~/.claude/skills/book-to-skill # then, in your agent session: /book-to-skill /path/to/book.pdf [skill-name] ``` **As a standalone CLI** (just the text extractor, optional): ```bash # not on PyPI yet β€” pip takes it from the repository pip install "book-to-skill[pdf,epub,docx] @ git+https://github.com/virgiliojr94/book-to-skill.git" book-to-skill /path/to/book.pdf --mode text ``` ## Learn more - :material-sitemap:{ .lg .middle } __[Architecture](architecture.md)__ --- How the deterministic extractor and the spec-driven generator fit together. - :material-speedometer:{ .lg .middle } __[Performance](performance.md)__ --- The measured Discovery Loop Tax and real per-conversion token cost. - :material-book-open-page-variant:{ .lg .middle } __[Skill Reference](skill-reference.md)__ --- The full `SKILL.md` spec: every step, depth budget, and quality rule. - :material-heart:{ .lg .middle } __[Sponsor](https://github.com/sponsors/virgiliojr94)__ --- book-to-skill is free and MIT. Sponsoring funds reviews, releases, and fixes. --- ## File: docs/PERFORMANCE.md --- seo_title: "Performance & Token Cost - book-to-skill Benchmarks" --- # Performance & Cost All numbers below are **measured**, not estimated, using `tiktoken` (cl100k_base) for token counts and `tools/discovery_tax.py` for the discovery model. Reproduce any of them with the commands shown. ## Extraction (real conversions) Measured with `pdftotext` (PDF) and `ebooklib` (EPUB): | Book | Format | Pages | Tokens | Chapters auto-detected | |------|--------|------:|-------:|-----------------------:| | Think Python 2 | PDF | 244 | 119K | 19 | | Working Backwards | PDF | 371 | 175K | 10 | | Pro Git | PDF | 501 | 229K | β€” † | | Moby-Dick | EPUB | β€” | 301K | 133 | † Pro Git heads chapters with section titles (no `Chapter N`), so it does not auto-segment. Moby-Dick's bodies use bare titles, but its Roman-numeral table of contents is detected (133) β€” see *Known limitations* in the README. **Extraction method matters for technical books.** On a 103-page technical PDF: | Method | Time | Tables | Code blocks | |--------|-----:|-------:|------------:| | pdftotext | 0.1s | 0 | 0 | | Docling (technical mode) | 164s | 48 | 36 | pdftotext is instant but flattens structure; Docling is ~1.5s/page but preserves tables and code as markdown. Pick text mode for prose, technical mode for code/tables. ## The Discovery Loop Tax Tokens entering context to answer **one** targeted question. book-to-skill loads a resident core (~4K) plus one compiled chapter (~1K) β‰ˆ **5,000 tokens**. | Book (chapter size) | Context-dump | Discovery loop | book-to-skill | vs dump / loop | |---------------------|-------------:|---------------:|--------------:|:--------------:| | Think Python 2 (small) | 119,264 | 12,152 | ~5,000 | 24Γ— / 2.4Γ— | | Working Backwards (medium) | 175,253 | 33,444 | ~5,000 | 35Γ— / 6.7Γ— | | AI Engineering (large) | 256,287 | 77,866 | ~5,000 | 51Γ— / 15.6Γ— | ```bash python3 tools/discovery_tax.py --full-text /tmp/book_skill_work/full_text.txt --target-chapter 5 ``` - The **context-dump** advantage (24–51Γ—) is the strongest claim: that cost recurs on *every conversation turn*. - The **discovery-loop** advantage (2.4–15.6Γ—) is a one-time cost and a model using the book's real ToC/chapter sizes; it scales with chapter size. ## Generation cost One-pass full conversion, estimated from measured tokens (Claude Sonnet 4.5, \$3 / \$15 per MTok input/output): | Book | Input | Output | ~Cost | |------|------:|-------:|------:| | Think Python 2 | 155K | 28K | \$0.88 | | Working Backwards | 228K | 19K | \$0.96 | | Pro Git | 298K | 23K | \$1.23 | | Moby-Dick | 391K | 17K | \$1.42 | Roughly **\$1 per book** for a full skill β€” paid once. Re-reading the same PDF into context every session costs far more over time (see the Discovery Loop Tax above). ## Generated-skill output quality A before/after of the adaptive-depth change (`v1.0.0`, #20) on one chapter: | Artifact | Old spec | New spec | |----------|---------:|---------:| | Chapter file (tokens) | 473 | 1,219 | | Worked example present | no | yes | | Cheatsheet decision rules | 0 | 32 | | Cheatsheet keyword/definition lines | 9 | 0 | The new spec turns the cheatsheet from a glossary into a decision layer and gives study-depth chapters a reproduced worked example.