{"owner":"firecrawl","repo":"pdf-inspector","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md","CLAUDE.md"],"skills":{"AGENTS.md":"# pdf-inspector\n\nFast PDF text extraction to structured Markdown. CLI binary: `pdf2md`. Detection binary: `detect-pdf`.\n\n## Build & Test\n\n```bash\ncargo fmt                                    # format\ncargo clippy -- -D warnings                  # lint (enforced, zero warnings)\ncargo test                                   # unit + integration tests (267+ unit, 73+ integration)\ncargo build --release                        # release binary for benchmarks\n```\n\nAll three must pass before committing.\n\n## Binaries\n\n- `pdf2md` — extract PDF → Markdown. Supports `--json` for structured output.\n- `detect-pdf` — classify PDF type (TextBased/Scanned/Mixed/ImageBased). Supports `--analyze --json`.\n\n## Architecture\n\n```\nsrc/\n  lib.rs                        – public API, process_pdf_with_options, encoding issue detection\n  detector.rs                   – PDF type classification, tiled-scan detection, page sampling\n  types.rs                      – TextItem, TextLine, PdfRect, PdfLine\n  tounicode.rs                  – CMap/ToUnicode parsing, CID decoding\n  text_utils.rs                 – CJK/RTL handling, Otsu threshold, ligature expansion, NFKC\n  extractor/\n    mod.rs                      – top-level extraction orchestrator\n    content_stream.rs           – PDF operator state machine (Tj/TJ/Td/Tm/q/Q)\n    fonts.rs                    – font width/encoding, CMapDecisionCache, TrueType cmap fallback\n    layout.rs                   – column detection (histogram), newspaper/tabular classification,\n                                  spanning-line pre-masking, sidebar detection\n  tables/\n    detect_rects.rs             – rect-based table detection (union-find clustering)\n    detect_heuristic.rs         – heuristic table detection (gap-histogram, body-font tables)\n    detect_lines.rs             – line-based table detection (H/V line grids)\n    grid.rs                     – column/row boundaries, cell assignment\n    format.rs                   – table→Markdown formatting, continuation row merging\n  markdown/\n    convert.rs                  – core line→Markdown loop, struct-tree role support\n    analysis.rs                 – font stats, heading tiers, paragraph thresholds\n    classify.rs                 – line classification (header, list, code, caption)\n    preprocess.rs               – drop cap merging, heading line merging\n    postprocess.rs              – dot leaders, hyphenation, page numbers, URL formatting\n```\n\n## Key design decisions\n\n- **Primary audience is AI agents.** Output optimized for token efficiency and semantic quality, not visual formatting. No cosmetic padding.\n- **Three table detection strategies** run in priority order: rect-based → line-based → heuristic. First valid result wins.\n- **Column detection** uses horizontal projection histograms with valley detection. Multi-item spanning lines (titles, headers) are pre-masked using column-aware thresholds before column assignment.\n- **Newspaper vs tabular** classification determines reading order: newspaper reads columns sequentially, tabular Y-interleaves them.\n- **Tiled-scan detection** catches scanned PDFs with JBIG2/strip images where no single tile exceeds the template threshold but aggregate area does (≥2M pixels).\n- **Garbage text upgrade** reclassifies Mixed PDFs as Scanned when extracted text is <50% alphanumeric.\n- **Tagged PDF support** uses structure tree roles (H1-H6, P, L, Code, BlockQuote) when available, falling back to font-size heuristics.\n\n## Testing\n\n- **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.\n- **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.\n- **Regression suite**: sibling repo `pdf-evals` with ~200 snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing. While iterating, prefer a subset run (`bench.py test -q` for the quick set, or `-s <name>` for a named test set) and save the full `bench.py test` for the final pre-commit check.\n- **Semantic quality**: run `bench.py score` in `pdf-evals` for the semantic verdict (TEDS + MHS + reading order + char/word + list preservation, composited). Character-level diff alone misclassifies structural improvements (e.g., column-detection rewrites) as regressions — `score` is the tie-breaker. See `pdf-evals/CLAUDE.md` \"Semantic scoring\".\n\n## Debugging\n\n```bash\nRUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf\n```\n\n## Conventions\n\n- Clippy: use `is_some_and(...)` not `map_or(false, ...)`\n- lopdf quirk: `ParseError` is private — match by string for `InvalidFileHeader`\n- Column limit for tables: 25 (wide statistical tables)\n- `propagate_merged_cells` skipped for >10 columns (spanning rects = background fills)\n","CLAUDE.md":"# pdf-inspector\n\nFast PDF text extraction to structured Markdown. CLI binary: `pdf2md`. Detection binary: `detect-pdf`.\n\n## Build & Test\n\n```bash\ncargo fmt                                    # format\ncargo clippy -- -D warnings                  # lint (enforced, zero warnings)\ncargo test                                   # unit + integration tests (267+ unit, 73+ integration)\ncargo build --release                        # release binary for benchmarks\n```\n\nAll three must pass before committing.\n\n## Binaries\n\n- `pdf2md` — extract PDF → Markdown. Supports `--json` for structured output.\n- `detect-pdf` — classify PDF type (TextBased/Scanned/Mixed/ImageBased). Supports `--analyze --json`.\n\n## Architecture\n\n```\nsrc/\n  lib.rs                        – public API, process_pdf_with_options, encoding issue detection\n  detector.rs                   – PDF type classification, tiled-scan detection, page sampling\n  types.rs                      – TextItem, TextLine, PdfRect, PdfLine\n  tounicode.rs                  – CMap/ToUnicode parsing, CID decoding\n  text_utils.rs                 – CJK/RTL handling, Otsu threshold, ligature expansion, NFKC\n  extractor/\n    mod.rs                      – top-level extraction orchestrator\n    content_stream.rs           – PDF operator state machine (Tj/TJ/Td/Tm/q/Q)\n    fonts.rs                    – font width/encoding, CMapDecisionCache, TrueType cmap fallback\n    layout.rs                   – column detection (histogram), newspaper/tabular classification,\n                                  spanning-line pre-masking, sidebar detection\n  tables/\n    detect_rects.rs             – rect-based table detection (union-find clustering)\n    detect_heuristic.rs         – heuristic table detection (gap-histogram, body-font tables)\n    detect_lines.rs             – line-based table detection (H/V line grids)\n    grid.rs                     – column/row boundaries, cell assignment\n    format.rs                   – table→Markdown formatting, continuation row merging\n  markdown/\n    convert.rs                  – core line→Markdown loop, struct-tree role support\n    analysis.rs                 – font stats, heading tiers, paragraph thresholds\n    classify.rs                 – line classification (header, list, code, caption)\n    preprocess.rs               – drop cap merging, heading line merging\n    postprocess.rs              – dot leaders, hyphenation, page numbers, URL formatting\n```\n\n## Key design decisions\n\n- **Primary audience is AI agents.** Output optimized for token efficiency and semantic quality, not visual formatting. No cosmetic padding.\n- **Three table detection strategies** run in priority order: rect-based → line-based → heuristic. First valid result wins.\n- **Column detection** uses horizontal projection histograms with valley detection. Multi-item spanning lines (titles, headers) are pre-masked using column-aware thresholds before column assignment.\n- **Newspaper vs tabular** classification determines reading order: newspaper reads columns sequentially, tabular Y-interleaves them.\n- **Tiled-scan detection** catches scanned PDFs with JBIG2/strip images where no single tile exceeds the template threshold but aggregate area does (≥2M pixels).\n- **Garbage text upgrade** reclassifies Mixed PDFs as Scanned when extracted text is <50% alphanumeric.\n- **Tagged PDF support** uses structure tree roles (H1-H6, P, L, Code, BlockQuote) when available, falling back to font-size heuristics.\n\n## Testing\n\n- **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.\n- **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.\n- **Regression suite**: sibling repo `pdf-evals` with ~200 snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing. While iterating, prefer a subset run (`bench.py test -q` for the quick set, or `-s <name>` for a named test set) and save the full `bench.py test` for the final pre-commit check.\n- **Semantic quality**: run `bench.py score` in `pdf-evals` for the semantic verdict (TEDS + MHS + reading order + char/word + list preservation, composited). Character-level diff alone misclassifies structural improvements (e.g., column-detection rewrites) as regressions — `score` is the tie-breaker. See `pdf-evals/CLAUDE.md` \"Semantic scoring\".\n\n## Debugging\n\n```bash\nRUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf\n```\n\n## Conventions\n\n- Clippy: use `is_some_and(...)` not `map_or(false, ...)`\n- lopdf quirk: `ParseError` is private — match by string for `InvalidFileHeader`\n- Column limit for tables: 25 (wide statistical tables)\n- `propagate_merged_cells` skipped for >10 columns (spanning rects = background fills)\n"},"files":{"AGENTS.md":"# pdf-inspector\n\nFast PDF text extraction to structured Markdown. CLI binary: `pdf2md`. Detection binary: `detect-pdf`.\n\n## Build & Test\n\n```bash\ncargo fmt                                    # format\ncargo clippy -- -D warnings                  # lint (enforced, zero warnings)\ncargo test                                   # unit + integration tests (267+ unit, 73+ integration)\ncargo build --release                        # release binary for benchmarks\n```\n\nAll three must pass before committing.\n\n## Binaries\n\n- `pdf2md` — extract PDF → Markdown. Supports `--json` for structured output.\n- `detect-pdf` — classify PDF type (TextBased/Scanned/Mixed/ImageBased). Supports `--analyze --json`.\n\n## Architecture\n\n```\nsrc/\n  lib.rs                        – public API, process_pdf_with_options, encoding issue detection\n  detector.rs                   – PDF type classification, tiled-scan detection, page sampling\n  types.rs                      – TextItem, TextLine, PdfRect, PdfLine\n  tounicode.rs                  – CMap/ToUnicode parsing, CID decoding\n  text_utils.rs                 – CJK/RTL handling, Otsu threshold, ligature expansion, NFKC\n  extractor/\n    mod.rs                      – top-level extraction orchestrator\n    content_stream.rs           – PDF operator state machine (Tj/TJ/Td/Tm/q/Q)\n    fonts.rs                    – font width/encoding, CMapDecisionCache, TrueType cmap fallback\n    layout.rs                   – column detection (histogram), newspaper/tabular classification,\n                                  spanning-line pre-masking, sidebar detection\n  tables/\n    detect_rects.rs             – rect-based table detection (union-find clustering)\n    detect_heuristic.rs         – heuristic table detection (gap-histogram, body-font tables)\n    detect_lines.rs             – line-based table detection (H/V line grids)\n    grid.rs                     – column/row boundaries, cell assignment\n    format.rs                   – table→Markdown formatting, continuation row merging\n  markdown/\n    convert.rs                  – core line→Markdown loop, struct-tree role support\n    analysis.rs                 – font stats, heading tiers, paragraph thresholds\n    classify.rs                 – line classification (header, list, code, caption)\n    preprocess.rs               – drop cap merging, heading line merging\n    postprocess.rs              – dot leaders, hyphenation, page numbers, URL formatting\n```\n\n## Key design decisions\n\n- **Primary audience is AI agents.** Output optimized for token efficiency and semantic quality, not visual formatting. No cosmetic padding.\n- **Three table detection strategies** run in priority order: rect-based → line-based → heuristic. First valid result wins.\n- **Column detection** uses horizontal projection histograms with valley detection. Multi-item spanning lines (titles, headers) are pre-masked using column-aware thresholds before column assignment.\n- **Newspaper vs tabular** classification determines reading order: newspaper reads columns sequentially, tabular Y-interleaves them.\n- **Tiled-scan detection** catches scanned PDFs with JBIG2/strip images where no single tile exceeds the template threshold but aggregate area does (≥2M pixels).\n- **Garbage text upgrade** reclassifies Mixed PDFs as Scanned when extracted text is <50% alphanumeric.\n- **Tagged PDF support** uses structure tree roles (H1-H6, P, L, Code, BlockQuote) when available, falling back to font-size heuristics.\n\n## Testing\n\n- **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.\n- **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.\n- **Regression suite**: sibling repo `pdf-evals` with ~200 snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing. While iterating, prefer a subset run (`bench.py test -q` for the quick set, or `-s <name>` for a named test set) and save the full `bench.py test` for the final pre-commit check.\n- **Semantic quality**: run `bench.py score` in `pdf-evals` for the semantic verdict (TEDS + MHS + reading order + char/word + list preservation, composited). Character-level diff alone misclassifies structural improvements (e.g., column-detection rewrites) as regressions — `score` is the tie-breaker. See `pdf-evals/CLAUDE.md` \"Semantic scoring\".\n\n## Debugging\n\n```bash\nRUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf\n```\n\n## Conventions\n\n- Clippy: use `is_some_and(...)` not `map_or(false, ...)`\n- lopdf quirk: `ParseError` is private — match by string for `InvalidFileHeader`\n- Column limit for tables: 25 (wide statistical tables)\n- `propagate_merged_cells` skipped for >10 columns (spanning rects = background fills)\n","CLAUDE.md":"# pdf-inspector\n\nFast PDF text extraction to structured Markdown. CLI binary: `pdf2md`. Detection binary: `detect-pdf`.\n\n## Build & Test\n\n```bash\ncargo fmt                                    # format\ncargo clippy -- -D warnings                  # lint (enforced, zero warnings)\ncargo test                                   # unit + integration tests (267+ unit, 73+ integration)\ncargo build --release                        # release binary for benchmarks\n```\n\nAll three must pass before committing.\n\n## Binaries\n\n- `pdf2md` — extract PDF → Markdown. Supports `--json` for structured output.\n- `detect-pdf` — classify PDF type (TextBased/Scanned/Mixed/ImageBased). Supports `--analyze --json`.\n\n## Architecture\n\n```\nsrc/\n  lib.rs                        – public API, process_pdf_with_options, encoding issue detection\n  detector.rs                   – PDF type classification, tiled-scan detection, page sampling\n  types.rs                      – TextItem, TextLine, PdfRect, PdfLine\n  tounicode.rs                  – CMap/ToUnicode parsing, CID decoding\n  text_utils.rs                 – CJK/RTL handling, Otsu threshold, ligature expansion, NFKC\n  extractor/\n    mod.rs                      – top-level extraction orchestrator\n    content_stream.rs           – PDF operator state machine (Tj/TJ/Td/Tm/q/Q)\n    fonts.rs                    – font width/encoding, CMapDecisionCache, TrueType cmap fallback\n    layout.rs                   – column detection (histogram), newspaper/tabular classification,\n                                  spanning-line pre-masking, sidebar detection\n  tables/\n    detect_rects.rs             – rect-based table detection (union-find clustering)\n    detect_heuristic.rs         – heuristic table detection (gap-histogram, body-font tables)\n    detect_lines.rs             – line-based table detection (H/V line grids)\n    grid.rs                     – column/row boundaries, cell assignment\n    format.rs                   – table→Markdown formatting, continuation row merging\n  markdown/\n    convert.rs                  – core line→Markdown loop, struct-tree role support\n    analysis.rs                 – font stats, heading tiers, paragraph thresholds\n    classify.rs                 – line classification (header, list, code, caption)\n    preprocess.rs               – drop cap merging, heading line merging\n    postprocess.rs              – dot leaders, hyphenation, page numbers, URL formatting\n```\n\n## Key design decisions\n\n- **Primary audience is AI agents.** Output optimized for token efficiency and semantic quality, not visual formatting. No cosmetic padding.\n- **Three table detection strategies** run in priority order: rect-based → line-based → heuristic. First valid result wins.\n- **Column detection** uses horizontal projection histograms with valley detection. Multi-item spanning lines (titles, headers) are pre-masked using column-aware thresholds before column assignment.\n- **Newspaper vs tabular** classification determines reading order: newspaper reads columns sequentially, tabular Y-interleaves them.\n- **Tiled-scan detection** catches scanned PDFs with JBIG2/strip images where no single tile exceeds the template threshold but aggregate area does (≥2M pixels).\n- **Garbage text upgrade** reclassifies Mixed PDFs as Scanned when extracted text is <50% alphanumeric.\n- **Tagged PDF support** uses structure tree roles (H1-H6, P, L, Code, BlockQuote) when available, falling back to font-size heuristics.\n\n## Testing\n\n- **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.\n- **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.\n- **Regression suite**: sibling repo `pdf-evals` with ~200 snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing. While iterating, prefer a subset run (`bench.py test -q` for the quick set, or `-s <name>` for a named test set) and save the full `bench.py test` for the final pre-commit check.\n- **Semantic quality**: run `bench.py score` in `pdf-evals` for the semantic verdict (TEDS + MHS + reading order + char/word + list preservation, composited). Character-level diff alone misclassifies structural improvements (e.g., column-detection rewrites) as regressions — `score` is the tie-breaker. See `pdf-evals/CLAUDE.md` \"Semantic scoring\".\n\n## Debugging\n\n```bash\nRUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf\n```\n\n## Conventions\n\n- Clippy: use `is_some_and(...)` not `map_or(false, ...)`\n- lopdf quirk: `ParseError` is private — match by string for `InvalidFileHeader`\n- Column limit for tables: 25 (wide statistical tables)\n- `propagate_merged_cells` skipped for >10 columns (spanning rects = background fills)\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","title":"AGENTS.md","content":"# pdf-inspector\n\nFast PDF text extraction to structured Markdown. CLI binary: `pdf2md`. Detection binary: `detect-pdf`.\n\n## Build & Test\n\n```bash\ncargo fmt                                    # format\ncargo clippy -- -D warnings                  # lint (enforced, zero warnings)\ncargo test                                   # unit + integration tests (267+ unit, 73+ integration)\ncargo build --release                        # release binary for benchmarks\n```\n\nAll three must pass before committing.\n\n## Binaries\n\n- `pdf2md` — extract PDF → Markdown. Supports `--json` for structured output.\n- `detect-pdf` — classify PDF type (TextBased/Scanned/Mixed/ImageBased). Supports `--analyze --json`.\n\n## Architecture\n\n```\nsrc/\n  lib.rs                        – public API, process_pdf_with_options, encoding issue detection\n  detector.rs                   – PDF type classification, tiled-scan detection, page sampling\n  types.rs                      – TextItem, TextLine, PdfRect, PdfLine\n  tounicode.rs                  – CMap/ToUnicode parsing, CID decoding\n  text_utils.rs                 – CJK/RTL handling, Otsu threshold, ligature expansion, NFKC\n  extractor/\n    mod.rs                      – top-level extraction orchestrator\n    content_stream.rs           – PDF operator state machine (Tj/TJ/Td/Tm/q/Q)\n    fonts.rs                    – font width/encoding, CMapDecisionCache, TrueType cmap fallback\n    layout.rs                   – column detection (histogram), newspaper/tabular classification,\n                                  spanning-line pre-masking, sidebar detection\n  tables/\n    detect_rects.rs             – rect-based table detection (union-find clustering)\n    detect_heuristic.rs         – heuristic table detection (gap-histogram, body-font tables)\n    detect_lines.rs             – line-based table detection (H/V line grids)\n    grid.rs                     – column/row boundaries, cell assignment\n    format.rs                   – table→Markdown formatting, continuation row merging\n  markdown/\n    convert.rs                  – core line→Markdown loop, struct-tree role support\n    analysis.rs                 – font stats, heading tiers, paragraph thresholds\n    classify.rs                 – line classification (header, list, code, caption)\n    preprocess.rs               – drop cap merging, heading line merging\n    postprocess.rs              – dot leaders, hyphenation, page numbers, URL formatting\n```\n\n## Key design decisions\n\n- **Primary audience is AI agents.** Output optimized for token efficiency and semantic quality, not visual formatting. No cosmetic padding.\n- **Three table detection strategies** run in priority order: rect-based → line-based → heuristic. First valid result wins.\n- **Column detection** uses horizontal projection histograms with valley detection. Multi-item spanning lines (titles, headers) are pre-masked using column-aware thresholds before column assignment.\n- **Newspaper vs tabular** classification determines reading order: newspaper reads columns sequentially, tabular Y-interleaves them.\n- **Tiled-scan detection** catches scanned PDFs with JBIG2/strip images where no single tile exceeds the template threshold but aggregate area does (≥2M pixels).\n- **Garbage text upgrade** reclassifies Mixed PDFs as Scanned when extracted text is <50% alphanumeric.\n- **Tagged PDF support** uses structure tree roles (H1-H6, P, L, Code, BlockQuote) when available, falling back to font-size heuristics.\n\n## Testing\n\n- **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.\n- **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.\n- **Regression suite**: sibling repo `pdf-evals` with ~200 snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing. While iterating, prefer a subset run (`bench.py test -q` for the quick set, or `-s <name>` for a named test set) and save the full `bench.py test` for the final pre-commit check.\n- **Semantic quality**: run `bench.py score` in `pdf-evals` for the semantic verdict (TEDS + MHS + reading order + char/word + list preservation, composited). Character-level diff alone misclassifies structural improvements (e.g., column-detection rewrites) as regressions — `score` is the tie-breaker. See `pdf-evals/CLAUDE.md` \"Semantic scoring\".\n\n## Debugging\n\n```bash\nRUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf\n```\n\n## Conventions\n\n- Clippy: use `is_some_and(...)` not `map_or(false, ...)`\n- lopdf quirk: `ParseError` is private — match by string for `InvalidFileHeader`\n- Column limit for tables: 25 (wide statistical tables)\n- `propagate_merged_cells` skipped for >10 columns (spanning rects = background fills)\n","category":"root","tokens":1227},{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# pdf-inspector\n\nFast PDF text extraction to structured Markdown. CLI binary: `pdf2md`. Detection binary: `detect-pdf`.\n\n## Build & Test\n\n```bash\ncargo fmt                                    # format\ncargo clippy -- -D warnings                  # lint (enforced, zero warnings)\ncargo test                                   # unit + integration tests (267+ unit, 73+ integration)\ncargo build --release                        # release binary for benchmarks\n```\n\nAll three must pass before committing.\n\n## Binaries\n\n- `pdf2md` — extract PDF → Markdown. Supports `--json` for structured output.\n- `detect-pdf` — classify PDF type (TextBased/Scanned/Mixed/ImageBased). Supports `--analyze --json`.\n\n## Architecture\n\n```\nsrc/\n  lib.rs                        – public API, process_pdf_with_options, encoding issue detection\n  detector.rs                   – PDF type classification, tiled-scan detection, page sampling\n  types.rs                      – TextItem, TextLine, PdfRect, PdfLine\n  tounicode.rs                  – CMap/ToUnicode parsing, CID decoding\n  text_utils.rs                 – CJK/RTL handling, Otsu threshold, ligature expansion, NFKC\n  extractor/\n    mod.rs                      – top-level extraction orchestrator\n    content_stream.rs           – PDF operator state machine (Tj/TJ/Td/Tm/q/Q)\n    fonts.rs                    – font width/encoding, CMapDecisionCache, TrueType cmap fallback\n    layout.rs                   – column detection (histogram), newspaper/tabular classification,\n                                  spanning-line pre-masking, sidebar detection\n  tables/\n    detect_rects.rs             – rect-based table detection (union-find clustering)\n    detect_heuristic.rs         – heuristic table detection (gap-histogram, body-font tables)\n    detect_lines.rs             – line-based table detection (H/V line grids)\n    grid.rs                     – column/row boundaries, cell assignment\n    format.rs                   – table→Markdown formatting, continuation row merging\n  markdown/\n    convert.rs                  – core line→Markdown loop, struct-tree role support\n    analysis.rs                 – font stats, heading tiers, paragraph thresholds\n    classify.rs                 – line classification (header, list, code, caption)\n    preprocess.rs               – drop cap merging, heading line merging\n    postprocess.rs              – dot leaders, hyphenation, page numbers, URL formatting\n```\n\n## Key design decisions\n\n- **Primary audience is AI agents.** Output optimized for token efficiency and semantic quality, not visual formatting. No cosmetic padding.\n- **Three table detection strategies** run in priority order: rect-based → line-based → heuristic. First valid result wins.\n- **Column detection** uses horizontal projection histograms with valley detection. Multi-item spanning lines (titles, headers) are pre-masked using column-aware thresholds before column assignment.\n- **Newspaper vs tabular** classification determines reading order: newspaper reads columns sequentially, tabular Y-interleaves them.\n- **Tiled-scan detection** catches scanned PDFs with JBIG2/strip images where no single tile exceeds the template threshold but aggregate area does (≥2M pixels).\n- **Garbage text upgrade** reclassifies Mixed PDFs as Scanned when extracted text is <50% alphanumeric.\n- **Tagged PDF support** uses structure tree roles (H1-H6, P, L, Code, BlockQuote) when available, falling back to font-size heuristics.\n\n## Testing\n\n- **Unit tests**: inline `#[cfg(test)] mod tests` in each module with synthetic data.\n- **Integration tests**: `tests/integration_tests.rs` with fixture PDFs in `tests/fixtures/`.\n- **Regression suite**: sibling repo `pdf-evals` with ~200 snapshot PDFs. Run `cargo build --release` then `bench.py test` in that repo before committing. While iterating, prefer a subset run (`bench.py test -q` for the quick set, or `-s <name>` for a named test set) and save the full `bench.py test` for the final pre-commit check.\n- **Semantic quality**: run `bench.py score` in `pdf-evals` for the semantic verdict (TEDS + MHS + reading order + char/word + list preservation, composited). Character-level diff alone misclassifies structural improvements (e.g., column-detection rewrites) as regressions — `score` is the tie-breaker. See `pdf-evals/CLAUDE.md` \"Semantic scoring\".\n\n## Debugging\n\n```bash\nRUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf\nRUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf\n```\n\n## Conventions\n\n- Clippy: use `is_some_and(...)` not `map_or(false, ...)`\n- lopdf quirk: `ParseError` is private — match by string for `InvalidFileHeader`\n- Column limit for tables: 25 (wide statistical tables)\n- `propagate_merged_cells` skipped for >10 columns (spanning rects = background fills)\n","category":"root","tokens":1227}]}