pdf-inspector Documentation

File: README.md Copy Chapter

pdf-inspector

Fast Rust library for PDF classification and text extraction. By default it detects whether a PDF is text-based or scanned, extracts text with position awareness, and converts to clean Markdown without OCR. Native Rust and CLI consumers can opt into selective OCR. Includes bindings for Python, Node.js, and browser WebAssembly.

Built by Firecrawl to handle text-based PDFs locally in under 200ms, skipping expensive OCR services for the ~54% of PDFs that don't need them.

Features Copy Chapter

• Smart classification — Detect TextBased, Scanned, ImageBased, or Mixed PDFs in ~10-50ms by sampling content streams. Returns a confidence score (0.0-1.0) and per-page OCR routing.
• Text extraction — Position-aware extraction with font info, X/Y coordinates, and automatic multi-column reading order.
• Markdown conversion — Headings (H1-H4 via font size ratios), bullet/numbered/letter lists, code blocks (monospace font detection), tables (rectangle-based and heuristic), bold/italic formatting, URL linking, and page breaks.
• Table detection — Dual-mode: rectangle-based detection from PDF drawing ops, plus heuristic detection from text alignment. Handles financial tables, footnotes, and continuation tables across pages.
• CID font support — ToUnicode CMap decoding for Type0/Identity-H fonts, UTF-16BE, UTF-8, and Latin-1 encodings.
• Multi-column layout — Automatic detection of newspaper-style columns, sequential reading order, and RTL text support.
• Encoding issue detection — Automatically flags broken font encodings so callers can fall back to OCR.
• Selective OCR — Rust, CLI, Python, and Node can render only pages that need OCR, run PP-OCRv6 Small locally, and preserve per-page provenance and hosted-fallback recommendations.
• Single document load — The document is parsed once and shared between detection and extraction, avoiding redundant I/O.
• Browser WebAssembly — Run the same Rust parser locally in browsers and Web Workers, with embedded CMaps and no server round trip.
• Lightweight by default — The default Rust and browser builds remain pure extraction. Native Python and Node packages include the OCR integration, but PDFium, ONNX Runtime, and model files remain external and are touched only when a page is routed to OCR.

Benchmark Copy Chapter

Evaluated on the opendataloader-bench corpus (200 PDFs). Only local engines without model-based PDF parsing are shown; OCR was disabled. Scores are 0-1, higher is better.

Results were refreshed on July 31, 2026, on an Apple M4 Pro. Engine versions were pdf-inspector 0.2.6, LiteParse 2.10.1, OpenDataLoader 2.2.1, PyMuPDF4LLM 0.2.0, and MarkItDown 0.1.5. Speed is the median of five alternating or rotating complete corpus runs after an excluded warm-up run, with each parser processing documents sequentially in a single process.

The complete parser configuration, per-document predictions, evaluator output, and generated charts are available in the reproducible results branch.

Best fit: Native-text PDFs where speed, reading order, and table structure matter. In this comparison, pdf-inspector delivered the higher overall, reading-order, and table scores, along with the fastest complete run. That makes it a strong local default for reports, research papers, financial documents, invoices, and legal PDFs that need clean, structured Markdown without adding OCR latency or infrastructure.

Use the paired benchmark harness to compare two local builds against the exact same corpus and evaluator revision.

Quick start Copy Chapter

Python Copy Section

pip install pdf-inspector

import pdf_inspector

result = pdf_inspector.process_pdf("document.pdf")
print(result.pdf_type)   # "text_based", "scanned", "image_based", "mixed"
print(result.markdown)   # Markdown string or None

# Selective OCR; clean text PDFs do not load the external OCR runtime.
ocr = pdf_inspector.process_pdf_with_ocr("document.pdf")
print(ocr.pages_routed_to_ocr)

Full API reference: docs/python.md

Node.js Copy Section

npm install @firecrawl/pdf-inspector

import { readFileSync } from 'fs';
import { processPdf, processPdfWithOcr } from '@firecrawl/pdf-inspector';

const pdf = readFileSync('document.pdf');
const result = processPdf(pdf);
console.log(result.pdfType);   // "TextBased", "Scanned", "ImageBased", "Mixed"
console.log(result.markdown);  // Markdown string or null

const ocr = await processPdfWithOcr(pdf); // selective OCR, off the event loop
console.log(ocr.pagesRoutedToOcr);

Full API reference: napi/README.md

Browser WebAssembly Copy Section

npm install @firecrawl/pdf-inspector-wasm

import init, { processPdf } from '@firecrawl/pdf-inspector-wasm';

await init();
const response = await fetch('/document.pdf');
const pdf = new Uint8Array(await response.arrayBuffer());
const result = processPdf(pdf);

console.log(result.pdfType);
console.log(result.markdown);

Full API reference: wasm/README.md

Rust Copy Section

Install from crates.io:

cargo add pdf-inspector

Or add it manually:

[dependencies]
pdf-inspector = "1"

use pdf_inspector::process_pdf;

let result = process_pdf("document.pdf")?;
println!("Type: {:?}", result.pdf_type);
if let Some(markdown) = &result.markdown {
    println!("{}", markdown);
}

Full API reference: docs/rust-api.md

CLI Copy Section

# Install the CLI tools
cargo install pdf-inspector

# Convert PDF to Markdown
pdf2md document.pdf

# JSON output (for piping)
pdf2md document.pdf --json

# Positioned TextItem JSON, including is_underline metadata
pdf2md document.pdf --items-json

# Raw markdown only (no headers)
pdf2md document.pdf --raw

# Token-efficient output (collapses long dot leaders and similar source padding)
pdf2md document.pdf --compact

# Insert page break markers ()
pdf2md document.pdf --pages

# Process only specific pages
pdf2md document.pdf --select-pages 1,3,5-10

# Detection only (no extraction)
detect-pdf document.pdf
detect-pdf document.pdf --json

# Detection + layout analysis (tables, columns)
detect-pdf document.pdf --analyze --json

Rust and CLI consumers opt into OCR at build time:

cargo install pdf-inspector --features ocr --bin pdf2md
PDFIUM_LIB_PATH=/path/to/libpdfium ORT_DYLIB_PATH=/path/to/libonnxruntime \
  pdf2md scan.pdf --ocr auto --json

The OCR JSON envelope is versioned and reports routed pages, per-page source
and confidence, warnings, and pages recommended for the hosted document
pipeline. Native Python and Node packages expose the same pipeline without a
source-build feature. All native entry points still require separately
installed PDFium and ONNX Runtime libraries only when OCR is routed. See the
OCR runtime setup guide for pinned downloads, platform
support, model-cache behavior, and hosted-fallback integration. See the
Rust API guide for lower-level controls.

From a source checkout, use cargo run --bin pdf2md -- document.pdf or cargo run --bin detect-pdf -- document.pdf instead.

Architecture Copy Chapter

PDF bytes
  │
  ├─► detector         → PdfType (TextBased / Scanned / ImageBased / Mixed)
  │
  └─► extractor
        ├─ fonts        → font widths, encodings
        ├─ content_stream → walk PDF operators → TextItems + PdfRects
        ├─ xobjects     → Form XObject text, image placeholders
        ├─ links        → hyperlinks, AcroForm fields
        └─ layout       → column detection → line grouping → reading order
              │
              ├─► tables
              │     ├─ detect_rects      → rectangle-based tables (union-find)
              │     ├─ detect_heuristic  → alignment-based tables
              │     ├─ grid              → column/row assignment → cells
              │     └─ format            → cells → Markdown table
              │
              └─► markdown
                    ├─ analysis     → font stats, heading tiers
                    ├─ preprocess   → merge headings, drop caps
                    ├─ convert      → line loop + table/image insertion
                    ├─ classify     → captions, lists, code
                    └─ postprocess  → cleanup → final Markdown

The document is loaded once via load_document_from_path / load_document_from_mem and shared between the detection and extraction stages, so there's no redundant parsing.

Project structure Copy Section

src/
  lib.rs                — Public API, PdfOptions builder, convenience functions
  python.rs             — PyO3 Python bindings
  types.rs              — Shared types: TextItem, TextLine, PdfRect, ItemType
  text_utils.rs         — Character/text helpers (CJK, RTL, ligatures, bold/italic)
  process_mode.rs       — ProcessMode enum (DetectOnly, Analyze, Full)
  detector.rs           — Fast PDF type detection without full document load
  glyph_names.rs        — Adobe Glyph List → Unicode mapping
  tounicode.rs          — ToUnicode CMap parsing for CID-encoded text
  extractor/            — Text extraction pipeline
  tables/               — Table detection and formatting
  markdown/             — Markdown conversion and structure detection
  bin/                  — CLI tools (pdf2md, detect_pdf)
napi/                   — Node.js/Bun bindings (napi-rs)
wasm/                   — Browser bindings (wasm-bindgen)

How classification works Copy Chapter

1. Parse the xref table and page tree (no full object load)
2. Select pages based on ScanStrategy (default: all pages with early exit)
3. Look for Tj/TJ (text operators) and Do (image operators) in content streams
4. Classify based on text operator presence across sampled pages

This detects 300+ page PDFs in milliseconds. The result includes pages_needing_ocr — a list of specific page numbers that lack text, enabling per-page OCR routing instead of all-or-nothing.

Scan strategies Copy Section

Markdown output Copy Chapter

The converter handles:

Use case: smart PDF routing Copy Chapter

pdf-inspector was built for pipelines that process PDFs at scale. Instead of sending every PDF through OCR:

PDF arrives
  → pdf-inspector classifies it (~20ms)
  → TextBased + high confidence?
      YES → extract locally (~150ms), done
      NO  → send to OCR service (2-10s)

This saves cost and latency for the majority of PDFs that are already text-based (reports, papers, invoices, legal docs).

Debugging Copy Chapter

See docs/debugging.md for RUST_LOG environment variable usage.

License Copy Chapter

MIT

File: docs/benchmarking.md Copy Chapter

Benchmarking against OpenDataLoader

The paired harness runs two pdf2md binaries through the same local
OpenDataLoader corpus, evaluates both outputs, and reports aggregate and
per-document deltas. This avoids comparing results produced from different
corpus revisions or evaluator versions.

Build a candidate and provide a released or worktree build as the baseline:

cargo build --release
python3 scripts/bench_opendataloader.py \
  --bench-dir ../opendataloader-bench \
  --baseline ../pdf-inspector-main/target/release/pdf2md \
  --candidate target/release/pdf2md \
  --max-document-regression 0.02 \
  --json-output /tmp/pdf-inspector-benchmark.json

Pass --reference-evaluation path/to/evaluation.json to report the candidate
delta against another evaluation, and add --require-reference-lead to make a
negative reference delta fail the run. By default, the candidate must not
regress the baseline overall score or introduce missing predictions. Use
--min-overall-delta to require a specific aggregate gain.

The OpenDataLoader repository is external and keeps its normal
prediction/pdf-inspector output. Paired evaluation copies each run into a
temporary directory before evaluating it, so the baseline and candidate cannot
overwrite one another.

Published comparison protocol Copy Chapter

The public benchmark table was refreshed on July 31, 2026, on an Apple M4 Pro
using pdf-inspector 0.2.6, LiteParse 2.10.1, OpenDataLoader 2.2.1,
PyMuPDF4LLM 0.2.0, and MarkItDown 0.1.5. Every engine processed the same 200
PDFs sequentially in a single process with OCR disabled. Reported speed is the
median of five alternating or rotating complete corpus runs after an excluded
warm-up run; quality scores come from the benchmark evaluator over all 200
outputs. Raw timings, predictions, evaluations, and charts are available in the
results branch.

Optional backend evidence probe Copy Chapter

The evidence probe compares positioned pdf2md items with MuPDF structured
text on the same pages. It is intended to find deterministic extraction or
layout evidence that could justify a future native implementation; it does not
merge MuPDF output into Markdown, invoke OCR, or add a runtime dependency.

Install MuPDF's mutool, build pdf2md, then run:

python3 scripts/probe_backend_evidence.py document.pdf \
  --pdf2md target/release/pdf2md \
  --json-output /tmp/backend-evidence.json

The report flags pages when MuPDF exposes a material net token gain, repeated
alignment anchors absent from local evidence, or additional image blocks. The
JSON includes bounded token samples and page-level counts so promising cases
can be inspected without treating backend disagreement as automatically
correct. Thresholds are configurable with --min-token-gain,
--min-alternate-only-ratio, and --min-anchor-gain.

File: docs/debugging.md Copy Chapter

Debugging with RUST_LOG

Structured logging via RUST_LOG replaces the former debug binaries. Set the environment variable to control which sections emit debug output on stderr:

# Raw PDF content stream operators (replaces dump_ops)
RUST_LOG=pdf_inspector::extractor::content_stream=trace cargo run --bin pdf2md -- file.pdf > /dev/null

# Font metadata, encodings, ligatures (replaces debug_fonts / debug_ligatures)
RUST_LOG=pdf_inspector::extractor::fonts=debug cargo run --bin pdf2md -- file.pdf > /dev/null

# ToUnicode CMap parsing
RUST_LOG=pdf_inspector::tounicode=debug cargo run --bin pdf2md -- file.pdf > /dev/null

# Text items per page with x/y/width (replaces debug_spaces / debug_pages)
RUST_LOG=pdf_inspector::extractor=debug cargo run --bin pdf2md -- file.pdf > /dev/null

# Column detection and reading order (replaces debug_order)
RUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf > /dev/null

# Y-gap analysis and paragraph thresholds (replaces debug_ygaps)
RUST_LOG=pdf_inspector::markdown::analysis=debug cargo run --bin pdf2md -- file.pdf > /dev/null

# Table detection
RUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf > /dev/null

# Everything
RUST_LOG=pdf_inspector=debug cargo run --bin pdf2md -- file.pdf > /dev/null

File: docs/publishing.md Copy Chapter

Publishing

Every pdf-inspector distribution uses one shared semantic version:

• Rust crate: pdf-inspector
• Python package: pdf-inspector
• Node package: @firecrawl/pdf-inspector and its platform packages
• Browser package: @firecrawl/pdf-inspector-wasm
• Internal NAPI and WASM Rust crates

Cargo.toml is the canonical version source. Update every manifest and lockfile
with:

python3 scripts/version.py <version>

Verify that nothing has diverged with:

python3 scripts/version.py --check

CI and every publishing workflow run this check before building or publishing.

Release steps Copy Chapter

1. Choose the next shared semantic version and run scripts/version.py.
2. Review the manifest and lockfile changes in the version-bump pull request.
3. Merge the pull request to main.
4. The crates.io, PyPI, Node, and WASM workflows independently build and
   publish that version from the same commit.
5. After all registries succeed, create one v<version> GitHub release that
   links to each package and describes changes since the previous shared tag.

The independent workflows are intentionally idempotent. A manual dispatch from
main can repair a partial release, and already-published artifacts are skipped.

Trusted publishers Copy Chapter

The repositories use GitHub Actions OIDC instead of long-lived registry tokens.
Configure each registry's trusted publisher for firecrawl/pdf-inspector and
its corresponding workflow:

• crates.io: publish-crate.yml, environment crates-io
• PyPI: publish-pypi.yml, environment pypi
• npm Node package: publish.yml
• npm WASM package: publish-wasm.yml

The WASM package must exist before npm trusted publishing can be configured. If
it ever needs to be bootstrapped again, build and inspect it before publishing:

wasm-pack build wasm --target web --scope firecrawl --out-dir pkg --release
npm pack --dry-run ./wasm/pkg
npm publish ./wasm/pkg --access public

File: docs/python.md Copy Chapter

pdf-inspector

Fast PDF classification, text extraction, and selective OCR. Detects whether a PDF is text-based or scanned, extracts text with position awareness, and converts clean native and OCR results to Markdown. Python bindings via PyO3 for the pdf-inspector Rust library.

Built by Firecrawl to handle text-based PDFs locally in under 200ms, skipping expensive OCR services for the ~54% of PDFs that don't need them.

Features Copy Chapter

• Smart classification — text_based / scanned / image_based / mixed in ~10–50ms, with a confidence score and per-page OCR routing.
• Markdown conversion — headings, lists, code blocks, bold/italic, URL linking, and dual-mode table detection (PDF drawing ops + text-alignment heuristics).
• Layout-aware extraction — multi-column reading order, position and font info per text item, RTL support.
• Robust text decoding — CID/Type0 fonts via ToUnicode CMaps, plus automatic flagging of broken encodings so callers can fall back to OCR.
• Selective OCR — auto routes only pages rejected by native extraction; force OCRs every selected page; off keeps the result/provenance contract without external runtime work.
• External artifacts — the wheel embeds no OCR models, PDFium, or ONNX Runtime; clean auto requests never load or download them.

Benchmark Copy Chapter

opendataloader-bench corpus (200 PDFs), local engines without model-based PDF parsing; OCR disabled. Scores 0–1, higher is better:

Refreshed July 31, 2026, on Apple M4 Pro; speed is the median of five complete corpus runs after an excluded warm-up. Full methodology and versions are in the repo README, with raw timings and artifacts in the results branch.

Install Copy Chapter

pip install pdf-inspector

Prebuilt wheels cover CPython ≥3.8 on Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), and Windows (x64). Other platforms build from source, which requires a Rust toolchain. For local development in a repo checkout:

pip install maturin
maturin develop --release

OCR calls that route work require compatible PDFium and ONNX Runtime shared
libraries. Set PDFIUM_LIB_PATH and ORT_DYLIB_PATH when they are not on the
platform library search path. The pinned OCR model set is downloaded and
checksum-verified on the first routed page; use offline=True with a warm
cache or model_directory to prohibit network access. See the
OCR runtime setup guide
for pinned downloads, supported platforms, and hosted-fallback behavior.

Usage Copy Chapter

/* Detailed source-code truncated for AI context efficiency. */

API reference Copy Chapter

Types Copy Chapter

Type stubs (pdf_inspector.pyi) ship with the package. Result types at a glance:

/* Detailed source-code truncated for AI context efficiency. */

File: docs/rust-api.md Copy Chapter

pdf-inspector

Fast PDF classification and text extraction. The default build detects whether a PDF is text-based or scanned, extracts text with position awareness, and converts to clean Markdown without OCR. It is pure Rust, has no ML models or external services, and uses lopdf for PDF parsing. Native Rust and CLI consumers can opt into selective OCR. Also available for Python and Node.js.

Built by Firecrawl to handle text-based PDFs locally in under 200ms, skipping expensive OCR services for the ~54% of PDFs that don't need them.

Features Copy Chapter

• Smart classification — TextBased / Scanned / ImageBased / Mixed in ~10–50ms, with a confidence score and per-page OCR routing.
• Markdown conversion — headings, lists, code blocks, bold/italic, URL linking, and dual-mode table detection (PDF drawing ops + text-alignment heuristics).
• Layout-aware extraction — multi-column reading order, position and font info per text item, RTL support.
• Robust text decoding — CID/Type0 fonts via ToUnicode CMaps, plus automatic flagging of broken encodings so callers can fall back to OCR.
• Lightweight — pure Rust, no ML models, no external services; single PDF dependency (lopdf).

Benchmark Copy Chapter

opendataloader-bench corpus (200 PDFs), local engines without model-based PDF parsing; OCR disabled. Scores 0–1, higher is better:

Refreshed July 31, 2026, on Apple M4 Pro; speed is the median of five complete corpus runs after an excluded warm-up. Full methodology and versions are in the repo README, with raw timings and artifacts in the results branch.

Install Copy Chapter

cargo add pdf-inspector

For the latest unreleased changes, use the git dependency instead:

[dependencies]
pdf-inspector = { git = "https://github.com/firecrawl/pdf-inspector" }

The crate also ships CLI binaries — pdf2md (PDF → Markdown, with --json, --pages, --select-pages, and the opt-in token-saving --compact profile) and detect-pdf (classification, with --analyze --json):

cargo install pdf-inspector

Usage Copy Chapter

Detect and extract in one call:

use pdf_inspector::process_pdf;

let result = process_pdf("document.pdf")?;

println!("Type: {:?}", result.pdf_type);       // TextBased, Scanned, ImageBased, Mixed
println!("Confidence: {:.0}%", result.confidence * 100.0);
println!("Pages: {}", result.page_count);

if let Some(markdown) = &result.markdown {
    println!("{}", markdown);
}

Fast metadata-only detection (no text extraction or markdown generation):

use pdf_inspector::detect_pdf;

let info = detect_pdf("document.pdf")?;

match info.pdf_type {
    pdf_inspector::PdfType::TextBased => {
        // Extract locally — fast and free
    }
    _ => {
        // Route to OCR service
        // info.pages_needing_ocr tells you exactly which pages
    }
}

Customize processing with PdfOptions:

use pdf_inspector::{process_pdf_with_options, PdfOptions, ProcessMode, DetectionConfig, ScanStrategy};

// Analyze layout without generating markdown
let result = process_pdf_with_options(
    "document.pdf",
    PdfOptions::new().mode(ProcessMode::Analyze),
)?;

// Full extraction with custom detection strategy
let result = process_pdf_with_options(
    "large.pdf",
    PdfOptions::new().detection(DetectionConfig {
        strategy: ScanStrategy::Sample(5),
        ..Default::default()
    }),
)?;

// Process only specific pages
let result = process_pdf_with_options(
    "document.pdf",
    PdfOptions::new().pages([1, 3, 5]),
)?;

Process from a byte buffer (no filesystem needed):

use pdf_inspector::process_pdf_mem;

let bytes = std::fs::read("document.pdf")?;
let result = process_pdf_mem(&bytes)?;

Vision extension contracts Copy Section

The native-only vision feature exposes the stable seam used by OCR
integrations without selecting or embedding an inference runtime. The
separate model-cache feature adds pinned artifact management:

• PageRenderer and OcrEngine traits;
• renderer-neutral owned page buffers and affine pixel↔PDF transforms;
• OcrOptions and opt-in Off/Auto/Force routing modes;
• positioned OCR results and per-page provenance types; and
• a versioned PP-OCRv6 Small manifest with checksum-verified, locked, atomic
  model-cache installation and explicit offline-directory overrides.

[dependencies]
pdf-inspector = { version = "1", features = ["vision", "model-cache"] }

The OCR contracts preserve existing behavior by default: OCR is Off and
model resolution is never reached. ModelStore itself does not access the
network. The optional model-download feature provides an
HTTPS downloader that streams pinned artifacts into the checksum-verified
cache only after routing has selected OCR work. Offline consumers set an
explicit model directory and ModelDownloadPolicy::Offline. Renderer-only
consumers do not enable model-cache or model-download and therefore do not
compile their filesystem, hashing, or HTTP dependencies.

use pdf_inspector::vision::{
    ModelDownloadPolicy, ModelStore, OcrMode, OcrOptions, PP_OCR_V6_SMALL,
};

let ocr = OcrOptions::new()
    .mode(OcrMode::Auto)
    .model_directory("/opt/firecrawl/models/pp-ocrv6-small")
    .model_downloads(ModelDownloadPolicy::Offline);
// Verifies exact sizes and SHA-256 digests before an engine opens the files.
let models = ModelStore::from_options(&ocr)?.resolve(&PP_OCR_V6_SMALL)?;
println!("using {} at {}", models.manifest_id(), models.revision());

Optional native page rendering Copy Section

The render-pdfium feature adds a native-only page renderer backed by
firecrawl-pdfium. It is the
rendering boundary for OCR pipelines; enabling it does not include an OCR
model or change the existing extraction functions. It implies vision,
and PdfiumRenderer implements the renderer-neutral PageRenderer trait.

[dependencies]
pdf-inspector = { version = "1", features = ["render-pdfium"] }

PDFium is loaded at runtime and is not bundled into the crate. Set
PDFIUM_LIB_PATH to the platform shared library, place that library next to
the executable, or use another discovery route supported by
firecrawl-pdfium. A load failure reports this prerequisite directly.

use pdf_inspector::vision::{PdfiumRenderer, RenderOptions};

let renderer = PdfiumRenderer::load()?;
let bytes = std::fs::read("document.pdf")?;
let pages = renderer.render_pages(
    &bytes,
    &[1, 3], // 1-indexed, matching pages_needing_ocr
    None,    // optional PDF password
    &RenderOptions::new().dpi(150.0),
)?;

for page in pages {
    // Owned RGB pixels can leave the PDFium critical section and be sent to
    // an OCR worker. OCR pixel boxes can be mapped back to PDF coordinates.
    let rect = page.pixel_rect_to_pdf_rect(20.0, 30.0, 100.0, 24.0);
    println!("page {}: {}x{}, rect={rect:?}", page.page(), page.width(), page.height());
}

Browser WASM remains on the default text-only path and does not expose native
PDFium rendering.

Optional OCR engine Copy Section

The native-only ocr-oar feature adds a CPU PP-OCRv6 Small implementation of
OcrEngine backed by OAR and ONNX Runtime. It implies model-cache, but does
not enable model auto-download, ONNX Runtime download, or PDF rendering. Model
files remain external, must match the pinned manifest, and are opened only
after ModelStore verifies their exact size and SHA-256 digest. Install an
ONNX Runtime shared library separately and set ORT_DYLIB_PATH to its full
path when it is not available through the platform library search path. The
runtime is resolved only when an OCR engine is first constructed; clean
Auto requests do not require it. The feature currently requires Rust 1.95
or newer, matching OAR 0.9.1's MSRV.

[dependencies]
pdf-inspector = { version = "1", features = ["ocr-oar", "render-pdfium"] }

Direct engine invocation is intentionally separate from extraction routing and
native/OCR fusion:

use pdf_inspector::vision::{
    ModelDownloadPolicy, ModelStore, OarOcrEngine, OcrEngine, OcrMode,
    OcrOptions, PdfiumRenderer, RenderOptions, PP_OCR_V6_SMALL,
};

let options = OcrOptions::new()
    .mode(OcrMode::Force)
    .minimum_confidence(0.45)
    .model_directory("/opt/firecrawl/models/pp-ocrv6-small")
    .model_downloads(ModelDownloadPolicy::Offline);
let models = ModelStore::from_options(&options)?.resolve(&PP_OCR_V6_SMALL)?;
let engine = OarOcrEngine::from_models(&models)?;

let renderer = PdfiumRenderer::load()?;
let bytes = std::fs::read("scan.pdf")?;
let pages = renderer.render_pages(&bytes, &[1], None, &RenderOptions::new())?;
let ocr_pages = engine.recognize(&pages, &options)?;

for span in &ocr_pages[0].spans {
    println!("{:.3}: {}", span.confidence, span.text);
}

The engine accepts renderer-neutral RGB, RGBA, and grayscale pages, preserves
OAR's positioned quadrilaterals in bitmap coordinates, filters spans using
minimum_confidence, and records the pinned model revision in every OcrPage.
OcrMode::Off is rejected at the engine boundary so default options cannot run
inference accidentally.

Selective routing and lazy model acquisition Copy Section

route_ocr_pages applies the existing detector/text-quality recommendations to
the configured mode. Auto processes only recommended pages, Force processes
all pages (or an explicit page selection), and Off always returns an empty
route. run_ocr_pages renders only that route, checks that both dependencies
preserve its order, and retains each bitmap's PDF transform for fusion.

[dependencies]
pdf-inspector = { version = "1", features = [
  "render-pdfium",
  "ocr-oar",
  "model-download",
] }

use pdf_inspector::vision::{
    route_ocr_pages, run_ocr_pages, HttpModelDownloader, ModelStore,
    OarOcrEngine, OcrMode, OcrOptions, PdfiumRenderer, RenderOptions,
    PP_OCR_V6_SMALL,
};

let bytes = std::fs::read("scan.pdf")?;
let extraction = pdf_inspector::extract_pages_markdown_mem(&bytes, None)?;
let options = OcrOptions::new().mode(OcrMode::Auto);
let routed = route_ocr_pages(
    options.mode,
    extraction.pages.len() as u32,
    &extraction.pages_needing_ocr,
    None,
)?;

if !routed.is_empty() {
    // No HTTP request or model initialization occurs before this point.
    let store = ModelStore::from_options(&options)?;
    let models = store.resolve_or_download(
        &PP_OCR_V6_SMALL,
        options.model_downloads,
        &HttpModelDownloader::default(),
    )?;
    let run = run_ocr_pages(
        &PdfiumRenderer::load()?,
        &OarOcrEngine::from_models(&models)?,
        &bytes,
        &routed,
        None,
        &RenderOptions::new(),
        &options,
    )?;
    println!("OCR processed {} pages", run.pages.len());
}

The downloader accepts HTTPS only, checks a declared content length, caps the
response stream to the pinned size plus one byte, and delegates final size and
SHA-256 verification to ModelStore. The store serializes installation across
processes and publishes completed artifacts atomically. Warm caches make no
network calls; offline mode and explicit model directories never download.

OCR Markdown assembly and native fusion Copy Section

fuse_ocr_pages maps OCR polygons back into PDF coordinates and sends the
result through pdf-inspector's existing deterministic reading-order, table,
and Markdown pipeline. Pages whose native extraction was rejected use OCR
output. When Force runs on a clean native page, normalized duplicate OCR
blocks are removed and only additional image-backed text is retained.

use pdf_inspector::vision::{fuse_ocr_pages, OcrFusionOptions};

let fused = fuse_ocr_pages(
    &extraction.pages,
    &run,
    extraction.pages.len() as u32,
    &OcrFusionOptions::new().render_dpi(150.0),
)?;

for page in &fused.pages {
    println!("{}", page.markdown);
    if page.provenance.hosted_recommended {
        eprintln!(
            "page {} needs the hosted document pipeline",
            page.page_number,
        );
    }
}

Each page carries Native, Ocr, or Fused provenance, the exact OCR model
revision, accepted-page confidence, local stage timings, and non-fatal
warnings. A page that required OCR recommends the hosted pipeline when local
OCR is missing, empty, or below the configurable page-confidence threshold.
This keeps the lightweight path explicit about cases it cannot finish well.

Complete OCR API Copy Section

The ocr convenience feature enables the renderer, OCR engine, verified
model acquisition, routing, and fusion layers together. It is the intended
downstream application integration boundary; lower-level features remain
available for consumers that bring their own renderer, model package manager,
or engine.

[dependencies]
pdf-inspector = { version = "1", features = ["ocr"] }

use pdf_inspector::vision::{process_pdf_with_ocr, OcrPdfOptions};

let result = process_pdf_with_ocr(
    "document.pdf",
    OcrPdfOptions::auto().page_numbers([1, 2, 3]),
)?;

println!("{}", result.markdown);
println!("OCR pages: {:?}", result.pages_routed_to_ocr);
println!(
    "Hosted fallback pages: {:?}",
    result.pages_recommending_hosted,
);

Native extraction always runs first. In Auto, a clean PDF returns before
PDFium loading, model-cache access, HTTP, or OAR initialization. Model files
remain external and the default crate feature set remains unchanged. Off
provides the same native-only behavior through the OCR result/provenance
shape; Force renders every selected page. OCR uses the existing deterministic
table, column, reading-order, and Markdown assembly path; no learned layout
model is included.

The OCR runtime setup guide
lists the pinned PDFium and ONNX Runtime builds, environment variables, model
cache behavior, and the error boundary downstream hosted fallbacks should use.

For ambiguous mixed pages, Auto privately retains clean native fragments
instead of discarding them when OCR is selected. After recognition it compares
script-agnostic text quality, OCR confidence, character overlap, and material
new coverage. Exact native text wins over a duplicate or weak OCR hypothesis;
complementary image-backed text is fused; and pages where both candidates are
weak recommend the hosted document pipeline. A page routed because native
coverage appeared incomplete also recommends hosted processing when confident
OCR only duplicates the retained fragment: the agreement preserves trustworthy
text, but neither hypothesis proves full-page coverage. Public native-only
extraction continues to suppress pages marked unreliable, and clean text
documents pay no renderer or model-initialization cost.

In Auto, pages routed only for suspicious font encoding or vectorized text
first get a bounded positioned-text probe through PDFium. A credible recovered
text layer with sufficient geometric page coverage skips rasterization and
model loading for that page; garbled, partial, or insubstantial recovery
continues through OCR. Recovered tables are reflected in the same document
metadata as tables found by the primary extractor.

The one-call API keeps the most recently used verified OCR engine in process.
Long-lived workers therefore verify the pinned artifacts and build the ONNX
sessions once, then reuse those loaded sessions across documents. The cache is
bounded to one model configuration and keyed by normalized model/runtime paths
plus the pinned manifest revision and artifact digests; switching the model
directory, runtime library, or compiled manifest replaces it. An active engine
owns the model data it already verified, so mutating artifacts in place does
not hot-reload a running process; restart the process when intentionally
replacing files at the same paths. CPU inference uses at most four intra-op
threads per ONNX session so a single small page does not oversubscribe larger
hosts, and recognizes variable-width line crops individually to avoid
padding-heavy CPU batches. The high-level pipeline renders and fuses at most
four routed pages at a time, bounding bitmap memory on long documents.

Build the CLI with the same opt-in feature:

cargo install pdf-inspector --features ocr --bin pdf2md
cargo build --release --features ocr --bin pdf2md
pdf2md document.pdf --ocr auto --raw
pdf2md document.pdf --ocr auto --json
pdf2md document.pdf --ocr auto --ocr-offline --ocr-model-dir /opt/models/pp-ocrv6-small

CLI controls include --ocr-dpi, --ocr-min-confidence,
--ocr-hosted-threshold, --select-pages, and the existing encrypted-PDF
--password option. JSON output has schema_version: 1 and includes per-page Markdown, source/model
provenance, confidence, timings, warnings, routed pages, and hosted-fallback
recommendations. Page numbers in OcrPdfResult and its per-page provenance
are 1-indexed, matching the PDF page numbers accepted by
OcrPdfOptions::page_numbers.

Extract per-page Markdown (one string per page, plus document-wide layout
metadata):

use pdf_inspector::extract_pages_markdown;

// Pass `None` for every page in document order, or a slice of 0-indexed
// pages to restrict the output (caller-supplied order is preserved).
let result = extract_pages_markdown("document.pdf", None)?;

for page in &result.pages {
    if page.needs_ocr {
        // Route this page to OCR
    } else {
        println!("Page {}: {}", page.page, page.markdown);
    }
}

println!("Complex layout? {}", result.is_complex);

Extract structure-tree elements from tagged PDFs, and join them against
extract_text_with_positions to attach semantic roles (heading levels,
paragraphs, table cells) to extracted text:

use pdf_inspector::{extract_structure_elements, extract_text_with_positions};
use std::collections::HashMap;

// One entry per marked-content reference, sorted by (page, mcid); empty for
// untagged PDFs. Pages are 1-indexed to match `TextItem::page`, so the
// (page, mcid) pair is a direct join key.
let elements = extract_structure_elements("tagged.pdf", None)?;
let roles: HashMap<(u32, i64), &str> = elements
    .iter()
    .map(|e| ((e.page, e.mcid), e.role.as_str()))
    .collect();

for item in extract_text_with_positions("tagged.pdf")? {
    if let Some(mcid) = item.mcid {
        if let Some(role) = roles.get(&(item.page, mcid)) {
            if role.starts_with('H') {
                println!("{}: {}", role, item.text);
            }
        }
    }
}

Processing modes Copy Chapter

Functions Copy Chapter

Low-level detection functions are also available via the detector module (detect_pdf_type, detect_pdf_type_with_config, etc.) for callers who need PdfTypeResult instead of PdfProcessResult.

Types Copy Chapter

File: napi/README.md Copy Chapter

PDF Inspector

Fast PDF classification and region-based text extraction for Node.js/Bun. Native Rust performance via napi-rs.

Built by Firecrawl for hybrid OCR pipelines — extract text from PDF structure where possible, fall back to OCR only when needed.

Features Copy Chapter

• Smart classification — text-based / scanned / image-based / mixed in ~10–50ms, with a confidence score and per-page OCR routing.
• Region-based extraction — pull text from bounding boxes with per-region quality checks (needsOcr).
• Layout-aware — multi-column reading order, position and font info per text item, RTL support.
• Robust text decoding — CID/Type0 fonts via ToUnicode CMaps, plus automatic flagging of broken encodings so callers can fall back to OCR.
• Selective OCR — Auto routes only pages rejected by native extraction and returns source/model provenance plus hosted-fallback recommendations.
• External artifacts — the native package embeds no OCR models, PDFium, or ONNX Runtime; clean Auto requests never load or download them.

Benchmark Copy Chapter

opendataloader-bench corpus (200 PDFs), local engines without model-based PDF parsing; OCR disabled. Scores 0–1, higher is better:

Refreshed July 31, 2026, on Apple M4 Pro; speed is the median of five complete corpus runs after an excluded warm-up. Full methodology and versions are in the repo README, with raw timings and artifacts in the results branch.

Install Copy Chapter

npm install @firecrawl/pdf-inspector
# or
bun add @firecrawl/pdf-inspector

Prebuilt binaries for Linux x64/ARM64 (glibc and musl/Alpine), macOS ARM64, and Windows x64 — npm installs only the one matching your platform. No Rust toolchain needed.

OCR calls that route work require compatible PDFium and ONNX Runtime shared
libraries. Set PDFIUM_LIB_PATH and ORT_DYLIB_PATH when they are not on the
platform library search path. The pinned OCR model set is downloaded and
checksum-verified on the first routed page; use offline: true with a warm
cache or modelDirectory to prohibit network access. See the
OCR runtime setup guide
for pinned downloads, supported platforms, and hosted-fallback behavior.

API Copy Chapter

`processPdfWithOcr(buffer: Buffer, options?: OcrOptions): Promise` Copy Section

Run native extraction first and OCR only the pages selected by its quality
signals. The default mode is Auto; Off returns the same detailed result
shape without external runtime work, and Force OCRs every selected page.
The work runs on the libuv thread pool and never blocks Node's event loop.

import { OcrMode, processPdfWithOcr } from '@firecrawl/pdf-inspector'

const result = await processPdfWithOcr(pdf, {
  mode: OcrMode.Auto,
  pageNumbers: [1, 3], // 1-indexed
})

for (const page of result.pages) {
  console.log(page.pageNumber, page.provenance.source)
}
console.log(result.pagesRoutedToOcr)
console.log(result.pagesRecommendingHosted)

For offline deployments, pass modelDirectory and offline: true. Other
controls include dpi, minimumConfidence,
hostedRecommendationConfidence, and password.

`classifyPdf(buffer: Buffer): PdfClassification` Copy Section

Classify a PDF as TextBased, Scanned, Mixed, or ImageBased (~10-50ms). Returns which pages need OCR.

import { classifyPdf } from '@firecrawl/pdf-inspector'
import { readFileSync } from 'fs'

const pdf = readFileSync('document.pdf')
const result = classifyPdf(pdf)

console.log(result.pdfType)        // "TextBased" | "Scanned" | "Mixed" | "ImageBased"
console.log(result.pageCount)      // 42
console.log(result.pagesNeedingOcr) // [5, 12, 15] (0-indexed)
console.log(result.confidence)     // 0.875

`extractTextInRegions(buffer: Buffer, pageRegions: PageRegions[]): PageRegionTexts[]` Copy Section

Extract text within bounding-box regions from a PDF. Designed for hybrid OCR pipelines where a layout model detects regions in rendered page images, and this function extracts text from the PDF structure for text-based pages — skipping GPU OCR.

Each region result includes a needsOcr flag that signals unreliable extraction (empty text, GID-encoded fonts, garbage text, encoding issues). When the cause is a suspected garbled text layer, ocrReason is set to "suspected_garbled_text".

import { extractTextInRegions } from '@firecrawl/pdf-inspector'

const result = extractTextInRegions(pdf, [
  {
    page: 0, // 0-indexed
    regions: [
      [0, 0, 300, 400],    // [x1, y1, x2, y2] in PDF points, top-left origin
      [300, 0, 612, 400],
    ]
  }
])

for (const region of result[0].regions) {
  if (region.needsOcr) {
    // Unreliable text — send this region to OCR instead
  } else {
    console.log(region.text) // Extracted text in reading order
  }
}

Async variants Copy Section

processPdf, classifyPdf, and extractPagesMarkdown are synchronous and parse on the calling thread — in Node, that's the event loop. For a one-off call in a script that's fine, but in a server a large document can hold the loop for tens to hundreds of milliseconds.

processPdfAsync, classifyPdfAsync, and extractPagesMarkdownAsync take the same arguments and produce the same results, but run the parse on the libuv thread pool and return a promise, keeping the event loop free. The input buffer is copied before the call returns, so it's safe to reuse or mutate immediately:

import { classifyPdfAsync, extractPagesMarkdownAsync } from '@firecrawl/pdf-inspector'

const classification = await classifyPdfAsync(pdf)
if (classification.pdfType === 'TextBased') {
  const { pages } = await extractPagesMarkdownAsync(pdf)
  // ...
}

Types Copy Chapter

interface PdfClassification {
  pdfType: string          // "TextBased" | "Scanned" | "Mixed" | "ImageBased"
  pageCount: number
  pagesNeedingOcr: number[] // 0-indexed page numbers
  confidence: number        // 0.0 - 1.0
}

interface PageRegions {
  page: number              // 0-indexed
  regions: number[][]       // [[x1, y1, x2, y2], ...] in PDF points, top-left origin
}

interface PageRegionTexts {
  page: number
  regions: RegionText[]
}

interface RegionText {
  text: string
  needsOcr: boolean         // true when text is unreliable
  ocrReason?: string        // "suspected_garbled_text" when known
}

interface OcrPdfResult {
  markdown: string
  pages: OcrPageResult[]              // 1-indexed pages + provenance
  pageCount: number
  pagesRecommendedForOcr: number[]
  pagesRoutedToOcr: number[]
  pagesRecommendingHosted: number[]
  ocrReasonsByPage: PageOcrReasons[]
  pagesWithTables: number[]
  pagesWithColumns: number[]
  isComplex: boolean
  processingTimeMs: number
  renderTimeMs: number
  ocrTimeMs: number
}

Platforms Copy Chapter

Prebuilt binaries ship as platform-specific packages installed automatically via optionalDependencies:

License Copy Chapter

MIT

File: wasm/README.md Copy Chapter

@firecrawl/pdf-inspector-wasm

Browser WebAssembly bindings for pdf-inspector. Classify PDFs and extract structured Markdown locally from a Uint8Array, using the same Rust core as the native Node.js, Python, and Rust packages.

Install Copy Chapter

npm install @firecrawl/pdf-inspector-wasm

Usage Copy Chapter

import init, { processPdf } from "@firecrawl/pdf-inspector-wasm";

await init();

const response = await fetch("/annual-report.pdf");
const pdf = new Uint8Array(await response.arrayBuffer());
const result = processPdf(pdf);

console.log(result.pdfType);
console.log(result.markdown);

Pass options when you need selected pages or compact Markdown:

const result = processPdf(pdf, {
  pages: [1, 3, 5],
  profile: "compact",
  includePageMarkers: true,
});

The package also exports:

• detectPdf(pdf, options?) for detection without extraction.
• classifyPdf(pdf) for the lightweight result shape shared with the native Node.js API.
• extractText(pdf) for plain text.
• version() for the WASM package version.

Browser behavior Copy Chapter

• Parsing runs locally. PDF bytes are not uploaded anywhere.
• The build is single-threaded and does not require cross-origin isolation.
• CMaps are embedded so CJK font decoding does not depend on a filesystem.
• Extraction is synchronous after init(). For large documents, call it from a Web Worker to keep the UI responsive.
• Image-only documents still require a separate OCR step.

Build from source Copy Chapter

cargo install wasm-pack --version 0.15.0 --locked
wasm-pack build wasm --target web --scope firecrawl --release

License Copy Chapter

MIT