opendataloader-pdf Documentation

File: README.md Copy Chapter

OpenDataLoader PDF

PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.

🔍 PDF parser for AI data extraction — Extract Markdown, JSON (with bounding boxes), and HTML from any PDF. #1 in benchmarks (0.907 overall). Deterministic local mode + AI hybrid mode for complex pages.

• How accurate is it? — #1 in benchmarks: 0.907 overall, 0.928 table accuracy across 200 real-world PDFs including multi-column and scientific papers. Deterministic local mode + AI hybrid mode for complex pages (benchmarks)
• Scanned PDFs and OCR? — Yes. Built-in OCR (80+ languages) in hybrid mode. Works with poor-quality scans at 300 DPI+ (hybrid mode)
• Tables, formulas, images, charts? — Yes. Complex/borderless tables, LaTeX formulas, and AI-generated picture/chart descriptions all via hybrid mode (hybrid mode)
• How do I use this for RAG? — pip install opendataloader-pdf, convert in 3 lines. Outputs structured Markdown for chunking, JSON with bounding boxes for source citations, and HTML. LangChain integration available. Python, Node.js, Java SDKs (quick start | LangChain)

♿ PDF accessibility automation — Auto-tag untagged PDFs into screen-reader-ready Tagged PDFs at scale. First open-source tool to generate Tagged PDFs end-to-end.

• What's the problem? — Accessibility regulations are now enforced worldwide. Manual PDF remediation costs $50–200 per document and doesn't scale (regulations)
• What's free? — Layout analysis + auto-tagging (Apache 2.0). Untagged PDF in → Tagged PDF out. No proprietary SDK dependency (auto-tagging)
• What about PDF/UA compliance? — Converting Tagged PDF to PDF/UA-1 or PDF/UA-2 is an enterprise add-on. Auto-tagging generates the Tagged PDF; PDF/UA export is the final step (pipeline)
• Why trust this? — Built in collaboration with Dual Lab (veraPDF developers) based on PDF Association specifications, best practice guides and expertise of the PDF Community. Auto-tagging follows the Well-Tagged PDF specification, validated with veraPDF (collaboration)

Get Started in 30 Seconds Copy Chapter

Requires: Java 11+ and Python 3.10+ (Node.js | Java also available)

Before you start: run java -version. If not found, install JDK 11+ from Adoptium.

pip install -U opendataloader-pdf

import opendataloader_pdf

# Batch all files in one call — each convert() spawns a JVM process, so repeated calls are slow
opendataloader_pdf.convert(
    input_path=["file1.pdf", "file2.pdf", "folder/"],
    output_dir="output/",
    format="markdown,json"
)

Annotated PDF output — each element (heading, paragraph, table, image) detected with bounding boxes and semantic type.

What Problems Does This Solve? Copy Chapter

Capability Matrix Copy Chapter

Extraction Benchmarks Copy Chapter

opendataloader-pdf [hybrid] ranks #1 overall (0.907) across reading order, table, and heading extraction accuracy.

Scores normalized to [0, 1]. Higher is better for accuracy; lower is better for speed. Bold = best. Full benchmark details

Which Mode Should I Use? Copy Chapter

Quick Start Copy Chapter

Python Copy Section

pip install -U opendataloader-pdf

import opendataloader_pdf

# Batch all files in one call — each convert() spawns a JVM process, so repeated calls are slow
opendataloader_pdf.convert(
    input_path=["file1.pdf", "file2.pdf", "folder/"],
    output_dir="output/",
    format="markdown,json"
)

Node.js Copy Section

npm install @opendataloader/pdf

import { convert } from '@opendataloader/pdf';

await convert(['file1.pdf', 'file2.pdf', 'folder/'], {
  outputDir: 'output/',
  format: 'markdown,json'
});

Java Copy Section

<dependency>
  <groupId>org.opendataloader</groupId>
  <artifactId>opendataloader-pdf-core</artifactId>
</dependency>

Python Quick Start | Node.js Quick Start | Java Quick Start

Hybrid Mode: #1 Accuracy for Complex PDFs Copy Chapter

Hybrid mode combines fast local Java processing with AI backends. Simple pages stay local (0.02s); complex pages route to AI for +90% table accuracy.

Don't combine with --use-struct-tree on tagged PDFs. --use-struct-tree takes precedence, so the hybrid backend is not called (a warning is logged). If you want the hybrid backend, drop --use-struct-tree.

pip install -U "opendataloader-pdf[hybrid]"

Terminal 1 — Start the backend server:

opendataloader-pdf-hybrid --port 5002

Terminal 2 — Process PDFs:

# Batch all files in one call — each invocation spawns a JVM process, so repeated calls are slow
opendataloader-pdf --hybrid docling-fast file1.pdf file2.pdf folder/

Python:

# Batch all files in one call — each convert() spawns a JVM process, so repeated calls are slow
opendataloader_pdf.convert(
    input_path=["file1.pdf", "file2.pdf", "folder/"],
    output_dir="output/",
    hybrid="docling-fast"
)

OCR for Scanned PDFs Copy Section

Start the backend with --force-ocr for image-based PDFs with no selectable text:

opendataloader-pdf-hybrid --port 5002 --force-ocr

For non-English documents, specify the language:

opendataloader-pdf-hybrid --port 5002 --force-ocr --ocr-lang "ko,en"

Supported languages: en, ko, ja, ch_sim, ch_tra, de, fr, ar, and more.

Formula Extraction (LaTeX) Copy Section

Extract mathematical formulas as LaTeX from scientific PDFs:

# Server: enable formula enrichment
opendataloader-pdf-hybrid --enrich-formula

# Batch all files in one call — each invocation spawns a JVM process, so repeated calls are slow
opendataloader-pdf --hybrid docling-fast --hybrid-mode full file1.pdf file2.pdf folder/

Output in JSON:

{
  "type": "formula",
  "page number": 1,
  "bounding box": [226.2, 144.7, 377.1, 168.7],
  "content": "\\frac{f(x+h) - f(x)}{h}"
}

Note: Formula and picture description enrichments require --hybrid-mode full on the client side.

Chart & Image Description Copy Section

Generate AI descriptions for charts and images — useful for RAG search and accessibility alt text:

# Server
opendataloader-pdf-hybrid --enrich-picture-description

# Batch all files in one call — each invocation spawns a JVM process, so repeated calls are slow
opendataloader-pdf --hybrid docling-fast --hybrid-mode full file1.pdf file2.pdf folder/

Output in JSON:

{
  "type": "picture",
  "page number": 1,
  "bounding box": [72.0, 400.0, 540.0, 650.0],
  "description": "A bar chart showing waste generation by region from 2016 to 2030..."
}

Uses SmolVLM (256M), a lightweight vision model. Custom prompts supported via --picture-description-prompt.

Hancom Data Loader Integration — Coming Soon Copy Section

Enterprise-grade AI document analysis via Hancom Data Loader — customer-customized models trained on your domain-specific documents. 30+ element types (tables, charts, formulas, captions, footnotes, etc.), VLM-based image/chart understanding, complex table extraction (merged cells, nested tables), SLA-backed OCR for scanned documents, and native HWP/HWPX support. Supports PDF, DOCX, XLSX, PPTX, HWP, PNG, JPG. Live demo

Hybrid Mode Guide

Output Formats Copy Chapter

Combine formats: format="json,markdown"

JSON Output Example Copy Section

{
  "type": "heading",
  "id": 42,
  "level": "Title",
  "page number": 1,
  "bounding box": [72.0, 700.0, 540.0, 730.0],
  "heading level": 1,
  "font": "Helvetica-Bold",
  "font size": 24.0,
  "text color": "[0.0]",
  "content": "Introduction"
}

Full JSON Schema

Advanced Features Copy Chapter

Tagged PDF Support Copy Section

When a PDF has structure tags, OpenDataLoader extracts the exact layout the author intended — no guessing, no heuristics. Headings, lists, tables, and reading order are preserved from the source.

Output quality depends on tag quality. Not all tagged PDFs are well-tagged. For PDFs with sparse or incorrect tags, the default heuristic mode or --hybrid docling-fast often produces better results.

--use-struct-tree takes precedence over --hybrid. If both are set on a tagged PDF, the structure tree is used and the hybrid backend is not called (a well-tagged PDF already carries reading order and structure). Drop --use-struct-tree if you want the hybrid backend instead.

# Batch all files in one call — each convert() spawns a JVM process, so repeated calls are slow
opendataloader_pdf.convert(
    input_path=["file1.pdf", "file2.pdf", "folder/"],
    output_dir="output/",
    use_struct_tree=True           # Use native PDF structure tags
)

Most PDF parsers ignore structure tags entirely. Learn more

AI Safety: Prompt Injection Protection Copy Section

PDFs can contain hidden prompt injection attacks. OpenDataLoader automatically filters:

• Hidden text (transparent, zero-size fonts)
• Off-page content
• Suspicious invisible layers

To sanitize sensitive data (emails, URLs, phone numbers → placeholders), enable it explicitly:

# Batch all files in one call — each invocation spawns a JVM process, so repeated calls are slow
opendataloader-pdf file1.pdf file2.pdf folder/ --sanitize

AI Safety Guide

LangChain Integration Copy Section

pip install -U langchain-opendataloader-pdf

from langchain_opendataloader_pdf import OpenDataLoaderPDFLoader

loader = OpenDataLoaderPDFLoader(
    file_path=["file1.pdf", "file2.pdf", "folder/"],
    format="text"
)
documents = loader.load()

LangChain Docs | GitHub | PyPI

Advanced Options Copy Section

# Batch all files in one call — each convert() spawns a JVM process, so repeated calls are slow
opendataloader_pdf.convert(
    input_path=["file1.pdf", "file2.pdf", "folder/"],
    output_dir="output/",
    format="json,markdown,pdf",
    image_output="embedded",        # "off", "embedded" (Base64), or "external" (default)
    image_format="jpeg",            # "png" or "jpeg"
    use_struct_tree=True,           # Use native PDF structure
)

Full CLI Options Reference

PDF Accessibility & PDF/UA Conversion Copy Chapter

Problem: Millions of existing PDFs lack structure tags, failing accessibility regulations (EAA, ADA/Section 508, Korea Digital Inclusion Act). Manual remediation costs $50–200 per document and doesn't scale.

OpenDataLoader's approach: Built in collaboration with PDF Association and Dual Lab (developers of veraPDF, the industry-reference open-source PDF/A and PDF/UA validator). Auto-tagging follows the Well-Tagged PDF specification and is validated programmatically using veraPDF — automated conformance checks against PDF accessibility standards, not manual review. No existing open-source tool generates Tagged PDFs end-to-end — most rely on proprietary SDKs for the tag-writing step. OpenDataLoader does it all under Apache 2.0. (collaboration details)

Standards & Validation Copy Section

Accessibility Pipeline Copy Section

💼 Enterprise features are available on request. Contact us to get started.

Auto-Tagging Copy Section

Generate Tagged PDFs from untagged PDFs — output is a screen-reader-ready PDF with structure tags (headings, paragraphs, lists, tables, reading order).

import opendataloader_pdf

# Untagged PDF in → Tagged PDF out
opendataloader_pdf.convert(
    input_path=["file1.pdf", "file2.pdf", "folder/"],
    output_dir="output/",
    format="tagged-pdf"
)

# CLI
opendataloader-pdf --format tagged-pdf file1.pdf file2.pdf folder/

Combine with other formats: format="json,tagged-pdf".

End-to-End Compliance Workflow Copy Section

Existing PDFs (untagged)
    │
    ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐    ┌──────────────────┐
│  1. Audit       │───>│  2. Auto-Tag     │───>│  3. Export      │───>│  4. Studio       │
│  (check tags)   │    │  (→ Tagged PDF)  │    │  (PDF/UA)       │    │  (visual editor) │
└─────────────────┘    └──────────────────┘    └─────────────────┘    └──────────────────┘
        │                       │                       │                      │
        ▼                       ▼                       ▼                      ▼
  use_struct_tree      format="tagged-pdf"        PDF/UA export       Accessibility Studio
  (Available now)      (Available, Apache 2.0)    (Enterprise)        (Enterprise)

PDF Accessibility Guide

Roadmap Copy Chapter

Full Roadmap

Frequently Asked Questions Copy Chapter

What is the best PDF parser for RAG? Copy Section

For RAG pipelines, you need a parser that preserves document structure, maintains correct reading order, and provides element coordinates for citations. OpenDataLoader is designed specifically for this — it outputs structured JSON with bounding boxes, handles multi-column layouts with XY-Cut++, and runs locally without GPU. In hybrid mode, it ranks #1 overall (0.907) in benchmarks.

What is the best open-source PDF parser? Copy Section

OpenDataLoader PDF is the only open-source parser that combines: rule-based deterministic extraction (no GPU), bounding boxes for every element, XY-Cut++ reading order, built-in AI safety filters, native Tagged PDF support, and hybrid AI mode for complex documents. It ranks #1 in overall accuracy (0.907) while running locally on CPU.

How do I extract tables from PDF for LLM? Copy Section

OpenDataLoader detects tables using border analysis and text clustering, preserving row/column structure. For complex tables, enable hybrid mode for +90% accuracy improvement (0.489 to 0.928 TEDS score):

# Batch all files in one call — each convert() spawns a JVM process, so repeated calls are slow
opendataloader_pdf.convert(
    input_path=["file1.pdf", "file2.pdf", "folder/"],
    output_dir="output/",
    format="json",
    hybrid="docling-fast"           # For complex tables
)

How does it compare to docling, marker, or pymupdf4llm? Copy Section

OpenDataLoader [hybrid] ranks #1 overall (0.907) across reading order, table, and heading accuracy. Key differences: docling (0.882) is strong but lacks bounding boxes and AI safety filters. marker (0.861) requires GPU and is 1000x slower (53.932s/page). pymupdf4llm (0.732) is fast but has poor table (0.401) and heading (0.412) accuracy. OpenDataLoader is the only parser that combines deterministic local extraction, bounding boxes for every element, and built-in prompt injection protection. See full benchmark.

Can I use this without sending data to the cloud? Copy Section

Yes. OpenDataLoader runs 100% locally. No API calls, no data transmission — your documents never leave your environment. The hybrid mode backend also runs locally on your machine. Ideal for legal, healthcare, and financial documents.

Does it support OCR for scanned PDFs? Copy Section

Yes, via hybrid mode. Install with pip install "opendataloader-pdf[hybrid]", start the backend with --force-ocr, then process as usual. Supports multiple languages including Korean, Japanese, Chinese, Arabic, and more via --ocr-lang.

Does it work with Korean, Japanese, or Chinese documents? Copy Section

Yes. For digital PDFs, text extraction works out of the box. For scanned PDFs, use hybrid mode with --force-ocr --ocr-lang "ko,en" (or ja, ch_sim, ch_tra). Coming soon: Hancom Data Loader integration — enterprise-grade AI document analysis with built-in production-grade OCR and customer-customized models optimized for your specific document types and workflows.

How fast is it? Copy Section

Local mode processes 60+ pages per second on CPU (0.02s/page). Hybrid mode processes 2+ pages per second (0.46s/page) with significantly higher accuracy for complex documents. No GPU required. Benchmarked on Apple M4. Full benchmark details. With multi-process batch processing, throughput exceeds 100 pages per second on 8+ core machines.

Does it handle multi-column layouts? Copy Section

Yes. OpenDataLoader uses XY-Cut++ reading order analysis to correctly sequence text across multi-column pages, sidebars, and mixed layouts. This works in both local and hybrid modes without any configuration.

What is hybrid mode? Copy Section

Hybrid mode combines fast local Java processing with an AI backend. Simple pages are processed locally (0.02s/page); complex pages (tables, scanned content, formulas, charts) are automatically routed to the AI backend for higher accuracy. The backend runs locally on your machine — no cloud required. See Which Mode Should I Use? and Hybrid Mode Guide.

Does it work with LangChain? Copy Section

Yes. Install langchain-opendataloader-pdf for an official LangChain document loader integration. See LangChain docs.

How do I chunk PDFs for RAG? Copy Section

OpenDataLoader outputs structured Markdown with headings, tables, and lists preserved — ideal input for semantic chunking. Each element in JSON output includes type, heading level, and page number, so you can split by section or page boundary. For most RAG pipelines: parse with format="markdown" for text chunks, or format="json" when you need element-level control. Pair with LangChain's RecursiveCharacterTextSplitter or your own heading-based splitter for best results.

How do I cite PDF sources in RAG answers? Copy Section

Every element in JSON output includes a bounding box ([left, bottom, right, top] in PDF points) and page number. When your RAG pipeline returns an answer, map the source chunk back to its bounding box to highlight the exact location in the original PDF. This enables "click to source" UX — users see which paragraph, table, or figure the answer came from. No other open-source parser provides bounding boxes for every element by default.

How do I convert PDF to Markdown for LLM? Copy Section

import opendataloader_pdf

# Batch all files in one call — each convert() spawns a JVM process, so repeated calls are slow
opendataloader_pdf.convert(
    input_path=["file1.pdf", "file2.pdf", "folder/"],
    output_dir="output/",
    format="markdown"
)

OpenDataLoader preserves heading hierarchy, table structure, and reading order in the Markdown output. For complex documents with borderless tables or scanned pages, use hybrid mode (hybrid="docling-fast") for higher accuracy. The output is clean enough to feed directly into LLM context windows or RAG chunking pipelines.

Is there an automated PDF accessibility remediation tool? Copy Section

Yes. OpenDataLoader is the first open-source tool that automates PDF accessibility end-to-end. Built in collaboration with PDF Association and Dual Lab (veraPDF developers), auto-tagging follows the Well-Tagged PDF specification and is validated programmatically using veraPDF. The layout analysis engine detects document structure (headings, tables, lists, reading order) and generates accessibility tags automatically. Auto-tagging converts untagged PDFs into Tagged PDFs under Apache 2.0 — no proprietary SDK dependency. Use format="tagged-pdf" (Python/Node.js) or --format tagged-pdf (CLI). For organizations needing full PDF/UA compliance, enterprise add-ons provide PDF/UA export and a visual tag editor. This replaces manual remediation workflows that typically cost $50–200+ per document.

Is this really the first open-source PDF auto-tagging tool? Copy Section

Yes. Existing tools either depend on proprietary SDKs for writing structure tags, only output non-PDF formats (e.g., Docling outputs Markdown/JSON but cannot produce Tagged PDFs), or require manual intervention. OpenDataLoader is the first to do layout analysis → tag generation → Tagged PDF output entirely under an open-source license (Apache 2.0), with no proprietary dependency. Auto-tagging follows the PDF Association's Well-Tagged PDF specification and is validated using veraPDF, the industry-reference open-source PDF/A and PDF/UA validator.

How do I convert existing PDFs to PDF/UA? Copy Section

OpenDataLoader provides an end-to-end pipeline: audit existing PDFs for tags (use_struct_tree=True), auto-tag untagged PDFs into Tagged PDFs (format="tagged-pdf", free under Apache 2.0), and export as PDF/UA-1 or PDF/UA-2 (enterprise add-on). Auto-tagging follows the PDF Association's Well-Tagged PDF specification and is validated using veraPDF. Auto-tagging generates the Tagged PDF; PDF/UA export is the final step. Contact us for enterprise integration.

How do I make my PDFs accessible for EAA compliance? Copy Section

The European Accessibility Act requires accessible digital products by June 28, 2025. OpenDataLoader supports the full remediation workflow: audit → auto-tag → Tagged PDF → PDF/UA export. Auto-tagging follows the PDF Association's Well-Tagged PDF specification and is validated using veraPDF, ensuring standards-compliant output. Auto-tagging to Tagged PDF is open-source under Apache 2.0. PDF/UA export and accessibility studio are enterprise add-ons. See our Accessibility Guide.

Is OpenDataLoader PDF free? Copy Section

The core library is open-source under Apache 2.0 — free for commercial use. This includes all extraction features (text, tables, images, OCR, formulas, charts via hybrid mode), AI safety filters, Tagged PDF support, and auto-tagging to Tagged PDF. We are committed to keeping the core accessibility pipeline (layout analysis → auto-tagging → Tagged PDF) free and open-source. Enterprise add-ons (PDF/UA export, accessibility studio) are available for organizations needing end-to-end regulatory compliance.

Why did the license change from MPL 2.0 to Apache 2.0? Copy Section

MPL 2.0 requires file-level copyleft, which often triggers legal review before enterprise adoption. Apache 2.0 is fully permissive — no copyleft obligations, easier to integrate into commercial projects. If you are using a pre-2.0 version, it remains under MPL 2.0 and you can continue using it. Upgrading to 2.0+ means your project follows Apache 2.0 terms, which are strictly more permissive — no additional obligations, no action needed on your side.

Documentation Copy Chapter

• Quick Start (Python)
• Quick Start (Node.js)
• Quick Start (Java)
• JSON Schema Reference
• CLI Options
• Hybrid Mode Guide
• Tagged PDF Support
• AI Safety Features
• PDF Accessibility

Contributing Copy Chapter

We welcome contributions! See CONTRIBUTING.md for guidelines.

License Copy Chapter

Apache License 2.0

Note: Versions prior to 2.0 are licensed under the Mozilla Public License 2.0.

Found this useful? Give us a star to help others discover OpenDataLoader.

File: docs/superpowers/specs/2026-03-16-cid-font-detection-design.md Copy Chapter

CID Font Extraction Failure Detection

Issue: #286

Problem Copy Chapter

PDFs with CID-keyed fonts that lack ToUnicode mappings produce no usable text from veraPDF extraction. veraPDF replaces unmappable characters with U+FFFD (replacement character), which TextProcessor.replaceUndefinedCharacters() then converts to spaces. The result is empty or whitespace-only output with no indication to the user of what went wrong.

Users currently resort to external tools (e.g., pdfplumber) to pre-screen PDFs for CID issues before passing them to opendataloader-pdf.

Solution Copy Chapter

Detect pages with high replacement character ratios and:

1. Always: emit a WARNING log explaining the problem and suggesting --hybrid-mode
2. When hybrid mode is on: automatically route affected pages to OCR backend via TriageProcessor

No new CLI options. Hybrid mode setting is respected as-is.

Design Copy Chapter

Detection: Replacement Character Ratio Copy Section

TextProcessor.measureReplacementCharRatio(List) counts \uFFFD characters across all TextChunks on a page and returns the ratio (0.0–1.0).

Threshold: 30%. CID-affected pages typically show 90%+ replacement characters. 30% catches real problems while avoiding false positives from PDFs with occasional unmappable glyphs.

Measurement point: Inside ContentFilterProcessor.getFilteredContents(), immediately before replaceUndefinedCharacters() is called (line 74). At this point veraPDF has already inserted \uFFFD but the characters haven't been replaced with spaces yet, so measurement is accurate.

Safety of measurement point: The prior processing steps (mergeCloseTextChunks, trimTextChunksWhiteSpaces, filterConsecutiveSpaces, splitTextChunksByWhiteSpaces) do not affect U+FFFD characters. U+FFFD is not whitespace, so it is not trimmed, compressed, or used as a split boundary. The count is accurate at this position.

Zero-text pages: When a page has no TextChunk objects (e.g., image-only pages), the method returns 0.0 to avoid division by zero. This correctly avoids triggering the CID warning on non-text pages.

The method uses ChunkParser.REPLACEMENT_CHARACTER_STRING constant (not a hardcoded "\uFFFD" literal) to stay consistent with replaceUndefinedCharacters().

Data Flow Copy Section

The measured ratio is stored in StaticLayoutContainers per page:

ContentFilterProcessor.getFilteredContents()
  │
  ├─ TextProcessor.measureReplacementCharRatio() → ratio
  ├─ StaticLayoutContainers.setReplacementCharRatio(pageNumber, ratio)
  ├─ if ratio >= 0.3: LOGGER.warning(...)
  └─ TextProcessor.replaceUndefinedCharacters()  // existing call

Note: StaticLayoutContainers currently stores global ThreadLocal scalars and lists, not per-page maps. Per-page data (e.g., bounding boxes) lives in DocumentProcessor. This change introduces a new per-page Map pattern to StaticLayoutContainers. We place it here rather than DocumentProcessor because it is layout-metadata consumed by TriageProcessor, keeping the triage data path self-contained. The existing clearContainers() method must be updated to clear this map to prevent cross-document data leakage in multi-document processing.

Warning Log Copy Section

Emitted from ContentFilterProcessor when ratio >= 0.3:

WARNING: Page 3: 94% of characters are replacement characters (U+FFFD).
This PDF likely contains CID-keyed fonts without ToUnicode mappings.
Text extraction may be incomplete. Consider using --hybrid-mode for OCR fallback.

This fires regardless of hybrid mode setting.

Triage Routing Copy Section

In TriageProcessor.classifyPage(), a new Signal 0 is inserted before all existing signals (before TableBorder check). This signal only fires when hybrid mode is active, since classifyPage() is only called from HybridDocumentProcessor. In non-hybrid mode, only the warning log (from ContentFilterProcessor) is emitted:

double replacementRatio = StaticLayoutContainers.getReplacementCharRatio(pageNumber);
if (replacementRatio >= 0.3) {
    return TriageResult.backend(pageNumber, 1.0, signals);
}

Priority is highest (confidence 1.0) because a page with mostly broken text extraction gains nothing from Java-path processing.

Behavior Matrix Copy Section

Changes Copy Chapter

Modified Files Copy Section

New Files Copy Section

Modified Test Files Copy Section

Test Plan Copy Chapter

Unit Tests (TextProcessorTest) Copy Section

• testMeasureReplacementCharRatio_allReplacement — all U+FFFD → 1.0
• testMeasureReplacementCharRatio_noReplacement — normal text → 0.0
• testMeasureReplacementCharRatio_mixed — 30% U+FFFD → 0.3
• testMeasureReplacementCharRatio_emptyContents — empty list → 0.0
• testMeasureReplacementCharRatio_nonTextChunksIgnored — non-text objects skipped

Unit Tests (TriageProcessorTest) Copy Section

• testClassifyPage_highReplacementRatio_routesToBackend — ratio 0.5 → BACKEND
• testClassifyPage_lowReplacementRatio_noEffect — ratio 0.1 → JAVA (default)
• testClassifyPage_exactThreshold_routesToBackend — ratio 0.3 → BACKEND

Boundary Tests Copy Section

• testWarningNotEmitted_belowThreshold — ratio 0.29 → no warning log emitted
• testWarningEmitted_atThreshold — ratio 0.30 → warning log emitted

e2e Test (CidFontDetectionTest) Copy Section

• Load pre-generated cid-font-no-tounicode.pdf
• Run through ContentFilterProcessor.getFilteredContents()
• Assert: StaticLayoutContainers.getReplacementCharRatio(0) >= 0.3
• Assert: warning log contains "replacement characters"

Benchmark Regression Copy Section

• Existing benchmark PDFs are normal documents with near-zero replacement ratios
• New logic does not affect existing test/benchmark results

Not In Scope Copy Chapter

• New CLI options (no --cid-fallback or similar)
• npm run sync not required (no CLI option changes)
• API signature changes (backward compatible)
• Benchmark threshold changes

File: docs/superpowers/specs/2026-04-29-hybrid-hancom-ai-options-design.md Copy Chapter

Hybrid hancom-ai Backend Options & CLI Refactoring

Issue: not yet filed — this design precedes the issue/PRs.
Scope: opendataloader-pdf (core CLI) + opendataloader-pdfua (downstream CLI)

Problem Copy Chapter

Five hancom-ai backend behaviors are already wired through HybridConfig but not exposed on any CLI:

HancomAISchemaTransformer and HancomAIClient already read these via HybridConfig, but CLIOptions.java has no flags and applyHybridOptions() never sets them. They are reachable only through programmatic Config use.

A second, structural problem makes adding these flags painful:

• opendataloader-pdf defines all CLI options in CLIOptions.OPTION_DEFINITIONS (private, single source for options.json/Python/Node bindings).
• opendataloader-pdfua defines its own --hybrid, --hybrid-url, --hybrid-mode separately in Main.java, with different defaults, then forwards them through RemediationConfig (3 hybrid fields, 9 constructor overloads).

Adding 5 new flags to both CLIs the current way means duplicating definitions, expanding RemediationConfig to 8 hybrid fields, and creating yet more constructor overloads. The two CLIs will drift further apart.

Solution Copy Chapter

Two coupled changes, executed in order:

1. Add the 5 hancom-ai-specific options to core CLIOptions under a --hybrid-hancom-ai-* prefix, with a guard that rejects them when --hybrid is not hancom-ai.
2. Refactor core CLIOptions to be reusable so opendataloader-pdfua imports the full core option set and adds only its own pdfua-specific options on top. RemediationConfig embeds a core Config instead of carrying parallel hybrid fields.

After this, adding any future core CLI option propagates to pdfua with a rebuild — no Main.java edits.

Design Copy Chapter

Option naming: `--hybrid-hancom-ai-*` (full path) Copy Section

Decision rationale (recorded for future contributors):

• --hybrid-* alone (e.g. --hybrid-regionlist-strategy) was rejected because docling-fast and other backends will never support these knobs; the name would lie about scope.
• --hancom-ai-* alone (e.g. --hancom-ai-regionlist-strategy) was rejected because it breaks the existing --hybrid-mode/url/timeout/fallback grouping and gives users two mental models.
• --hybrid-hancom-ai-* mirrors the gh pr create / git remote add full-path convention: every option name encodes the context it belongs to. --help alphabetic sort keeps all hybrid options in one block.
• A single comma-separated --hybrid-hancom-ai-config mega-option was rejected: defeats Apache Commons CLI typo detection, breaks options.json codegen for Python/Node bindings, complicates Windows path escaping.

The five new options:

exported=false for the two debug options keeps them out of options.json and the auto-generated Python/Node bindings, while remaining usable on the Java CLI. Same pattern as existing legacy options at CLIOptions.java:201-208.

Validation Copy Section

Inside applyHybridOptions(), after --hybrid is parsed, before returning:

boolean usesHancomAiOnly =
    commandLine.hasOption(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION) ||
    commandLine.hasOption(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION) ||
    commandLine.hasOption(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION) ||
    commandLine.hasOption(HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION) ||
    commandLine.hasOption(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION);

if (usesHancomAiOnly && !Config.HYBRID_HANCOM_AI.equals(config.getHybrid())) {
    throw new IllegalArgumentException(
        "Options --hybrid-hancom-ai-* require --hybrid=hancom-ai");
}

Per-value validation (e.g. regionlistStrategy must be table-first/list-only) is already implemented in HybridConfig setters and re-thrown as IllegalArgumentException. CLI passes the raw value through; HybridConfig is the validation authority.

Core CLIOptions refactoring Copy Section

Goal: pdfua can register every core option in its own Options and ask core to apply them to a Config.

Two new public static methods on CLIOptions:

/** Register every core option onto an external Options. Used by downstream CLIs (pdfua). */
public static void addAllTo(Options options) {
    for (OptionDefinition def : OPTION_DEFINITIONS) {
        options.addOption(def.toOption());
    }
}

/** Apply parsed core options to a Config. Used by downstream CLIs after parse. */
public static void applyAllTo(Config config, CommandLine commandLine) {
    // body identical to current createConfigFromCommandLine,
    // minus `new Config()` and minus the positional-arg output-folder fallback
}

The existing defineOptions() and createConfigFromCommandLine() keep their signatures and behavior; internally they delegate to the two new methods. Backward compatibility for any existing callers is preserved.

OPTION_DEFINITIONS stays private. OptionDefinition stays private. We expose only the two operations downstream CLIs actually need.

pdfua/Main.java refactoring Copy Section

Replace the self-defined hybrid option block (Main.java:60-62, 99-101) with:

Options options = new Options();
CLIOptions.addAllTo(options);                 // all core options
options.addOption(null, "lang", true, ...);   // pdfua-specific only
options.addOption(null, "audit-bundle-mode", true, ...);
options.addOption(null, "font-embed-mode", true, ...);
options.addOption(null, "conformance", true, ...);
// ... other pdfua-only options

CommandLine cmd = parser.parse(options, args);

Config config = new Config();
CLIOptions.applyAllTo(config, cmd);
applyPdfuaDefaults(config);                   // pdfua's hybrid defaults

pdfua's hybrid defaults Copy Section

pdfua currently hardcodes different defaults from core: hybrid=hancom-ai, hybrid-url=http://localhost:18008, hybrid-mode=full (Main.java:60-62). After refactoring, these become an explicit override block:

private static void applyPdfuaDefaults(Config config) {
    if (Config.HYBRID_OFF.equals(config.getHybrid())) {
        config.setHybrid(Config.HYBRID_HANCOM_AI);
    }
    if (config.getHybridConfig().getUrl() == null) {
        config.getHybridConfig().setUrl("http://localhost:18008");
    }
    if (Config.HYBRID_MODE_AUTO.equals(config.getHybridConfig().getMode())) {
        config.getHybridConfig().setMode(Config.HYBRID_MODE_FULL);
    }
}

This makes pdfua's deviation from core defaults explicit and grep-able. User-supplied flags still win — the override only fires when the user did not specify a value.

RemediationConfig refactoring (Hard break) Copy Section

Current state: 3 flat hybrid fields (hybrid, hybridUrl, hybridMode) and 9 constructor overloads (RemediationConfig.java:42-105).

Target state: embed core Config directly. Keep only pdfua-specific fields. Builder replaces the constructor overloads.

public class RemediationConfig {
    private final Config coreConfig;          // ← core options live here

    // pdfua-only fields
    private final String input, output, lang;
    private final AuditBundleMode auditBundleMode;
    private final FontEmbedMode fontEmbedMode;
    private final List<String> conformances;
    private final int threads;
    private final boolean enrichPictureDescription;

    private RemediationConfig(Builder b) { ... }

    public Config getCoreConfig() { return coreConfig; }
    public String getHybrid() { return coreConfig.getHybrid(); }
    public HybridConfig getHybridConfig() { return coreConfig.getHybridConfig(); }
    // ... pdfua-only getters

    public static Builder builder() { return new Builder(); }
    public static class Builder { ... }
}

getHybridUrl() and getHybridMode() are removed (Hard break). All call sites move to getHybridConfig().getUrl() / getHybridConfig().getMode(). Verified call sites are confined to test code (AuditBundleEmitterTest, CertificateIssuerTest, AuditManifestBuilderTest, RemediationConfigAuditBundleTest) and RemediationProcessor.java:165-170. No external SDK consumers identified.

RemediationProcessor.java:163-170 simplifies to:

Config config = remediationConfig.getCoreConfig();
// (no per-field copy needed — Config already carries all hybrid state)

Files touched Copy Section

opendataloader-pdf (core)

• java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java
  • Add 5 option constants (long names + descriptions)
  • Add 5 entries to OPTION_DEFINITIONS (3 exported, 2 not)
  • Extend applyHybridOptions(): parse, set on HybridConfig, validate hancom-ai gate
  • Add public addAllTo(Options) and applyAllTo(Config, CommandLine)
  • defineOptions() and createConfigFromCommandLine() delegate to the new methods
• java/opendataloader-pdf-cli/src/test/java/org/opendataloader/pdf/cli/CLIOptionsTest.java
  • Tests for parsing each new option
  • Tests for the hancom-ai gate (using flag without --hybrid=hancom-ai throws)
• After core changes: run npm run sync to regenerate options.json + Python/Node bindings

opendataloader-pdfua

• src/main/java/org/opendataloader/pdf/Main.java
  • Replace self-defined hybrid options with CLIOptions.addAllTo(options)
  • Add applyPdfuaDefaults(config) helper
  • Pass coreConfig to RemediationConfig.Builder instead of individual hybrid strings
• src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java
  • Embed Config coreConfig, drop hybrid/hybridUrl/hybridMode fields
  • Replace 9 constructors with one Builder
  • Update getters: getHybrid() delegates, getHybridUrl()/getHybridMode() removed
• src/main/java/org/opendataloader/pdf/remediation/RemediationProcessor.java
  • Simplify the lines 163-170 hybrid forwarding block
• All test files instantiating RemediationConfig (5 files identified): switch to Builder

Build & verification Copy Section

Per opendataloader-pdfua/CLAUDE.md:

1. Core changes first: cd opendataloader-pdf/java && mvn install -DskipTests
2. Then pdfua: cd opendataloader-pdfua && mvn clean package
3. Run both test suites
4. Manually verify:
   • opendataloader-pdf input.pdf --hybrid=hancom-ai --hybrid-hancom-ai-regionlist-strategy=list-only works
   • opendataloader-pdf input.pdf --hybrid-hancom-ai-regionlist-strategy=list-only (no --hybrid) fails with clear error
   • opendataloader-pdfua input.pdf --hybrid-hancom-ai-ocr-strategy=force works (inherits core option, pdfua defaults still apply)

Phasing Copy Section

The work splits cleanly into independent commits/PRs:

1. PR 1 (core): Add 5 options + gate validation + tests + npm run sync. Self-contained, mergeable alone.
2. PR 2 (core): Extract addAllTo / applyAllTo public API. Pure refactoring, no behavior change.
3. PR 3 (pdfua): Switch Main.java to use core's API; refactor RemediationConfig to Builder + embedded Config; update tests.

PR 1 unblocks immediate user value. PRs 2 & 3 are coupled (PR 3 depends on PR 2 merge → core artifact rebuild) but neither blocks PR 1.

Out of scope Copy Chapter

• Migrating other parts of pdfua to a Builder-style config (only RemediationConfig is touched).
• Adding flags to docling-fast or other hybrid backends. Spec is hancom-ai-specific.
• Changing HybridConfig setter validation (already strict).
• Renaming or repackaging existing --hybrid-* options.
• Documentation site updates — opendataloader.org reference docs are CI-generated from options.json after core merge.

File: docs/superpowers/plans/2026-03-16-cid-font-detection.md Copy Chapter

CID Font Extraction Failure Detection — Implementation Plan

For agentic workers: REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Detect pages where CID font extraction failed (high U+FFFD ratio), emit warning logs, and auto-route to OCR backend in hybrid mode.

Architecture: Measure replacement character ratio in ContentFilterProcessor before replacement, store in StaticLayoutContainers, consume in TriageProcessor as highest-priority signal. Warning log fires regardless of hybrid mode.

Tech Stack: Java 11+, JUnit Jupiter, veraPDF API (ChunkParser.REPLACEMENT_CHARACTER_STRING)

File Structure Copy Chapter

All paths below are relative to java/opendataloader-pdf-core/src/.

Chunk 1: Measurement + Storage Copy Chapter

Task 1: Add per-page ratio storage to StaticLayoutContainers Copy Section

Files:

• Modify: main/java/org/opendataloader/pdf/containers/StaticLayoutContainers.java

• Step 1: Add ThreadLocal map field and imports

Add after line 40 (imageFormat field):

private static final ThreadLocal<Map> replacementCharRatios = ThreadLocal.withInitial(HashMap::new);

Add to imports:

import java.util.HashMap;
import java.util.Map;

• Step 2: Add getter and setter

Add after setImageFormat() (after line 145):

public static void setReplacementCharRatio(int pageNumber, double ratio) {
    replacementCharRatios.get().put(pageNumber, ratio);
}

public static double getReplacementCharRatio(int pageNumber) {
    return replacementCharRatios.get().getOrDefault(pageNumber, 0.0);
}

• Step 3: Clear in clearContainers()

Add inside clearContainers() method, after line 51 (imageFormat.set(...)):

replacementCharRatios.get().clear();

• Step 4: Compile check

Run: cd java && mvn compile -pl opendataloader-pdf-core -q
Expected: BUILD SUCCESS

• Step 5: Commit

git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/containers/StaticLayoutContainers.java
git commit -m "feat: add per-page replacement char ratio storage to StaticLayoutContainers"

Task 2: Add measureReplacementCharRatio to TextProcessor Copy Section

Files:

• Modify: main/java/org/opendataloader/pdf/processors/TextProcessor.java

• Test: test/java/org/opendataloader/pdf/processors/TextProcessorTest.java

• Step 1: Write failing tests

Add to TextProcessorTest.java after the last test method (before closing }):

/* Detailed source-code truncated for AI context efficiency. */

• Step 2: Run tests to verify they fail

Run: cd java && mvn test -pl opendataloader-pdf-core -Dtest=TextProcessorTest#testMeasureReplacementCharRatio_allReplacement -q
Expected: FAIL — measureReplacementCharRatio method not found

• Step 3: Implement measureReplacementCharRatio

Add to TextProcessor.java after replaceUndefinedCharacters() method (after line 53):

public static double measureReplacementCharRatio(List contents) {
    char replacementChar = ChunkParser.REPLACEMENT_CHARACTER_STRING.charAt(0);
    int totalChars = 0;
    int replacementChars = 0;
    for (IObject object : contents) {
        if (object instanceof TextChunk) {
            String value = ((TextChunk) object).getValue();
            totalChars += value.length();
            for (int i = 0; i < value.length(); i++) {
                if (value.charAt(i) == replacementChar) {
                    replacementChars++;
                }
            }
        }
    }
    if (totalChars == 0) {
        return 0.0;
    }
    return (double) replacementChars / totalChars;
}

• Step 4: Run tests to verify they pass

Run: cd java && mvn test -pl opendataloader-pdf-core -Dtest=TextProcessorTest -q
Expected: All tests PASS

• Step 5: Commit

git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/TextProcessor.java
git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/TextProcessorTest.java
git commit -m "feat: add measureReplacementCharRatio to TextProcessor

Counts U+FFFD replacement characters across TextChunks and returns
the ratio. Returns 0.0 for empty contents or pages with no text."

Chunk 2: Warning Log + Triage Routing Copy Chapter

Task 3: Add warning log in ContentFilterProcessor Copy Section

Files:

• Modify: main/java/org/opendataloader/pdf/processors/ContentFilterProcessor.java

• Step 1: Add import for StaticLayoutContainers

Add to imports:

import org.opendataloader.pdf.containers.StaticLayoutContainers;

• Step 2: Add measurement + warning before replaceUndefinedCharacters

Insert immediately before line 74 (TextProcessor.replaceUndefinedCharacters(...)) in getFilteredContents():

double replacementCharRatio = TextProcessor.measureReplacementCharRatio(pageContents);
StaticLayoutContainers.setReplacementCharRatio(pageNumber, replacementCharRatio);
if (replacementCharRatio >= 0.3) {
    LOGGER.log(Level.WARNING,
        "Page {0}: {1,number,#.#%} of characters are replacement characters (U+FFFD). "
        + "This PDF likely contains CID-keyed fonts without ToUnicode mappings. "
        + "Text extraction may be incomplete. Consider using --hybrid-mode for OCR fallback.",
        new Object[]{pageNumber + 1, replacementCharRatio});
}

• Step 3: Compile check

Run: cd java && mvn compile -pl opendataloader-pdf-core -q
Expected: BUILD SUCCESS

• Step 4: Commit

git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/ContentFilterProcessor.java
git commit -m "feat: detect CID font extraction failure and emit warning log

Measures U+FFFD ratio before replacement. Warns when >= 30% of
characters are replacement characters, suggesting hybrid mode."

Task 4: Add Signal 0 to TriageProcessor Copy Section

Files:

• Modify: main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java

• Test: test/java/org/opendataloader/pdf/hybrid/TriageProcessorTest.java

• Step 1: Write failing tests

Add to TriageProcessorTest.java before the // Helper methods comment:

@Test
public void testClassifyPage_highReplacementRatio_routesToBackend() {
    StaticLayoutContainers.clearContainers();
    StaticLayoutContainers.setReplacementCharRatio(0, 0.5);

    List contents = new ArrayList<>();
    contents.add(createTextChunk(10, 100, 200, 120, "text"));

    TriageResult result = TriageProcessor.classifyPage(contents, 0, new HybridConfig());

    Assertions.assertEquals(TriageDecision.BACKEND, result.getDecision());
    Assertions.assertEquals(1.0, result.getConfidence(), 0.001);
}

@Test
public void testClassifyPage_lowReplacementRatio_noEffect() {
    StaticLayoutContainers.clearContainers();
    StaticLayoutContainers.setReplacementCharRatio(0, 0.1);

    List contents = new ArrayList<>();
    contents.add(createTextChunk(10, 100, 200, 120, "normal text"));

    TriageResult result = TriageProcessor.classifyPage(contents, 0, new HybridConfig());

    Assertions.assertEquals(TriageDecision.JAVA, result.getDecision());
}

@Test
public void testClassifyPage_exactThreshold_routesToBackend() {
    StaticLayoutContainers.clearContainers();
    StaticLayoutContainers.setReplacementCharRatio(0, 0.3);

    List contents = new ArrayList<>();
    contents.add(createTextChunk(10, 100, 200, 120, "text"));

    TriageResult result = TriageProcessor.classifyPage(contents, 0, new HybridConfig());

    Assertions.assertEquals(TriageDecision.BACKEND, result.getDecision());
    Assertions.assertEquals(1.0, result.getConfidence(), 0.001);
}

• Step 2: Run tests to verify they fail

Run: cd java && mvn test -pl opendataloader-pdf-core -Dtest=TriageProcessorTest#testClassifyPage_highReplacementRatio_routesToBackend -q
Expected: FAIL — returns JAVA instead of BACKEND

• Step 3: Add Signal 0 to classifyPage

In TriageProcessor.java, in the classifyPage() method with TriageThresholds parameter, insert before the TableBorder check (before // Signal 1: TableBorder presence):

// Signal 0: CID font extraction failure (highest priority)
// Only fires in hybrid mode (classifyPage is only called from HybridDocumentProcessor)
double replacementRatio = StaticLayoutContainers.getReplacementCharRatio(pageNumber);
if (replacementRatio >= 0.3) {
    return TriageResult.backend(pageNumber, 1.0, signals);
}

Add import at top of file:

import org.opendataloader.pdf.containers.StaticLayoutContainers;

Also update the classifyPage() Javadoc (around line 617) to list Signal 0:

* Signal priority (highest to lowest):
 * <ol>
 *   <li>CID font extraction failure (replacement char ratio >= 30%)</li>
 *   <li>TableBorder presence</li>
 *   ...

• Step 4: Run tests to verify they pass

Run: cd java && mvn test -pl opendataloader-pdf-core -Dtest=TriageProcessorTest -q
Expected: All tests PASS

• Step 5: Run full test suite

Run: cd java && mvn test -pl opendataloader-pdf-core -q
Expected: All tests PASS — no regressions

• Step 6: Commit

git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java
git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/TriageProcessorTest.java
git commit -m "feat: add CID font detection signal to TriageProcessor

Signal 0 (highest priority): routes pages with >= 30% replacement
characters to backend for OCR fallback in hybrid mode."

Chunk 3: e2e Test + Test Fixture Copy Chapter

Task 5: Generate synthetic CID PDF test fixture Copy Section

Files:

• Create: java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py

• Create: java/opendataloader-pdf-core/src/test/resources/cid-font-no-tounicode.pdf

• Step 1: Create test resources directory

mkdir -p java/opendataloader-pdf-core/src/test/resources

• Step 2: Write the generation script

Create java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py.

This script generates a minimal PDF with a Type0 (CID) font that has no ToUnicode CMap. The PDF must cause veraPDF to emit \uFFFD for the majority of text characters when parsed.

Approach: Use raw PDF syntax to embed a CID font referencing CID values without a ToUnicode mapping. Alternatively, use reportlab which provides low-level CID font control.

The implementer should:

1. Generate the PDF
2. Verify it with opendataloader-pdf: cd java && mvn exec:java -pl opendataloader-pdf-cli -Dexec.mainClass="org.opendataloader.pdf.cli.CLIMain" -Dexec.args="../src/test/resources/cid-font-no-tounicode.pdf -f text" 2>&1
3. Confirm output is mostly empty/whitespace (indicating \uFFFD → space replacement)
4. Confirm WARNING log about replacement characters appears

If generating a proper CID PDF proves difficult, search odl-test-fixtures for an existing PDF with CID font issues, or use a known CID-problematic PDF from the Korean pharmaceutical fixtures (pdf-003 through pdf-007).

• Step 3: Commit fixture

git add java/opendataloader-pdf-core/src/test/resources/cid-font-no-tounicode.pdf
git add java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py
git commit -m "test: add synthetic CID font PDF test fixture

PDF with CID-keyed font without ToUnicode mapping for testing
replacement character detection. Generation script included."

Task 6: Write e2e integration test Copy Section

Files:

• Create: test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java

This test follows the pattern from TriageProcessorIntegrationTest.java:
DocumentProcessor.preprocessing() → ContentFilterProcessor.getFilteredContents() → assert ratio and warning.

• Step 1: Write the integration test

Create java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java:

/* Detailed source-code truncated for AI context efficiency. */

• Step 2: Run tests

Run: cd java && mvn test -pl opendataloader-pdf-core -Dtest=CidFontDetectionTest -q
Expected: Integration tests pass (or skip if fixture not yet generated). Boundary tests always pass.

• Step 3: Run full test suite

Run: cd java && mvn test -pl opendataloader-pdf-core -q
Expected: All tests PASS

• Step 4: Commit

git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java
git commit -m "test: add integration + boundary tests for CID font detection

Integration tests load cid-font-no-tounicode.pdf through full pipeline.
Boundary tests verify 29%/30% threshold behavior."

Chunk 4: Final Verification Copy Chapter

Task 7: Full test suite + benchmark regression check Copy Section

• Step 1: Run full Java test suite

Run: cd java && mvn test -q
Expected: All tests PASS across both core and cli modules

• Step 2: Check benchmark regression (if benchmark script exists)

Run: ./scripts/bench.sh --check-regression 2>/dev/null || echo "No benchmark script — skip"
Expected: PASS or skip

• Step 3: Review all changes

Run: git log --oneline main..HEAD
Expected commits (newest first):

1. test: add e2e tests for CID font detection pipeline
2. test: add synthetic CID font PDF test fixture
3. feat: add CID font detection signal to TriageProcessor
4. feat: detect CID font extraction failure and emit warning log
5. feat: add measureReplacementCharRatio to TextProcessor
6. feat: add per-page replacement char ratio storage to StaticLayoutContainers

• Step 4: Update issue #286 comment

Update the previously posted enhancement comment with the actual fix status. The earlier enhancement label should be removed since this is now a fix.

gh issue edit 286 --repo opendataloader-project/opendataloader-pdf --remove-label enhancement

File: docs/superpowers/plans/2026-04-18-hancom-ai-mock-server.md Copy Chapter

Hancom AI Mock Server Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Hancom AI 서버 부재 시 transformer 개발/디버그 사이클을 유지하기 위한 fixture-replay HTTP mock 서버 구현 + 클라이언트(HancomAIClient.java) REQUEST_ID 규약 패치.

Architecture: Python(FastAPI) 로컬 서버가 200개 벤치 PDF의 SHA256을 인덱싱하고, PDF 입력은 SHA256 룩업, 이미지 입력은 클라이언트가 REQUEST_ID에 인코딩한 (sha_short, page, obj, module)로 룩업, pdf2img는 PyMuPDF로 동적 300DPI 렌더링 + base64 응답. 클라이언트는 convert() 진입 시 PDF SHA256을 캐시하고 모든 호출에 REQUEST_ID 규약 적용.

Tech Stack:

• Mock 서버: Python 3.10+, FastAPI, uvicorn, PyMuPDF, pytest, httpx (TestClient)
• Client 패치: Java 8+, OkHttp, Jackson, Maven, JUnit 5
• 데이터: bundolee/kb-odl/raw/4-기술/2026-04-16_Q2-기술-ctx_hancom-ai-a11y_출력데이터-스키마/
• PDFs: opendataloader-project/opendataloader-bench/pdfs/

Spec: bundolee/kb-odl/raw/4-기술/2026-04-18_Q2-DEV-02-Code_hancom-ai-mock-server-design.md (commit dfe0671)

File Structure Copy Chapter

Mock 서버 (신규) Copy Section

opendataloader-project/opendataloader-pdfua/scripts/mock_server/

Client 패치 (수정) Copy Section

opendataloader-project/opendataloader-pdf/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/HancomAIClient.java

opendataloader-project/opendataloader-pdf/java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/HancomAIClientRequestIdTest.java (신규)

Phase 1 — Mock Server (10 tasks) Copy Chapter

Task 1: 프로젝트 스캐폴딩 Copy Section

Files:

• Create: opendataloader-project/opendataloader-pdfua/scripts/mock_server/pyproject.toml

• Create: opendataloader-project/opendataloader-pdfua/scripts/mock_server/__init__.py

• Create: opendataloader-project/opendataloader-pdfua/scripts/mock_server/tests/__init__.py

• Create: opendataloader-project/opendataloader-pdfua/scripts/mock_server/README.md

• Step 1: pyproject.toml 작성

[project]
name = "hancom-ai-mock-server"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = [
    "fastapi>=0.110",
    "uvicorn>=0.27",
    "pymupdf>=1.24",
    "python-multipart>=0.0.9",
]

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "httpx>=0.27",
]

[tool.pytest.ini_options]
testpaths = ["tests"]

• Step 2: 빈 패키지 마커 + README 초안

mock_server/__init__.py:

"""Hancom AI HOCR SDK fixture-replay mock server."""

mock_server/tests/__init__.py:

mock_server/README.md:

# Hancom AI Mock Server

Fixture-replay mock for `HancomAIClient`. See spec at
`bundolee/kb-odl/raw/4-기술/2026-04-18_Q2-DEV-02-Code_hancom-ai-mock-server-design.md`.

## Run

```bash
cd opendataloader-pdfua/scripts/mock_server
pip install -e ".[dev]"
python -m mock_server \
  --pdf-dir /path/to/opendataloader-bench/pdfs \
  --fixture-dir /path/to/kb-odl/raw/4-기술/2026-04-16_Q2-기술-ctx_hancom-ai-a11y_출력데이터-스키마 \
  --port 18008

Tests Copy Chapter

pytest -v

- [ ] **Step 3: 디렉토리 구조 확인**

Run: `ls opendataloader-project/opendataloader-pdfua/scripts/mock_server/`
Expected: `pyproject.toml  README.md  __init__.py  tests/`

- [ ] **Step 4: Commit**

```bash
cd opendataloader-project/opendataloader-pdfua
git add scripts/mock_server/
git commit -m "feat(mock-server): scaffold project structure"

Task 2: REQUEST_ID 파서 (TDD) Copy Section

Files:

• Test: opendataloader-project/opendataloader-pdfua/scripts/mock_server/tests/test_request_id.py

• Create: opendataloader-project/opendataloader-pdfua/scripts/mock_server/mock_server/request_id.py

• Step 1: 실패하는 테스트 작성

tests/test_request_id.py:

import pytest
from mock_server.request_id import parse_request_id, RequestIdParts


def test_parse_caption():
    parts = parse_request_id("odl-a3f1c9d2e7b8-p0-o5-caption")
    assert parts == RequestIdParts(sha_short="a3f1c9d2e7b8", page=0, obj=5, module="caption")


def test_parse_chart():
    parts = parse_request_id("odl-deadbeef0000-p3-o12-chart")
    assert parts == RequestIdParts(sha_short="deadbeef0000", page=3, obj=12, module="chart")


def test_parse_tsr():
    parts = parse_request_id("odl-aabbccddeeff-p0-o0-tsr")
    assert parts.module == "tsr"


def test_parse_invalid_returns_none():
    assert parse_request_id("odl-DOCUMENT_LAYOUT_WITH_OCR") is None
    assert parse_request_id("garbage") is None
    assert parse_request_id("") is None
    assert parse_request_id("odl-a3f1-p0-o0-unknown") is None  # bad module


def test_parse_pdf_module_request_id_returns_none():
    # PDF modules use REQUEST_ID for tracing only; mock matches via SHA256 of bytes.
    # Parser returns None for these so caller falls back to PDF lookup path.
    assert parse_request_id("odl-a3f1c9d2e7b8-dla-ocr") is None

• Step 2: 테스트 실행 (실패 확인)

Run: cd opendataloader-project/opendataloader-pdfua/scripts/mock_server && pytest tests/test_request_id.py -v
Expected: FAIL — ModuleNotFoundError: No module named 'mock_server.request_id'

• Step 3: 최소 구현

mock_server/request_id.py:

"""Parse client REQUEST_ID for image-input module lookup."""
import re
from dataclasses import dataclass
from typing import Optional

_PATTERN = re.compile(
    r"^odl-(?P<sha>[0-9a-f]{12})-p(?P\d+)-o(?P<obj>\d+)-(?P<module>caption|chart|tsr)$"
)


@dataclass(frozen=True)
class RequestIdParts:
    sha_short: str
    page: int
    obj: int
    module: str


def parse_request_id(request_id: str) -> Optional[RequestIdParts]:
    if not request_id:
        return None
    m = _PATTERN.match(request_id)
    if not m:
        return None
    return RequestIdParts(
        sha_short=m.group("sha"),
        page=int(m.group("page")),
        obj=int(m.group("obj")),
        module=m.group("module"),
    )

• Step 4: 테스트 통과 확인

Run: pytest tests/test_request_id.py -v
Expected: 5 passed

• Step 5: Commit

git add scripts/mock_server/mock_server/request_id.py scripts/mock_server/tests/test_request_id.py
git commit -m "feat(mock-server): REQUEST_ID parser for image module lookup"

Task 3: SHA256 인덱스 빌더 (TDD) Copy Section

Files:

• Test: scripts/mock_server/tests/test_index.py

• Test: scripts/mock_server/tests/conftest.py

• Create: scripts/mock_server/mock_server/index.py

• Step 1: conftest fixture 작성

tests/conftest.py:

import hashlib
import shutil
from pathlib import Path
import pytest


@pytest.fixture
def sample_pdf_dir(tmp_path):
    """Create a tiny PDF dir with 3 fake PDFs (just bytes, not parsed by index)."""
    d = tmp_path / "pdfs"
    d.mkdir()
    for i in range(1, 4):
        (d / f"pdf{i:03d}.pdf").write_bytes(f"FAKE_PDF_CONTENT_{i}".encode())
    return d


def sha256_hex(b: bytes) -> str:
    return hashlib.sha256(b).hexdigest()

• Step 2: 실패하는 테스트 작성

tests/test_index.py:

from mock_server.index import build_pdf_index, PdfIndexEntry
from .conftest import sha256_hex


def test_build_index_three_pdfs(sample_pdf_dir):
    index = build_pdf_index(sample_pdf_dir)
    assert len(index) == 3
    expected_sha = sha256_hex(b"FAKE_PDF_CONTENT_1")
    assert expected_sha in index
    entry = index[expected_sha]
    assert entry.basename == "pdf001"
    assert entry.path == sample_pdf_dir / "pdf001.pdf"


def test_build_index_empty_dir(tmp_path):
    index = build_pdf_index(tmp_path)
    assert index == {}


def test_build_index_sha_short_lookup(sample_pdf_dir):
    index = build_pdf_index(sample_pdf_dir)
    full_sha = next(iter(index))
    short = full_sha[:12]
    entry = index.find_by_short(short)
    assert entry is not None
    assert entry.basename in {"pdf001", "pdf002", "pdf003"}

• Step 3: 테스트 실행 (실패 확인)

Run: pytest tests/test_index.py -v
Expected: FAIL — module not found

• Step 4: 최소 구현

mock_server/index.py:

"""Build SHA256 index of benchmark PDFs at server boot."""
import hashlib
from dataclasses import dataclass
from pathlib import Path
from typing import Dict, Optional


@dataclass(frozen=True)
class PdfIndexEntry:
    basename: str
    path: Path


class PdfIndex(dict):
    """Dict {full_sha256: PdfIndexEntry} with helper for short-prefix lookup."""

    def find_by_short(self, sha_short: str) -> Optional[PdfIndexEntry]:
        for full_sha, entry in self.items():
            if full_sha.startswith(sha_short):
                return entry
        return None


def build_pdf_index(pdf_dir: Path) -> PdfIndex:
    index = PdfIndex()
    for path in sorted(Path(pdf_dir).glob("*.pdf")):
        sha = hashlib.sha256(path.read_bytes()).hexdigest()
        index[sha] = PdfIndexEntry(basename=path.stem, path=path)
    return index

• Step 5: 테스트 통과 확인

Run: pytest tests/test_index.py -v
Expected: 3 passed

• Step 6: Commit

git add scripts/mock_server/
git commit -m "feat(mock-server): SHA256 PDF index with short-prefix lookup"

Task 4: Lookup 로직 (TDD) Copy Section

Files:

• Test: scripts/mock_server/tests/test_lookup.py

• Test: scripts/mock_server/tests/conftest.py (확장)

• Create: scripts/mock_server/mock_server/lookup.py

• Step 1: conftest 확장 — 가짜 fixture dir

tests/conftest.py 끝에 추가:

import json


@pytest.fixture
def sample_fixture_dir(tmp_path):
    """Mimic the recorded data layout with one entry per module."""
    root = tmp_path / "fixtures"
    for sub in ["DLA_OCR", "DLA", "OCR", "TSR", "TSR_regionlist", "FIGURE"]:
        (root / sub).mkdir(parents=True)
    (root / "DLA_OCR" / "pdf001.json").write_text(json.dumps({"module": "DLA_OCR", "id": "pdf001"}))
    (root / "DLA" / "pdf001.json").write_text(json.dumps({"module": "DLA"}))
    (root / "OCR" / "pdf001.json").write_text(json.dumps({"module": "OCR"}))
    (root / "TSR" / "pdf002_p0_o0.json").write_text(json.dumps({"module": "TSR", "table": True}))
    (root / "TSR_regionlist" / "pdf003_p0_o0.json").write_text(json.dumps({"module": "TSR_regionlist"}))
    (root / "FIGURE" / "pdf001_p0_o5_caption.json").write_text(json.dumps({"caption": "test"}))
    (root / "FIGURE" / "pdf001_p0_o5_chart.json").write_text(json.dumps({"understanding": "TITLE | <0x0A> 1 | 2"}))
    return root

• Step 2: 실패 테스트 작성

tests/test_lookup.py:

/* Detailed source-code truncated for AI context efficiency. */

• Step 3: 실패 확인

Run: pytest tests/test_lookup.py -v
Expected: FAIL — module not found

• Step 4: 구현

mock_server/lookup.py:

"""Map (module, key) → fixture JSON path and load. Raises FixtureMiss on miss."""
import json
from pathlib import Path
from typing import Any


class FixtureMiss(Exception):
    """Raised when no fixture file matches the request."""


MODULE_TO_DIR = {
    "DOCUMENT_LAYOUT_WITH_OCR": "DLA_OCR",
    "DOCUMENT_LAYOUT_ANALYSIS": "DLA",
    "TEXT_RECOGNITION": "OCR",
    "TABLE_STRUCTURE_RECOGNITION": "TSR",
}


def lookup_pdf_module(fixture_dir: Path, module_name: str, basename: str) -> Any:
    sub = MODULE_TO_DIR.get(module_name)
    if sub is None:
        raise FixtureMiss(f"unknown PDF-input module: {module_name}")
    path = Path(fixture_dir) / sub / f"{basename}.json"
    if not path.exists():
        raise FixtureMiss(f"no fixture at {sub}/{basename}.json")
    return json.loads(path.read_text())


def lookup_image_module(
    fixture_dir: Path, module_short: str, basename: str, page: int, obj: int
) -> Any:
    if module_short in {"caption", "chart"}:
        path = Path(fixture_dir) / "FIGURE" / f"{basename}_p{page}_o{obj}_{module_short}.json"
        if not path.exists():
            raise FixtureMiss(f"no FIGURE fixture: {path.name}")
        return json.loads(path.read_text())
    if module_short == "tsr":
        primary = Path(fixture_dir) / "TSR" / f"{basename}_p{page}_o{obj}.json"
        fallback = Path(fixture_dir) / "TSR_regionlist" / f"{basename}_p{page}_o{obj}.json"
        for candidate in (primary, fallback):
            if candidate.exists():
                return json.loads(candidate.read_text())
        raise FixtureMiss(f"no TSR fixture: {basename}_p{page}_o{obj}.json (TSR/ or TSR_regionlist/)")
    raise FixtureMiss(f"unknown image module: {module_short}")

• Step 5: 테스트 통과 확인

Run: pytest tests/test_lookup.py -v
Expected: 10 passed

• Step 6: Commit

git add scripts/mock_server/
git commit -m "feat(mock-server): lookup with TSR_regionlist fallback"

Task 5: PDF Renderer (TDD) Copy Section

Files:

• Test: scripts/mock_server/tests/test_pdf_render.py

• Create: scripts/mock_server/mock_server/pdf_render.py

• Step 1: 실패 테스트

tests/test_pdf_render.py:

import io
import pytest
import fitz  # PyMuPDF
from mock_server.pdf_render import render_page_png


@pytest.fixture
def real_pdf(tmp_path):
    """Create a minimal 2-page PDF using PyMuPDF itself."""
    doc = fitz.open()
    doc.new_page(width=612, height=792)  # 1 page
    doc.new_page(width=612, height=792)  # 2 pages
    p = tmp_path / "minimal.pdf"
    doc.save(p)
    doc.close()
    return p


def test_render_first_page_returns_png_bytes(real_pdf):
    png_bytes = render_page_png(real_pdf, page_index=0, dpi=300)
    assert png_bytes[:8] == b"\x89PNG\r\n\x1a\n"


def test_render_dimensions_300dpi(real_pdf):
    from PIL import Image
    png_bytes = render_page_png(real_pdf, page_index=0, dpi=300)
    img = Image.open(io.BytesIO(png_bytes))
    # 612pt at 300dpi / 72pt-per-inch = 2550px (±1 for rounding)
    assert abs(img.width - 2550) <= 2
    assert abs(img.height - 3300) <= 2


def test_render_out_of_range_raises(real_pdf):
    with pytest.raises(IndexError):
        render_page_png(real_pdf, page_index=99)

Note: Pillow is a transitive dep of PyMuPDF; if not, add to dev deps.

• Step 2: 실패 확인

Run: pytest tests/test_pdf_render.py -v
Expected: FAIL — module not found

• Step 3: 구현

mock_server/pdf_render.py:

"""Render a PDF page to PNG bytes using PyMuPDF."""
from pathlib import Path
import fitz


def render_page_png(pdf_path: Path, page_index: int, dpi: int = 300) -> bytes:
    doc = fitz.open(str(pdf_path))
    try:
        if page_index < 0 or page_index >= doc.page_count:
            raise IndexError(f"page_index {page_index} out of range (0..{doc.page_count - 1})")
        page = doc.load_page(page_index)
        zoom = dpi / 72.0
        matrix = fitz.Matrix(zoom, zoom)
        pixmap = page.get_pixmap(matrix=matrix, alpha=False)
        return pixmap.tobytes("png")
    finally:
        doc.close()

• Step 4: pyproject.toml dev deps에 Pillow 추가 (테스트용)

Edit scripts/mock_server/pyproject.toml, replace [project.optional-dependencies] block with:

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "httpx>=0.27",
    "pillow>=10.0",
]

Reinstall: pip install -e ".[dev]"

• Step 5: 테스트 통과 확인

Run: pytest tests/test_pdf_render.py -v
Expected: 3 passed

• Step 6: Commit

git add scripts/mock_server/
git commit -m "feat(mock-server): PyMuPDF page renderer at 300dpi"

Task 6: FastAPI Server — /ping + /hocr/sdk PDF 분기 (TDD) Copy Section

Files:

• Test: scripts/mock_server/tests/test_server.py

• Create: scripts/mock_server/mock_server/server.py

• Step 1: 실패 테스트 (ping + DLA_OCR happy path + unknown PDF)

tests/test_server.py:

/* Detailed source-code truncated for AI context efficiency. */

• Step 2: 실패 확인

Run: pytest tests/test_server.py -v
Expected: FAIL — module not found

• Step 3: 구현

mock_server/server.py:

/* Detailed source-code truncated for AI context efficiency. */

• Step 4: 테스트 통과 확인

Run: pytest tests/test_server.py -v
Expected: 5 passed (ping x2, DLA_OCR happy x2, unknown PDF)

• Step 5: Commit

git add scripts/mock_server/
git commit -m "feat(mock-server): /ping + /hocr/sdk PDF branch with /api/v1 alias"

Task 7: 이미지 분기 추가 (TDD) Copy Section

Files:

• Modify: scripts/mock_server/mock_server/server.py

• Modify: scripts/mock_server/tests/test_server.py

• Step 1: 추가 테스트 작성

Append to tests/test_server.py:

/* Detailed source-code truncated for AI context efficiency. */

• Step 2: 실패 확인

Run: pytest tests/test_server.py -v -k image
Expected: 4 failed (image branch returns FIXTURE_MISS placeholder)

• Step 3: 이미지 분기 구현

In mock_server/server.py, replace the image branch placeholder line # image branch (Task 7) and the line below it with:

# image branch
        parts = parse_request_id(REQUEST_ID)
        if parts is None:
            hint = f"REQUEST_ID does not match odl-<sha12>-p<n>-o<n>-<caption|chart|tsr>"
            LOGGER.warning("FIXTURE_MISS bad REQUEST_ID: %s", REQUEST_ID)
            return _envelope(REQUEST_ID, False, "FIXTURE_MISS", [], hint)
        entry = pdf_index.find_by_short(parts.sha_short)
        if entry is None:
            hint = f"sha_short={parts.sha_short} not in {len(pdf_index)}-PDF index"
            LOGGER.warning("FIXTURE_MISS unknown sha_short: %s", parts.sha_short)
            return _envelope(REQUEST_ID, False, "FIXTURE_MISS", [], hint)
        try:
            data = lookup_image_module(
                Path(fixture_dir), parts.module, entry.basename, parts.page, parts.obj
            )
        except FixtureMiss as e:
            LOGGER.warning("FIXTURE_MISS image-lookup: %s", e)
            return _envelope(REQUEST_ID, False, "FIXTURE_MISS", [], str(e))
        return _envelope(REQUEST_ID, True, "SUCCESS", data)

• Step 4: 테스트 통과 확인

Run: pytest tests/test_server.py -v
Expected: 9 passed (5 from Task 6 + 4 new)

• Step 5: Commit

git add scripts/mock_server/
git commit -m "feat(mock-server): image module branch with REQUEST_ID parsing"

Task 8: pdf2img 엔드포인트 (TDD) Copy Section

Files:

• Modify: scripts/mock_server/mock_server/server.py

• Modify: scripts/mock_server/tests/test_server.py

• Modify: scripts/mock_server/tests/conftest.py (real PDF fixture)

• Step 1: conftest에 real PDF 추가 + 인덱스에 포함

Edit tests/conftest.py, replace sample_pdf_dir with:

@pytest.fixture
def sample_pdf_dir(tmp_path):
    """Create a tiny PDF dir: 3 fake byte files + 1 real PDF for pdf2img tests."""
    import fitz
    d = tmp_path / "pdfs"
    d.mkdir()
    for i in range(1, 4):
        (d / f"pdf{i:03d}.pdf").write_bytes(f"FAKE_PDF_CONTENT_{i}".encode())
    # real 2-page PDF named pdf004.pdf
    doc = fitz.open()
    doc.new_page(width=612, height=792)
    doc.new_page(width=612, height=792)
    doc.save(d / "pdf004.pdf")
    doc.close()
    return d

• Step 2: 실패 테스트 추가

Append to tests/test_server.py:

import base64


def test_pdf2img_returns_base64_png(client, sample_pdf_dir):
    pdf_bytes = (sample_pdf_dir / "pdf004.pdf").read_bytes()
    r = client.post(
        "/support/pdf2img",
        data={"REQUEST_ID": "odl-pdf2img-0", "PAGE_INDEX": "0"},
        files={"FILE": ("document.pdf", pdf_bytes, "application/pdf")},
    )
    assert r.status_code == 200
    body = r.json()
    assert body["SUCCESS"] is True
    png_b64 = body["RESULT"][0]["RESULT"]["PAGE_PNG_DATA"]
    assert png_b64
    assert base64.b64decode(png_b64)[:8] == b"\x89PNG\r\n\x1a\n"


def test_pdf2img_via_v1_alias(client, sample_pdf_dir):
    pdf_bytes = (sample_pdf_dir / "pdf004.pdf").read_bytes()
    r = client.post(
        "/api/v1/support/pdf2img",
        data={"REQUEST_ID": "x", "PAGE_INDEX": "1"},
        files={"FILE": ("d.pdf", pdf_bytes, "application/pdf")},
    )
    assert r.json()["SUCCESS"] is True


def test_pdf2img_unknown_pdf(client):
    r = client.post(
        "/support/pdf2img",
        data={"REQUEST_ID": "x", "PAGE_INDEX": "0"},
        files={"FILE": ("u.pdf", b"NOT_INDEXED", "application/pdf")},
    )
    body = r.json()
    assert body["SUCCESS"] is False
    assert body["MSG"] == "FIXTURE_MISS"


def test_pdf2img_out_of_range(client, sample_pdf_dir):
    pdf_bytes = (sample_pdf_dir / "pdf004.pdf").read_bytes()
    r = client.post(
        "/support/pdf2img",
        data={"REQUEST_ID": "x", "PAGE_INDEX": "99"},
        files={"FILE": ("d.pdf", pdf_bytes, "application/pdf")},
    )
    body = r.json()
    assert body["SUCCESS"] is False
    assert "page_index" in body["_mock_hint"]

• Step 3: 실패 확인

Run: pytest tests/test_server.py -v -k pdf2img
Expected: 4 failed (404 from FastAPI — endpoint not registered)

• Step 4: pdf2img 라우트 추가

In mock_server/server.py, inside create_app after the /hocr/sdk route, add:

@router.post("/support/pdf2img")
    async def pdf2img(
        REQUEST_ID: str = Form(""),
        PAGE_INDEX: int = Form(...),
        FILE: UploadFile = File(...),
    ):
        body = await FILE.read()
        sha = hashlib.sha256(body).hexdigest()
        entry = pdf_index.get(sha)
        if entry is None:
            hint = f"sha256={sha[:12]}... not in {len(pdf_index)}-PDF index"
            return _envelope(REQUEST_ID, False, "FIXTURE_MISS", [], hint)
        try:
            png_bytes = render_page_png(entry.path, PAGE_INDEX, dpi=300)
        except IndexError as e:
            return _envelope(REQUEST_ID, False, "FIXTURE_MISS", [], str(e))
        import base64
        b64 = base64.b64encode(png_bytes).decode("ascii")
        return _envelope(
            REQUEST_ID, True, "SUCCESS",
            {"RESULT": {"PAGE_PNG_DATA": b64}},
        )

• Step 5: 테스트 통과 확인

Run: pytest tests/test_server.py -v
Expected: 13 passed

• Step 6: Commit

git add scripts/mock_server/
git commit -m "feat(mock-server): /support/pdf2img with PyMuPDF dynamic rendering"

Task 9: CLI 엔트리포인트 Copy Section

Files:

• Create: scripts/mock_server/mock_server/__main__.py

• Step 1: 구현

mock_server/__main__.py:

"""CLI: python -m mock_server --pdf-dir ... --fixture-dir ... --port 18008"""
import argparse
import logging
import sys
from pathlib import Path

import uvicorn

from .server import create_app


def main(argv=None):
    parser = argparse.ArgumentParser(prog="mock_server")
    parser.add_argument("--pdf-dir", required=True, type=Path)
    parser.add_argument("--fixture-dir", required=True, type=Path)
    parser.add_argument("--port", type=int, default=18008)
    parser.add_argument("--host", default="127.0.0.1")
    args = parser.parse_args(argv)

    logging.basicConfig(
        level=logging.INFO,
        format="%(asctime)s %(levelname)s %(name)s %(message)s",
    )

    if not args.pdf_dir.is_dir():
        print(f"--pdf-dir not found: {args.pdf_dir}", file=sys.stderr)
        sys.exit(2)
    if not args.fixture_dir.is_dir():
        print(f"--fixture-dir not found: {args.fixture_dir}", file=sys.stderr)
        sys.exit(2)

    app = create_app(pdf_dir=args.pdf_dir, fixture_dir=args.fixture_dir)
    uvicorn.run(app, host=args.host, port=args.port, log_level="info")


if __name__ == "__main__":
    main()

• Step 2: smoke test — CLI 인자 검증

Run:

cd opendataloader-project/opendataloader-pdfua/scripts/mock_server
python -m mock_server --pdf-dir /nonexistent --fixture-dir /nonexistent

Expected: exit code 2, stderr --pdf-dir not found: /nonexistent

• Step 3: 실제 데이터로 부팅 smoke test

Run (in a separate terminal or background, then kill):

python -m mock_server \
  --pdf-dir /Users/benedict/Workspace/opendataloader-project/opendataloader-bench/pdfs \
  --fixture-dir /Users/benedict/Workspace/bundolee/kb-odl/raw/4-기술/2026-04-16_Q2-기술-ctx_hancom-ai-a11y_출력데이터-스키마 \
  --port 18008 &
sleep 2
curl -s http://127.0.0.1:18008/ping
kill %1

Expected: log indexed 200 PDFs, curl returns {"status":"ok"}.

• Step 4: Commit

git add scripts/mock_server/
git commit -m "feat(mock-server): CLI entrypoint with arg validation"

Task 10: README 마무리 + E2E 절차 기록 Copy Section

Files:

• Modify: scripts/mock_server/README.md

• Step 1: README 확장

Replace mock_server/README.md with:

# Hancom AI Mock Server

Fixture-replay mock for [HancomAIClient.java](../../../opendataloader-pdf/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/HancomAIClient.java).

Spec: `bundolee/kb-odl/raw/4-기술/2026-04-18_Q2-DEV-02-Code_hancom-ai-mock-server-design.md`

## Setup

```bash
cd opendataloader-pdfua/scripts/mock_server
pip install -e ".[dev]"

Run Copy Chapter

python -m mock_server \
  --pdf-dir /Users/benedict/Workspace/opendataloader-project/opendataloader-bench/pdfs \
  --fixture-dir /Users/benedict/Workspace/bundolee/kb-odl/raw/4-기술/2026-04-16_Q2-기술-ctx_hancom-ai-a11y_출력데이터-스키마 \
  --port 18008

Expect log: indexed 200 PDFs, listening on :18008.

Endpoints Copy Chapter

REQUEST_ID Convention Copy Chapter

Image-input modules (TSR / IMAGE_CAPTIONING / CHART_IMAGE_UNDERSTANDING) require:

odl-{sha_short}-p{page}-o{obj}-{module_short}

• sha_short = first 12 hex chars of source PDF SHA256
• module_short ∈ {caption, chart, tsr}

PDF-input modules (DLA_OCR / DLA / OCR / TSR-pdf / pdf2img) match via SHA256 of FILE bytes; REQUEST_ID is informational.

The Java client patch (Phase 2 of the plan) builds these IDs automatically.

End-to-End Smoke Test Copy Chapter

After both Phase 1 (server) and Phase 2 (client patch) are merged:

# Terminal 1
python -m mock_server --pdf-dir ... --fixture-dir ... --port 18008

# Terminal 2 — pdfua against the mock
cd opendataloader-project/opendataloader-pdfua
mvn package -q
java -jar target/*.jar \
  --input /path/to/opendataloader-bench/pdfs/01030000000001.pdf \
  --output /tmp/out \
  --hybrid hancom-ai \
  --hybrid-url http://localhost:18008

Expected: tagged PDF appears in /tmp/out, server log shows DLA_OCR + pdf2img + TSR + IMAGE_CAPTIONING calls all returning SUCCESS.

Tests Copy Chapter

pytest -v

Expected: ~22 tests pass.

FIXTURE_MISS Debugging Copy Chapter

When SUCCESS:false, response includes _mock_hint:

• sha256=... not in 200-PDF index → input PDF is not in the recorded benchmark set; either add it or use a benchmark PDF.
• REQUEST_ID does not match odl-<sha12>-p<n>-o<n>-<...> → client did not apply the REQUEST_ID convention; check Phase 2 patch.
• no FIXTURE_<...> → the recorded set lacks this specific page/object; this can happen for new combinations.

- [ ] **Step 2: Commit**

```bash
git add scripts/mock_server/README.md
git commit -m "docs(mock-server): README with endpoints, REQUEST_ID convention, E2E smoke"

Phase 2 — Client Patch (4 tasks) Copy Chapter

Task 11: Test for REQUEST_ID building (TDD) Copy Section

Files:

• Create: opendataloader-project/opendataloader-pdf/java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/HancomAIClientRequestIdTest.java

• Step 1: 실패 테스트 작성

HancomAIClientRequestIdTest.java:

/* Detailed source-code truncated for AI context efficiency. */

• Step 2: pom.xml에 mockwebserver 의존성 확인

Run: grep -n "mockwebserver" opendataloader-project/opendataloader-pdf/java/opendataloader-pdf-core/pom.xml
Expected: dependency line exists. If not, add to pom.xml <dependencies>:

<dependency>
  <groupId>com.squareup.okhttp3</groupId>
  <artifactId>mockwebserver</artifactId>
  <version>4.12.0</version>
  <scope>test</scope>
</dependency>

• Step 3: 실패 확인

Run:

cd opendataloader-project/opendataloader-pdf/java
mvn -pl opendataloader-pdf-core test -Dtest=HancomAIClientRequestIdTest

Expected: COMPILE FAIL — invokeCallModule, invokeCallImageCaptioning, setSourcePdfShaShort not defined.

• Step 4: Commit (test only, expecting fail)

cd opendataloader-project/opendataloader-pdf
git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/HancomAIClientRequestIdTest.java java/opendataloader-pdf-core/pom.xml
git commit -m "test(hybrid): failing test for HancomAIClient REQUEST_ID convention"

Task 12: HancomAIClient 패치 — REQUEST_ID 빌드 Copy Section

Files:

• Modify: opendataloader-project/opendataloader-pdf/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/HancomAIClient.java

• Step 1: SHA256 헬퍼 + 인스턴스 필드 + 모듈 단축명 맵 추가

In HancomAIClient.java after the existing instance field declarations (after private final HybridConfig config;), add:

private String sourcePdfShaShort = "unknown";

    private static final java.util.Map<String, String> MODULE_SHORT;
    static {
        java.util.Map<String, String> m = new java.util.HashMap<>();
        m.put("DOCUMENT_LAYOUT_WITH_OCR", "dla-ocr");
        m.put("DOCUMENT_LAYOUT_ANALYSIS", "dla");
        m.put("TEXT_RECOGNITION", "ocr");
        m.put("TABLE_STRUCTURE_RECOGNITION", "tsr");
        m.put("IMAGE_CAPTIONING", "caption");
        m.put("CHART_IMAGE_UNDERSTANDING", "chart");
        MODULE_SHORT = java.util.Collections.unmodifiableMap(m);
    }

    private static String sha256ShortHex(byte[] data) {
        try {
            java.security.MessageDigest md = java.security.MessageDigest.getInstance("SHA-256");
            byte[] hash = md.digest(data);
            StringBuilder sb = new StringBuilder(12);
            for (int i = 0; i < 6; i++) sb.append(String.format("%02x", hash[i]));
            return sb.toString();
        } catch (java.security.NoSuchAlgorithmException e) {
            return "nohash000000";
        }
    }

    // Test hook
    void setSourcePdfShaShort(String s) { this.sourcePdfShaShort = s; }

• Step 2: convert(HybridRequest) 진입에서 SHA 계산

In convert(...) method, immediately after byte[] pdfBytes = request.getPdfBytes();, add:

this.sourcePdfShaShort = sha256ShortHex(pdfBytes);

• Step 3: callModule REQUEST_ID 변경

In callModule(byte[] pdfBytes, String moduleName) method, replace the line:

.addFormDataPart("REQUEST_ID", "odl-" + moduleName)

with:

.addFormDataPart("REQUEST_ID",
                "odl-" + sourcePdfShaShort + "-" + MODULE_SHORT.getOrDefault(moduleName, moduleName))

• Step 4: callModuleImage 시그니처 + REQUEST_ID 변경

Replace the entire callModuleImage method with:

/* Detailed source-code truncated for AI context efficiency. */

Update the single existing call site of callModuleImage (line ~410 in original, inside recognizeTableStructures) — find:

JsonNode tsrResult = callModuleImage(cropPng, "TABLE_STRUCTURE_RECOGNITION");

and replace with:

int objId = obj.has("object_id") ? obj.get("object_id").asInt() : -1;
                    JsonNode tsrResult = callModuleImage(cropPng, "TABLE_STRUCTURE_RECOGNITION", pageNum, objId);

• Step 5: callImageCaptioning 시그니처 + REQUEST_ID 변경

Replace the callImageCaptioning method signature and body with:

private String callImageCaptioning(byte[] pngBytes, int pageNum, int objectId) throws IOException {
        String requestId = "odl-" + sourcePdfShaShort + "-p" + pageNum + "-o" + objectId + "-caption";
        MultipartBody body = new MultipartBody.Builder()
            .setType(MultipartBody.FORM)
            .addFormDataPart("REQUEST_ID", requestId)
            .addFormDataPart("OPEN_API_NAME", "IMAGE_CAPTIONING")
            .addFormDataPart("DATA_FORMAT", "image")
            .addFormDataPart("FILE", "figure.png",
                RequestBody.create(pngBytes, MEDIA_TYPE_PNG))
            .build();

        Request httpRequest = new Request.Builder()
            .url(baseUrl + SDK_ENDPOINT)
            .post(body)
            .build();

        try (Response response = httpClient.newCall(httpRequest).execute()) {
            if (!response.isSuccessful()) return null;

            ResponseBody respBody = response.body();
            if (respBody == null) return null;

            JsonNode root = objectMapper.readTree(respBody.string());
            if (!root.has("SUCCESS") || !root.get("SUCCESS").asBoolean()) return null;

            JsonNode result = root.get("RESULT");
            if (result == null || !result.isArray() || result.size() == 0) return null;

            JsonNode page = result.get(0);
            if (page.isArray() && page.size() > 0) page = page.get(0);

            return page.has("caption") ? page.get("caption").asText("") : null;
        }
    }

Update the single existing call site (inside captionFigures, line ~294 in original) — find:

String caption = callImageCaptioning(croppedPng);

and replace with:

int objIdForCaption = fig.has("object_id") ? fig.get("object_id").asInt() : -1;
                    String caption = callImageCaptioning(croppedPng, pageNum, objIdForCaption);

• Step 6: fetchPageImage REQUEST_ID 강화 (mock 매칭 무관, 디버그 일관성)

In fetchPageImage(...), replace:

.addFormDataPart("REQUEST_ID", "odl-pdf2img-" + pageIndex)

with:

.addFormDataPart("REQUEST_ID",
                "odl-" + sourcePdfShaShort + "-pdf2img-p" + pageIndex)

• Step 7: 테스트 hook 메서드 추가 (package-private)

At the very bottom of the class (before the closing }), add:

// --- Test hooks (package-private) ---

    void invokeCallModule(byte[] pdfBytes, String moduleName) throws IOException {
        this.sourcePdfShaShort = sha256ShortHex(pdfBytes);
        callModule(pdfBytes, moduleName);
    }

    void invokeCallImageCaptioning(byte[] pngBytes, int pageNum, int objectId) throws IOException {
        callImageCaptioning(pngBytes, pageNum, objectId);
    }

• Step 8: 컴파일 + 새 테스트 통과 확인

Run:

cd opendataloader-project/opendataloader-pdf/java
mvn -pl opendataloader-pdf-core test -Dtest=HancomAIClientRequestIdTest

Expected: 2 passed.

• Step 9: 기존 테스트도 회귀 없는지 확인

Run:

mvn -pl opendataloader-pdf-core test

Expected: BUILD SUCCESS, all tests pass.

• Step 10: Commit

cd opendataloader-project/opendataloader-pdf
git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/HancomAIClient.java
git commit -m "feat(hybrid): HancomAIClient REQUEST_ID convention with sha-short, page, obj

For mock-server fixture lookup. Real Hancom AI server treats REQUEST_ID
as opaque client-defined string; no behavior change there.

Spec: bundolee/kb-odl/.../2026-04-18_Q2-DEV-02-Code_hancom-ai-mock-server-design.md"

Task 13: 통합 smoke — pdfua → mock 전체 호출 체인 Copy Section

Files: (코드 변경 없음, 검증만)

• Step 1: pdf-core 로컬 install

Run:

cd opendataloader-project/opendataloader-pdf/java
mvn -pl opendataloader-pdf-core install -DskipTests

Expected: BUILD SUCCESS, artifact installed to ~/.m2.

• Step 2: pdfua 빌드

Run:

cd opendataloader-project/opendataloader-pdfua
mvn package -DskipTests

Expected: BUILD SUCCESS, jar at target/.

• Step 3: mock 서버 백그라운드 기동

Run:

cd opendataloader-project/opendataloader-pdfua/scripts/mock_server
python -m mock_server \
  --pdf-dir /Users/benedict/Workspace/opendataloader-project/opendataloader-bench/pdfs \
  --fixture-dir /Users/benedict/Workspace/bundolee/kb-odl/raw/4-기술/2026-04-16_Q2-기술-ctx_hancom-ai-a11y_출력데이터-스키마 \
  --port 18008 > /tmp/mock-server.log 2>&1 &
sleep 2
curl -sf http://127.0.0.1:18008/ping

Expected: {"status":"ok"}, log contains indexed 200 PDFs.

• Step 4: pdfua 실행

Run:

mkdir -p /tmp/mock-out
cd opendataloader-project/opendataloader-pdfua
java -jar target/*.jar \
  --input /Users/benedict/Workspace/opendataloader-project/opendataloader-bench/pdfs/01030000000001.pdf \
  --output /tmp/mock-out \
  --hybrid hancom-ai \
  --hybrid-url http://localhost:18008

Expected: exit 0, output PDF in /tmp/mock-out/, server log shows DLA_OCR + pdf2img + (TSR or IMAGE_CAPTIONING based on document) all SUCCESS.

• Step 5: FIXTURE_MISS 의도적 트리거 — 외부 PDF

Run:

echo "%PDF-1.4 fake" > /tmp/notindexed.pdf
java -jar opendataloader-project/opendataloader-pdfua/target/*.jar \
  --input /tmp/notindexed.pdf \
  --output /tmp/mock-out2 \
  --hybrid hancom-ai \
  --hybrid-url http://localhost:18008

Expected: server log contains FIXTURE_MISS pdf warning with sha prefix; pdfua may either fail loud or fall back gracefully — record actual behavior in commit message.

• Step 6: 서버 종료 + 로그 보존

Run:

kill %1 || pkill -f "python -m mock_server"
cp /tmp/mock-server.log opendataloader-project/opendataloader-pdfua/scripts/mock_server/e2e-smoke.log

• Step 7: 결과 검토 + commit

Run:

ls -la /tmp/mock-out/
head -50 opendataloader-project/opendataloader-pdfua/scripts/mock_server/e2e-smoke.log

cd opendataloader-project/opendataloader-pdfua
git add scripts/mock_server/e2e-smoke.log
git commit -m "chore(mock-server): record e2e smoke log against benchmark PDF #01030000000001"

Task 14: 회의록에 완료 보고 추가 Copy Section

Files:

• Modify: bundolee/kb-odl/raw/운영/회의록/2026-04-16_hancom-ai-a11y_2차회의-매핑표-리뷰.md (다음 액션 표 업데이트) 또는 별도 진행 보고

• Step 1: 회의록 다음 액션 표 갱신

In the meeting notes' "다음 액션" table (around line 63), find rows referencing the blocked development and append a status note. If unsure where, add a new row below the existing table:

| 8 | hancom-ai mock 서버 + client REQUEST_ID 패치 완료 | 기술 | 2026-04-18 — transformer 재설계 막힘 해소 |

• Step 2: Commit (in kb-odl)

cd /Users/benedict/Workspace/bundolee/kb-odl
git add "raw/운영/회의록/2026-04-16_hancom-ai-a11y_2차회의-매핑표-리뷰.md"
git commit -m "회의록 갱신: hancom-ai mock 서버 구축 완료, transformer 재설계 unblock"

Self-Review Checklist (작성자 자체 검증) Copy Chapter

Spec coverage:

• ✅ 6개 모듈 (DLA_OCR/DLA/OCR/TSR/IMAGE_CAPTIONING/CHART) — Task 4 lookup, Task 6/7 server
• ✅ pdf2img 동적 변환 — Task 5 renderer, Task 8 endpoint
• ✅ /ping 엔드포인트 — Task 6
• ✅ /api/v1 alias — Task 6
• ✅ DATA_FORMAT 분기 — Task 6/7
• ✅ TSR_regionlist 폴백 — Task 4
• ✅ Unknown PDF FIXTURE_MISS — Task 6
• ✅ Client 패치 (REQUEST_ID 규약 6개 모듈 단축명, sha_short, page, obj) — Task 11/12
• ✅ E2E smoke — Task 13

Placeholder scan: TBD/TODO/"구현 나중에" 없음. 모든 코드 step에 완전한 코드 블록 첨부.

Type consistency: RequestIdParts(sha_short, page, obj, module) 일관, MODULE_TO_DIR(server-side mapping) vs MODULE_SHORT(client-side mapping) 분리 명확. lookup_image_module(..., module_short, basename, page, obj) 시그니처 일관.

Java 컴파일 위험요소:

• HexFormat은 Java 17+. 만약 빌드 타깃이 Java 8/11이면 테스트에서 직접 hex 변환 헬퍼로 대체 필요. Task 11 Step 1 컴파일 실패 시 즉시 발견.
• mockwebserver 의존성 추가 필요 (Task 11 Step 2에서 검증).

Execution Handoff Copy Chapter

Plan complete and saved to opendataloader-project/opendataloader-pdf/docs/superpowers/plans/2026-04-18-hancom-ai-mock-server.md.

Two execution options:

1. Subagent-Driven (recommended) — fresh subagent per task with two-stage review between tasks. Best for this plan since it spans two repos (Python + Java) and has a clear TDD rhythm.

2. Inline Execution — execute tasks in this session with checkpoints. Faster turnaround but heavier on the main context.

Which approach?

File: docs/superpowers/plans/2026-04-29-hybrid-hancom-ai-options.md Copy Chapter

Hybrid hancom-ai Options & CLI Refactoring Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Expose 5 hancom-ai-specific knobs as --hybrid-hancom-ai-* CLI options, then refactor CLIOptions so opendataloader-pdfua reuses the entire core option set instead of redefining its own.

Architecture: Three independent commits. (1) Add 5 options + gate validation in core. (2) Extract addAllTo / applyAllTo public API on CLIOptions. (3) Refactor pdfua's Main.java and RemediationConfig to consume that API; collapse 9 RemediationConfig constructors into a Builder; embed Config directly. Hard-break removed getters since all call sites are internal/test.

Tech Stack: Java 11, Apache Commons CLI, JUnit 5, Maven multi-module (opendataloader-pdf-core, opendataloader-pdf-cli, opendataloader-pdfua).

File Structure Copy Chapter

opendataloader-pdf (core)

opendataloader-pdfua

Phase 1 — Add 5 hybrid-hancom-ai-* options to core Copy Chapter

Task 1.1: Add option name + description constants Copy Section

Files:

• Modify: opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java (insert after line 137, the existing HYBRID_FALLBACK_DESC)

• Step 1: Add the 5 long-option constants and descriptions

Insert after the HYBRID_FALLBACK_DESC block (currently around line 137) and before the // ===== Stdout Output ===== section header:

// ===== Hybrid hancom-ai backend-specific =====
    private static final String HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION =
            "hybrid-hancom-ai-regionlist-strategy";
    private static final String HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_DESC =
            "DLA label 7 (regionlist) handling. Requires --hybrid=hancom-ai. "
            + "Values: table-first (default; check TSR overlap), list-only (skip TSR, always treat as list)";

    private static final String HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION =
            "hybrid-hancom-ai-ocr-strategy";
    private static final String HYBRID_HANCOM_AI_OCR_STRATEGY_DESC =
            "OCR strategy. Requires --hybrid=hancom-ai. "
            + "Values: off (stream-only), auto (default; stream first, OCR fallback), force (OCR-only)";

    private static final String HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION =
            "hybrid-hancom-ai-image-cache";
    private static final String HYBRID_HANCOM_AI_IMAGE_CACHE_DESC =
            "Page image cache backing. Requires --hybrid=hancom-ai. "
            + "Values: memory (default), disk";

    private static final String HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION =
            "hybrid-hancom-ai-save-crops";
    private static final String HYBRID_HANCOM_AI_SAVE_CROPS_DESC =
            "Persist cropped figure images to disk for debugging. Requires --hybrid=hancom-ai";

    private static final String HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION =
            "hybrid-hancom-ai-crop-output-dir";
    private static final String HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_DESC =
            "Output directory for --hybrid-hancom-ai-save-crops. Requires --hybrid=hancom-ai";

• Step 2: Verify file still compiles

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli compile -DskipTests -q
Expected: BUILD SUCCESS (constants added but unused — no warning since private static final is fine).

• Step 3: Commit

cd opendataloader-pdf && git add java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java
git commit -m "$(cat <<'EOF'
feat(cli): add --hybrid-hancom-ai-* option name constants

Defines long-name and description constants for 5 hancom-ai-specific
flags. Wiring into OPTION_DEFINITIONS and parser comes in subsequent
commits.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 1.2: Register options in OPTION_DEFINITIONS Copy Section

Files:

• Modify: opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java (the OPTION_DEFINITIONS list, currently lines 166-208)

• Step 1: Add 3 exported entries to OPTION_DEFINITIONS (after HYBRID_FALLBACK_LONG_OPTION line, around line 196)

Insert these three lines immediately after the new OptionDefinition(HYBRID_FALLBACK_LONG_OPTION, ...) line and before the new OptionDefinition(TO_STDOUT_LONG_OPTION, ...) line:

new OptionDefinition(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION, null, "string",
                    "table-first", HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_DESC, true),
            new OptionDefinition(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION, null, "string",
                    "auto", HYBRID_HANCOM_AI_OCR_STRATEGY_DESC, true),
            new OptionDefinition(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION, null, "string",
                    "memory", HYBRID_HANCOM_AI_IMAGE_CACHE_DESC, true),

• Step 2: Add 2 hidden entries (exported=false) after the legacy options block (around line 208)

Insert after the NO_JSON_REPORT_LONG_OPTION line (the last legacy entry). Note the trailing )/; of the list — adjust so the new lines come before the closing ):

new OptionDefinition(HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION, null, "boolean",
                    false, HYBRID_HANCOM_AI_SAVE_CROPS_DESC, false),
            new OptionDefinition(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION, null, "string",
                    null, HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_DESC, false),

• Step 3: Add a "options registered" sanity test

Modify opendataloader-pdf/java/opendataloader-pdf-cli/src/test/java/org/opendataloader/pdf/cli/CLIOptionsTest.java — append the following test methods at the end of the class (before the closing }):

@Test
    void testDefineOptions_containsHybridHancomAiRegionlistStrategy() {
        assertTrue(options.hasOption("hybrid-hancom-ai-regionlist-strategy"));
    }

    @Test
    void testDefineOptions_containsHybridHancomAiOcrStrategy() {
        assertTrue(options.hasOption("hybrid-hancom-ai-ocr-strategy"));
    }

    @Test
    void testDefineOptions_containsHybridHancomAiImageCache() {
        assertTrue(options.hasOption("hybrid-hancom-ai-image-cache"));
    }

    @Test
    void testDefineOptions_containsHybridHancomAiSaveCrops() {
        assertTrue(options.hasOption("hybrid-hancom-ai-save-crops"));
    }

    @Test
    void testDefineOptions_containsHybridHancomAiCropOutputDir() {
        assertTrue(options.hasOption("hybrid-hancom-ai-crop-output-dir"));
    }

• Step 4: Run the sanity tests

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest -q
Expected: All 5 new tests PASS. (No parsing yet — these only verify Apache Commons CLI registration.)

• Step 5: Commit

cd opendataloader-pdf && git add java/opendataloader-pdf-cli
git commit -m "$(cat <<'EOF'
feat(cli): register --hybrid-hancom-ai-* in OPTION_DEFINITIONS

Three options (regionlist-strategy, ocr-strategy, image-cache) are
exported to options.json and wrapper bindings; two debug options
(save-crops, crop-output-dir) stay hidden via exported=false.

Sanity tests verify Apache Commons CLI registration. Parsing logic
and Config wiring come next.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 1.3: Parse new options into HybridConfig Copy Section

Files:

• Modify: opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java — applyHybridOptions() method (around lines 463-557)

• Step 1: Write failing test for --hybrid-hancom-ai-regionlist-strategy=list-only

Append to CLIOptionsTest.java:

@Test
    void testCreateConfig_withHybridHancomAiRegionlistStrategy() throws ParseException {
        String[] args = {"--hybrid", "hancom-ai",
                         "--hybrid-hancom-ai-regionlist-strategy", "list-only",
                         testPdf.getAbsolutePath()};
        CommandLine cmd = parser.parse(options, args);
        Config config = CLIOptions.createConfigFromCommandLine(cmd);
        assertEquals("list-only", config.getHybridConfig().getRegionlistStrategy());
    }

• Step 2: Run, expect FAIL

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest#testCreateConfig_withHybridHancomAiRegionlistStrategy -q
Expected: FAIL — value still default table-first because parsing hasn't been added yet.

• Step 3: Add parsing for all 5 new options

Insert this block inside applyHybridOptions(), immediately after the existing if (commandLine.hasOption(HYBRID_FALLBACK_LONG_OPTION)) { config.getHybridConfig().setFallbackToJava(true); } line and before if (commandLine.hasOption(TO_STDOUT_LONG_OPTION)) { ... }:

if (commandLine.hasOption(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION)) {
            String value = commandLine.getOptionValue(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION);
            if (value != null && !value.trim().isEmpty()) {
                config.getHybridConfig().setRegionlistStrategy(value.trim().toLowerCase(Locale.ROOT));
            }
        }
        if (commandLine.hasOption(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION)) {
            String value = commandLine.getOptionValue(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION);
            if (value != null && !value.trim().isEmpty()) {
                config.getHybridConfig().setOcrStrategy(value.trim().toLowerCase(Locale.ROOT));
            }
        }
        if (commandLine.hasOption(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION)) {
            String value = commandLine.getOptionValue(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION);
            if (value != null && !value.trim().isEmpty()) {
                config.getHybridConfig().setImageCache(value.trim().toLowerCase(Locale.ROOT));
            }
        }
        if (commandLine.hasOption(HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION)) {
            config.getHybridConfig().setSaveCrops(true);
        }
        if (commandLine.hasOption(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION)) {
            String value = commandLine.getOptionValue(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION);
            if (value != null && !value.trim().isEmpty()) {
                config.getHybridConfig().setCropOutputDir(value.trim());
            }
        }

• Step 4: Run the test, expect PASS

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest#testCreateConfig_withHybridHancomAiRegionlistStrategy -q
Expected: PASS.

• Step 5: Add coverage tests for the other four options

Append to CLIOptionsTest.java:

/* Detailed source-code truncated for AI context efficiency. */

• Step 6: Run all new tests, expect PASS

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest -q
Expected: All tests PASS.

• Step 7: Commit

cd opendataloader-pdf && git add java/opendataloader-pdf-cli
git commit -m "$(cat <<'EOF'
feat(cli): parse --hybrid-hancom-ai-* into HybridConfig

Wires --hybrid-hancom-ai-{regionlist-strategy,ocr-strategy,image-cache,
save-crops,crop-output-dir} through applyHybridOptions(). HybridConfig
setters validate enum values and throw IllegalArgumentException on
invalid input.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 1.4: Add hancom-ai gate validation Copy Section

Files:

• Modify: opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java — applyHybridOptions()

• Step 1: Write failing test — using the option without --hybrid=hancom-ai should throw

Append to CLIOptionsTest.java:

@Test
    void testCreateConfig_hybridHancomAiOption_withoutHancomAi_throws() {
        String[] args = {"--hybrid-hancom-ai-regionlist-strategy", "list-only",
                         testPdf.getAbsolutePath()};
        // No --hybrid set, defaults to off.
        IllegalArgumentException ex = assertThrows(IllegalArgumentException.class, () -> {
            CommandLine cmd = parser.parse(options, args);
            CLIOptions.createConfigFromCommandLine(cmd);
        });
        assertTrue(ex.getMessage().contains("--hybrid-hancom-ai-"),
                "Error should mention the offending prefix, got: " + ex.getMessage());
        assertTrue(ex.getMessage().contains("hancom-ai"),
                "Error should mention required backend, got: " + ex.getMessage());
    }

    @Test
    void testCreateConfig_hybridHancomAiOption_withDoclingFast_throws() {
        String[] args = {"--hybrid", "docling-fast",
                         "--hybrid-hancom-ai-ocr-strategy", "force",
                         testPdf.getAbsolutePath()};
        assertThrows(IllegalArgumentException.class, () -> {
            CommandLine cmd = parser.parse(options, args);
            CLIOptions.createConfigFromCommandLine(cmd);
        });
    }

• Step 2: Run, expect FAIL

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest#testCreateConfig_hybridHancomAiOption_withoutHancomAi_throws -q
Expected: FAIL — currently no validation, so the call succeeds silently.

• Step 3: Add the gate at the end of applyHybridOptions()

Insert this block at the very end of applyHybridOptions() (after the --to-stdout handling, before the closing }):

boolean usesHancomAiOnly =
                commandLine.hasOption(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION) ||
                commandLine.hasOption(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION) ||
                commandLine.hasOption(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION) ||
                commandLine.hasOption(HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION) ||
                commandLine.hasOption(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION);
        if (usesHancomAiOnly && !Config.HYBRID_HANCOM_AI.equals(config.getHybrid())) {
            throw new IllegalArgumentException(
                    "Options --hybrid-hancom-ai-* require --hybrid=hancom-ai (got --hybrid="
                    + config.getHybrid() + ")");
        }

• Step 4: Run both gate tests, expect PASS

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest -q
Expected: All tests PASS, including both new gate tests AND all earlier ones (which use --hybrid hancom-ai).

• Step 5: Commit

cd opendataloader-pdf && git add java/opendataloader-pdf-cli
git commit -m "$(cat <<'EOF'
feat(cli): reject --hybrid-hancom-ai-* without --hybrid=hancom-ai

Without this guard, these flags would silently no-op when the user
selects a different backend. Fail fast with a message naming the
offending prefix and required backend.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 1.5: Regenerate options.json and wrapper bindings Copy Section

Files:

• Modify: opendataloader-pdf/options.json

• Modify: opendataloader-pdf/python/..., opendataloader-pdf/node/... (auto-generated)

• Step 1: Run npm sync

Run: cd opendataloader-pdf && npm run sync
Expected: Command completes; git status shows changes to options.json and python/node binding files. The 3 exported new options (regionlist-strategy, ocr-strategy, image-cache) appear; the 2 hidden ones (save-crops, crop-output-dir) do not.

• Step 2: Spot-check options.json

Verify the diff contains the 3 new options with correct defaults:

cd opendataloader-pdf && grep -A 2 "hybrid-hancom-ai" options.json

Expected: 3 entries for hybrid-hancom-ai-regionlist-strategy, hybrid-hancom-ai-ocr-strategy, hybrid-hancom-ai-image-cache with their defaults table-first / auto / memory. No entry for save-crops or crop-output-dir.

• Step 3: Commit

cd opendataloader-pdf && git add options.json python node
git commit -m "$(cat <<'EOF'
chore: regenerate bindings for --hybrid-hancom-ai-* options

Auto-generated by npm run sync. Adds the 3 exported hancom-ai
options to options.json, Python bindings, and Node bindings.
The 2 hidden debug options (save-crops, crop-output-dir) stay
out of the public surface.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Phase 2 — Extract reusable CLIOptions API Copy Chapter

This phase is a pure refactor — behavior unchanged.

Task 2.1: Add public `addAllTo(Options)` method Copy Section

Files:

• Modify: opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java — defineOptions() (around line 210)

• Step 1: Write a test that exercises addAllTo against a fresh Options instance

Append to CLIOptionsTest.java:

@Test
    void testAddAllTo_registersAllOptions() {
        Options ext = new Options();
        CLIOptions.addAllTo(ext);
        assertTrue(ext.hasOption("hybrid"));
        assertTrue(ext.hasOption("hybrid-mode"));
        assertTrue(ext.hasOption("hybrid-hancom-ai-regionlist-strategy"));
        assertTrue(ext.hasOption("format"));
        assertTrue(ext.hasOption("threads"));
    }

• Step 2: Run, expect FAIL (compile error — method does not exist)

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest#testAddAllTo_registersAllOptions -q
Expected: COMPILATION FAILURE — cannot find symbol: addAllTo.

• Step 3: Add addAllTo and refactor defineOptions to use it

Replace the existing defineOptions method (currently lines ~210-217) with:

public static Options defineOptions() {
        Options options = new Options();
        addAllTo(options);
        return options;
    }

    /**
     * Registers every core CLI option onto an external {@link Options} instance.
     * Used by downstream CLIs (e.g. opendataloader-pdfua) that want to inherit
     * the entire core option set and add their own options on top.
     *
     * @param options the Options instance to populate
     */
    public static void addAllTo(Options options) {
        for (OptionDefinition def : OPTION_DEFINITIONS) {
            options.addOption(def.toOption());
        }
    }

• Step 4: Run, expect PASS

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest -q
Expected: All tests PASS — addAllTo works and all existing tests still work via defineOptions.

• Step 5: Commit

cd opendataloader-pdf && git add java/opendataloader-pdf-cli
git commit -m "$(cat <<'EOF'
refactor(cli): extract CLIOptions.addAllTo(Options) public API

Splits Options registration out of defineOptions() so downstream CLIs
can register every core option onto their own Options instance and
add their own options on top. defineOptions() now delegates.

Pure refactor — no behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 2.2: Add public `applyAllTo(Config, CommandLine)` method Copy Section

Files:

• Modify: opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java — createConfigFromCommandLine() (around lines 219-287)

• Step 1: Write a test that exercises applyAllTo directly

Append to CLIOptionsTest.java:

@Test
    void testApplyAllTo_appliesCoreOptions() throws ParseException {
        Options ext = new Options();
        CLIOptions.addAllTo(ext);
        ext.addOption(null, "downstream-only", true, "Downstream-specific");

        String[] args = {"--hybrid", "hancom-ai",
                         "--threads", "2",
                         "--downstream-only", "value",
                         testPdf.getAbsolutePath()};
        CommandLine cmd = parser.parse(ext, args);

        Config config = new Config();
        config.setOutputFolder(tempDir.toString());
        CLIOptions.applyAllTo(config, cmd);

        assertEquals("hancom-ai", config.getHybrid());
        assertEquals(2, config.getThreads());
        // downstream-only is not consumed by applyAllTo — that is the caller's job.
        assertEquals("value", cmd.getOptionValue("downstream-only"));
    }

• Step 2: Run, expect FAIL (compile error)

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest#testApplyAllTo_appliesCoreOptions -q
Expected: COMPILATION FAILURE — cannot find symbol: applyAllTo.

• Step 3: Refactor createConfigFromCommandLine to delegate to applyAllTo

Replace the existing createConfigFromCommandLine (around lines 219-287) with this two-method form:

/* Detailed source-code truncated for AI context efficiency. */

The output-folder positional handling stays in createConfigFromCommandLine only — pdfua does not have positional input args, so it should not pay for that branch.

• Step 4: Run all CLI tests, expect PASS

Run: cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -q
Expected: All tests PASS — both the new applyAllTo test and every existing test (the latter still go through createConfigFromCommandLine, which now delegates).

• Step 5: Commit

cd opendataloader-pdf && git add java/opendataloader-pdf-cli
git commit -m "$(cat <<'EOF'
refactor(cli): extract CLIOptions.applyAllTo(Config, CommandLine)

Splits per-option Config wiring out of createConfigFromCommandLine()
so downstream CLIs (opendataloader-pdfua) can apply core options to
their own Config without inheriting the positional-arg output-folder
fallback. createConfigFromCommandLine() delegates.

Pure refactor — no behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 2.3: Install core to local Maven for pdfua Copy Section

Per opendataloader-pdfua/CLAUDE.md, pdfua consumes core via Maven; cross-repo changes require mvn install on core first.

• Step 1: Install core to local Maven

Run: cd opendataloader-pdf/java && mvn install -DskipTests -q
Expected: BUILD SUCCESS. The new core artifact (with addAllTo / applyAllTo) is now available to pdfua.

Phase 3 — Refactor pdfua to consume core CLIOptions Copy Chapter

Task 3.1: Add Builder to RemediationConfig Copy Section

Files:

• Modify: opendataloader-pdfua/src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java

This task adds a Builder without removing the existing constructors yet — this lets us migrate call sites incrementally while keeping the test suite green.

• Step 1: Add coreConfig field, the Builder class, and a builder-based constructor

After the existing private final MockMode mockMode; line (around line 52), add:

private final org.opendataloader.pdf.api.Config coreConfig;

After the last existing constructor (around line 125, ending this.mockMode = ...;), add:

/* Detailed source-code truncated for AI context efficiency. */

Also: in each existing constructor (the 5 of them, ending at line 125), add this line after the existing this.mockMode = ...; assignment:

this.coreConfig = buildCoreConfigForLegacyConstructor();

And add this private helper after the constructors (before public String getInput()):

private org.opendataloader.pdf.api.Config buildCoreConfigForLegacyConstructor() {
        org.opendataloader.pdf.api.Config c = new org.opendataloader.pdf.api.Config();
        if (this.hybrid != null) {
            c.setHybrid(this.hybrid);
        }
        if (this.hybridUrl != null) {
            c.getHybridConfig().setUrl(this.hybridUrl);
        }
        if (this.hybridMode != null) {
            c.getHybridConfig().setMode(this.hybridMode);
        }
        return c;
    }

This keeps the existing 5 constructors source-compatible during migration. They will be removed in Task 3.5.

• Step 2: Verify pdfua still compiles and tests still pass

Run: cd opendataloader-pdfua && mvn test -q
Expected: BUILD SUCCESS, all tests pass. (We added members; we did not change existing behavior.)

• Step 3: Commit

cd opendataloader-pdfua && git add src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java
git commit -m "$(cat <<'EOF'
feat(remediation): add RemediationConfig.Builder + embedded coreConfig

Adds a Builder API and a coreConfig field carrying the full core
opendataloader-pdf Config. Existing 5 constructors keep working —
they construct a synthetic coreConfig from the flat hybrid fields
to maintain semantics. Migration of call sites and removal of the
legacy constructors follows.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 3.2: Refactor pdfua/Main.java to use core CLIOptions Copy Section

Files:

• Modify: opendataloader-pdfua/src/main/java/org/opendataloader/pdf/Main.java

• Step 1: Replace buildOptions() body — remove hybrid options, add addAllTo

Replace buildOptions() (lines 94-125) with:

/* Detailed source-code truncated for AI context efficiency. */

Note: --threads is removed here because it now comes from core CLIOptions. The pdfua-specific note about HancomAIClient thread-safety becomes a runtime warning instead — that is handled in BatchProcessor and is out of scope for this plan. (If the warning text was important, file a separate issue.)

• Step 2: Replace runWithOptions body

Replace runWithOptions (lines 52-92) with:

/* Detailed source-code truncated for AI context efficiency. */

• Step 3: Run pdfua tests

Run: cd opendataloader-pdfua && mvn test -q
Expected: PASS. MainAuditBundleCliTest exercises CLI parsing and should now route through core CLIOptions transparently.

• Step 4: Manual smoke test of inherited core options

Run: cd opendataloader-pdfua && mvn -q exec:java -Dexec.mainClass=org.opendataloader.pdf.Main -Dexec.args="--input src/test/resources --output /tmp/pdfua-test --hybrid-hancom-ai-regionlist-strategy list-only --hybrid-hancom-ai-ocr-strategy auto --audit-bundle none --mock off" 2>&1 | head -20

If the project doesn't have exec:java configured, skip this step — the test suite covers wiring.

Expected: Either runs without "Unrecognized option" errors for --hybrid-hancom-ai-*, or fails on a downstream concern (network/file). The point is that argument parsing accepts the flags.

• Step 5: Commit

cd opendataloader-pdfua && git add src/main/java/org/opendataloader/pdf/Main.java
git commit -m "$(cat <<'EOF'
refactor(pdfua): inherit core CLIOptions instead of redefining

Replace pdfua's hand-rolled --hybrid/--hybrid-url/--hybrid-mode/
--threads definitions with org.opendataloader.pdf.cli.CLIOptions.
addAllTo(). pdfua now automatically inherits every core option,
including the new --hybrid-hancom-ai-* flags.

pdfua's deviating defaults (hybrid=hancom-ai, url=localhost:18008,
mode=full) are now in an explicit applyPdfuaDefaults() helper
instead of buried in cmd.getOptionValue() defaults.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 3.3: Simplify RemediationProcessor's hybrid forwarding Copy Section

Files:

• Modify: opendataloader-pdfua/src/main/java/org/opendataloader/pdf/remediation/RemediationProcessor.java (lines 163-170)

• Step 1: Read the current hybrid block to confirm context

Run: cd opendataloader-pdfua && sed -n '160,175p' src/main/java/org/opendataloader/pdf/remediation/RemediationProcessor.java
Expected output (approximately):

Config config = ...;  // may need to inspect actual surrounding code
        ...
        if (remediationConfig.getHybridUrl() != null || !"docling-fast".equals(remediationConfig.getHybrid())) {
            config.setHybrid(remediationConfig.getHybrid());
            if (remediationConfig.getHybridUrl() != null) {
                config.getHybridConfig().setUrl(remediationConfig.getHybridUrl());
            }
            config.getHybridConfig().setMode(remediationConfig.getHybridMode());
        }

• Step 2: Replace the hybrid block with coreConfig copy

Replace those lines (currently around 163-170) with:

// RemediationConfig now carries the full core Config. Adopt it directly.
        config = remediationConfig.getCoreConfig();

If the local Config config variable is constructed earlier in the method with other settings, you must merge them — read the full method first to verify. The simplest correct rewrite is:

1. Find where the local Config is first assigned in processSingle or whichever method contains lines 163-170.
2. Replace Config config = new Config(); with Config config = remediationConfig.getCoreConfig();
3. Delete the lines 163-170 hybrid block.
4. Keep any other config.setXxx() calls in the method — they continue to work on the borrowed Config instance.

If coreConfig is mutated by other settings inside processSingle and that mutation should not leak back to the caller, clone it first. But based on the spec's intent ("RemediationConfig embeds Config directly"), in-place mutation is acceptable since RemediationConfig is per-document and short-lived.

• Step 3: Run pdfua tests

Run: cd opendataloader-pdfua && mvn test -q
Expected: PASS.

• Step 4: Commit

cd opendataloader-pdfua && git add src/main/java/org/opendataloader/pdf/remediation/RemediationProcessor.java
git commit -m "$(cat <<'EOF'
refactor(remediation): forward coreConfig directly to RemediationProcessor

Replace the per-field hybrid forwarding block (setHybrid + setUrl +
setMode) with config = remediationConfig.getCoreConfig(). Now that
RemediationConfig embeds the full core Config, no manual copy is
needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 3.4: Migrate test call sites to Builder Copy Section

Files:

• Modify: opendataloader-pdfua/src/test/java/org/opendataloader/pdf/audit/AuditBundleEmitterTest.java

• Modify: opendataloader-pdfua/src/test/java/org/opendataloader/pdf/audit/certificate/CertificateIssuerTest.java

• Modify: opendataloader-pdfua/src/test/java/org/opendataloader/pdf/audit/manifest/AuditManifestBuilderTest.java

• Modify: opendataloader-pdfua/src/test/java/org/opendataloader/pdf/remediation/RemediationConfigAuditBundleTest.java

• Step 1: Find every new RemediationConfig( call site

Run:

cd opendataloader-pdfua && grep -rn "new RemediationConfig(" src/test

Record each file:line and inspect the constructor arguments.

• Step 2: For each call site, replace with Builder

For a call like:

RemediationConfig cfg = new RemediationConfig(
    "input.pdf", "/tmp/out", "en", "hancom-ai", "http://localhost:18008", "full",
    true, 1, EnumSet.of(ConformanceLevel.UA1), ReportLevel.NONE);

Rewrite as:

org.opendataloader.pdf.api.Config core = new org.opendataloader.pdf.api.Config();
core.setHybrid("hancom-ai");
core.getHybridConfig().setUrl("http://localhost:18008");
core.getHybridConfig().setMode("full");

RemediationConfig cfg = RemediationConfig.builder()
    .input("input.pdf")
    .output("/tmp/out")
    .lang("en")
    .coreConfig(core)
    .enrichPictureDescription(true)
    .threads(1)
    .conformances(EnumSet.of(ConformanceLevel.UA1))
    .reportLevel(ReportLevel.NONE)
    .build();

If the test only cares about a subset (e.g. doesn't pass hybrid args), use the relevant builder methods only — Builder defaults handle the rest.

• Step 3: Run pdfua tests

Run: cd opendataloader-pdfua && mvn test -q
Expected: PASS.

• Step 4: Commit

cd opendataloader-pdfua && git add src/test
git commit -m "$(cat <<'EOF'
test(remediation): migrate RemediationConfig instantiations to Builder

Switches all 4 test files (AuditBundleEmitterTest, CertificateIssuerTest,
AuditManifestBuilderTest, RemediationConfigAuditBundleTest) from
positional constructors to RemediationConfig.builder() + .build().

Prepares for removal of the legacy constructors in the next commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 3.5: Remove legacy constructors and `getHybridUrl`/`getHybridMode` Copy Section

Files:

• Modify: opendataloader-pdfua/src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java

• Step 1: Verify no production code references the legacy constructors

Run:

cd opendataloader-pdfua && grep -rn "new RemediationConfig(" src/main

Expected: No output. (Main.java should now use builder().)

If any matches appear, migrate them to Builder before continuing.

• Step 2: Verify no code references getHybridUrl / getHybridMode

Run:

cd opendataloader-pdfua && grep -rn "getHybridUrl\|getHybridMode" src

Expected: Only matches in RemediationConfig.java itself (the methods being removed).

If any other call sites appear, replace with getCoreConfig().getHybridConfig().getUrl() / .getMode().

• Step 3: Remove the 5 legacy constructors and the buildCoreConfigForLegacyConstructor helper

In RemediationConfig.java, delete:

• All 5 public constructors (currently lines 54-125)
• The buildCoreConfigForLegacyConstructor() helper added in Task 3.1

Keep the private RemediationConfig(Builder b) constructor.

• Step 4: Remove getHybridUrl() and getHybridMode() getters; redirect getHybrid() to delegate

Replace the three lines (currently around 130-132):

public String getHybrid()                       { return hybrid; }
    public String getHybridUrl()                    { return hybridUrl; }
    public String getHybridMode()                   { return hybridMode; }

with:

public String getHybrid()                       { return coreConfig.getHybrid(); }
    public org.opendataloader.pdf.hybrid.HybridConfig getHybridConfig() {
        return coreConfig.getHybridConfig();
    }

Also remove the now-unused fields hybrid, hybridUrl, hybridMode (lines ~42-44) and their assignments from the Builder constructor body. Delegate via coreConfig everywhere.

• Step 5: Verify compile and tests

Run: cd opendataloader-pdfua && mvn compile -q && mvn test -q
Expected: BUILD SUCCESS and all tests PASS.

• Step 6: Commit

cd opendataloader-pdfua && git add src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java
git commit -m "$(cat <<'EOF'
refactor(remediation)!: remove legacy RemediationConfig constructors

BREAKING: Removes the 5 positional constructors and the flat
hybrid/hybridUrl/hybridMode fields. RemediationConfig now consists
of pdfua-specific fields plus an embedded core Config; hybrid state
is read via getCoreConfig().getHybrid() / getHybridConfig().

getHybridUrl() and getHybridMode() are removed — call sites use
getHybridConfig().getUrl() / .getMode() instead. Verified that no
production or test code outside RemediationConfig itself referenced
those getters.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
EOF
)"

Task 3.6: Final integration check Copy Section

• Step 1: Full build + test on both repos

Run:

cd opendataloader-pdf/java && mvn install -DskipTests -q
cd opendataloader-pdfua && mvn clean package -q

Expected: Both BUILD SUCCESS.

• Step 2: Run the full test suite for both

Run:

cd opendataloader-pdf/java && mvn test -q
cd opendataloader-pdfua && mvn test -q

Expected: All tests pass in both repos.

• Step 3: Verify --help output for both CLIs lists the new options

cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli exec:java \
  -Dexec.mainClass=org.opendataloader.pdf.cli.CLIMain -Dexec.args="--help" 2>&1 | \
  grep "hybrid-hancom-ai"

Expected: 3 lines (regionlist-strategy, ocr-strategy, image-cache).

cd opendataloader-pdfua && java -jar target/*-jar-with-dependencies.jar --input dummy --output dummy --help 2>&1 | grep "hybrid-hancom-ai"

Expected: 3 lines (same options inherited).

Note: pdfua's --help output requires --input / --output to be supplied since they're addRequiredOption. If this is awkward, this verification step can be done by inspecting Options programmatically in a quick MainTest.

• Step 4: No final commit needed — all phase-3 commits already pushed.

Self-Review Copy Chapter

Spec coverage check:

All spec items mapped to at least one task.

Placeholder check: No "TBD", "TODO", "fill in details", or stub steps. Every code step contains the actual code. Test code is complete (assertions, args). Commit messages are pre-written.

Type consistency: coreConfig field name used consistently. getCoreConfig() getter name used consistently. Builder method names match field names. applyPdfuaDefaults signature matches between definition (Task 3.2) and call site (also Task 3.2).

Plan complete and saved to `docs/superpowers/plans/2026-04-29-hybrid-hancom-ai-options.md`. Two execution options: Copy Chapter

1. Subagent-Driven (recommended) - dispatch a fresh subagent per task, review between tasks, fast iteration

2. Inline Execution - execute tasks in this session using executing-plans, batch execution with checkpoints

Which approach?

File: docs/hybrid/experiments/triage/triage-experiments.md Copy Chapter

name: triage-lab Copy Chapter

Triage Lab - Experiment Records

This skill manages experiment records and optimization history for triage logic.

Current Implementation Copy Chapter

File: java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java

Signal Priority (classifyPage method) Copy Section

1. hasTableBorder - TableBorder presence (confidence: 1.0)
2. hasVectorTableSignal - Grid lines, border lines, line art (confidence: 0.95)
3. hasTextTablePattern - Text patterns with consecutive validation (confidence: 0.9)
4. hasSuspiciousPattern - Y-overlap or large gap detection (confidence: 0.85)
5. lineToTextRatio > 0.3 - High line chunk ratio (confidence: 0.8)
6. alignedLineGroups >= 5 - Aligned baseline groups (confidence: 0.7)

Key Thresholds Copy Section

Experiment History Copy Chapter

Experiment 001 (2026-01-03): FP Cause Analysis Copy Section

Goal: Identify root causes of high False Positive rate

Baseline:

• Documents: 200 (42 with tables)
• TP: 41, TN: 48, FP: 110, FN: 1
• Precision: 27.15%, Recall: 97.62%, F1: 42.49%

FP by Signal:

Root Cause: Y-overlap check in hasSuspiciousPattern is too sensitive

• Condition previous.getTopY() < current.getBottomY() triggers on normal multi-column layouts

Experiments:

Recommendation:

• To maintain recall: Remove Y-overlap check (Precision +9%, Recall unchanged)
• To optimize F1: Use only reliable signals (F1 +22%, Recall -7%)

FN Documents:

• 01030000000110: Missed by all experiments (needs investigation)
• 01030000000122, 01030000000116, 01030000000117: Only detected by SuspiciousPattern

Applied: Y-overlap check removed (2026-01-03)

Experiment 002 (2026-01-03): Further FP Reduction Copy Section

Goal: Reduce remaining 72 FPs after Y-overlap removal

Current FP by Signal (after Experiment 001):

Experiment 2A: Gap Multiplier (hasSuspiciousPattern)

→ No effect (Y-overlap removal already optimized this signal)

Experiment 2B: AlignedLineGroups Threshold

→ Recommended: Threshold 5 (FP -5, Recall maintained)

Experiment 2C: Vector Signal Criteria

→ Minimal effect (FP -1, higher values reduce Recall)

Recommendation:

• Apply alignedLineGroups threshold 3 → 5
• Expected: FP 64 → 59 (-5), Recall 92.86% (maintained), F1 +1.92%

Applied: alignedLineGroups threshold 3 → 5 (2026-01-03)

Actual Results:

Next Steps:

• Investigate hasTableBorder FPs (14 cases, external library)
• Investigate unknown FPs (8 cases)

Experiment 003 (2026-01-03): VectorTableSignal & SuspiciousPattern Analysis Copy Section

Goal: Further reduce FP 67 while maintaining high Recall

Current FP by Signal (after Experiment 002):

VectorTableSignal Sub-signal Analysis (23 FPs):

→ hasAlignedShortLines is the primary cause of VectorSignal FPs

Experiments:

Analysis:

• hasTableBorder (14 FPs): External library, cannot modify
• hasVectorTableSignal (23 FPs): hasAlignedShortLines too aggressive
• hasSuspiciousPattern (19 FPs): Gap detection catches non-table layouts

Recommendation:

• Best for Recall: 003D (Disable SuspiciousPattern)
  • FP: 67 → 55 (-12), FN: 1 → 2 (+1), Recall: 95.24%
• Best for F1: 003I (Combined)
  • FP: 67 → 47 (-20), FN: 1 → 4 (+3), F1: 59.84%

FN Documents (003D):

• 01030000000122: Only detected by SuspiciousPattern (gap-based)
• 01030000000110: Never detected (needs separate investigation)

Applied: 003D - Disabled hasSuspiciousPattern (2026-01-03)

Actual Results:

Experiment 004 (2026-01-03): AlignedLineGroups Signal Analysis Copy Section

Goal: Further reduce FP 55 while maintaining Recall 95.24%

Current FP by Signal (after Experiment 003):

VectorTableSignal Sub-signal Analysis (23 FPs):

Experiments:

Analysis:

• alignedLineGroups signal caused 12 FPs but detected no additional true tables
• Disabling it removes all 12 FPs without any FN increase
• Best option for maintaining Recall while improving Precision

Applied: 004D - Disabled alignedLineGroups signal (2026-01-03)

Actual Results:

Next Steps:

• Investigate hasVectorTableSignal FPs (23 remaining) - hasAlignedShortLines main cause
• Investigate hasTableBorder FPs (14 cases, external library limitation)

Experiment 005 (2026-01-03): Large Image Signal for FN Reduction Copy Section

Goal: Reduce FN by detecting pages with large images (potential table/chart images)

Background:

• FN documents 01030000000110 and 01030000000122 contain images with tables
• 110: 28.64% page area (graph image)
• 122: 11.73% page area (table image)

Implementation:

• Added hasLargeImage signal to TriageProcessor
• Detects ImageChunk objects and calculates max image area / page area ratio

Experiments:

Analysis:

• 11% threshold achieves 100% Recall (all 42 table documents detected)
• Trade-off: FP increases from 43 → 83 (+40), Precision drops from 48% → 34%
• F1 decreases from 64% → 50% due to high FP increase
• Many FPs are documents with decorative images, diagrams, photos

Experiment 005B: Adding Aspect Ratio Condition

Observation: FN documents have wide images (ratio 1.79, 3.68), while FP documents often have square/tall images (ratio 0.6~1.5).

Final Configuration:

• Image area >= 11% of page area
• Image aspect ratio (width/height) >= 1.75

Trade-off:

• Achieves 100% Recall (all 42 table documents detected)
• FP increases from 43 → 55 (+12)
• F1 decreases from 64% → 60.43% (-3.57%)

Applied: 11% + aspect ratio 1.75 (2026-01-03)

Template for New Experiments Copy Chapter

### Experiment XXX (YYYY-MM-DD): [Title]

**Goal**: [What are you trying to improve?]

**Changes**: [What did you modify?]

**Results**:
| Config | Precision | Recall | F1 | FP | FN |
|--------|-----------|--------|-----|-----|-----|
| Before | | | | | |
| After | | | | | |

**Conclusion**: [What did you learn? Should this be applied?]

**Next Steps**: [What to try next?]

How to Run Experiments Copy Chapter

# Run triage accuracy test
./scripts/test-java.sh -Dtest=TriageProcessorIntegrationTest#testTriageAccuracyOnBenchmarkPDFs

# Debug specific document
./scripts/bench.sh --doc-id 01030000000110

Related Files Copy Chapter

• TriageProcessor.java
• TriageProcessorIntegrationTest.java

File: docs/hybrid/experiments/speed/speed-experiment-2026-01-03.md Copy Chapter

Docling Speed Experiment Results

Date: 2026-01-03 14:31:43

Summary Copy Chapter

Decision Copy Chapter

Phase 0 PASSED - FastAPI approach meets the < 0.8s threshold.

Proceed to Phase 1 implementation:

• Task 1.1: docling_subprocess_worker.py (skipped - FastAPI only)
• Task 1.2: hybrid_server.py (opendataloader-pdf-hybrid CLI)
• Task 2.1: DoclingSubprocessClient.java (skipped - FastAPI only)
• Task 2.2: DoclingFastServerClient.java
• Task 2.3: HybridClientFactory modification
• Task 3: Benchmark integration
• Task 4: Final validation

Subprocess approach also passed - both approaches available for implementation.

Detailed Statistics Copy Chapter

Baseline Copy Section

• Description: docling-serve HTTP API
• Timestamp: 2026-01-03 14:23:41
• Total documents: 200
• Successful: 200
• Failed: 0
• Total elapsed: 456.6s
• Average per doc: 2.2825s
• Min: 2.0045s
• Max: 8.0182s

Fastapi Copy Section

• Description: FastAPI server with docling SDK singleton
• Timestamp: 2026-01-03 14:27:18
• Total documents: 200
• Successful: 200
• Failed: 0
• Total elapsed: 137.1s
• Average per doc: 0.6855s
• Min: 0.1912s
• Max: 4.2420s

Subprocess Copy Section

• Description: Persistent Python subprocess with docling SDK
• Timestamp: 2026-01-03 14:30:50
• Total documents: 200
• Successful: 200
• Failed: 0
• Total elapsed: 132.4s
• Average per doc: 0.6612s
• Min: 0.1908s
• Max: 4.2498s

File: docs/hybrid/research/comparison-summary.md Copy Chapter

Docling vs OpenDataLoader Output Comparison

Test Document Copy Chapter

• File: 01030000000045.pdf (1 page with table)

Element Count Comparison Copy Chapter

Text Element Labels (Docling) Copy Chapter

Table Structure Comparison Copy Chapter

Note: Docling detects more rows in the table structure. This may be due to:

• Different table detection algorithms
• OpenDataLoader may have merged some rows
• Different handling of header rows

Bounding Box Comparison (Table) Copy Chapter

Coordinate mapping: Both use BOTTOMLEFT origin.

• Docling: {l, t, r, b} where t=top, b=bottom
• OpenDataLoader: [left, bottom, right, top]

So the actual coordinates match closely:

• Left: 53.22 ≈ 54.0
• Bottom: 234.74 ≈ 234.44
• Right: 373.94 ≈ 372.73
• Top: 439.98 ≈ 440.21

Schema Mapping Summary Copy Chapter

Key Differences Copy Chapter

1. Type naming: Docling uses label field for text types, OpenDataLoader uses type
2. Table structure: Docling detects more detailed row structure
3. Coordinate format: Same origin but different field order
4. Heading detection: Docling uses SectionHeaderItem with level, OpenDataLoader uses heading type with level

File: docs/hybrid/research/iobject-structure.md Copy Chapter

IObject Class Structure

Overview Copy Chapter

IObject is imported from org.verapdf.wcag.algorithms.entities.IObject (external verapdf-wcag-algs library).

JSON Output Types Copy Chapter

Based on sample response analysis, OpenDataLoader produces the following element types:

Element Types Copy Section

Common Fields (all types) Copy Section

{
  "type": "paragraph",
  "id": 17,
  "page number": 1,
  "bounding box": [left, bottom, right, top]  // PDF points, origin at bottom-left
}

Paragraph Fields Copy Section

{
  "type": "paragraph",
  "font": "ArialMT",
  "font size": 8.0,
  "text color": "[0.0, 0.0, 0.0, 0.7]",
  "content": "Text content here"
}

Heading Fields Copy Section

{
  "type": "heading",
  "level": "1",
  "content": "Heading text"
}

Table Structure Copy Section

{
  "type": "table",
  "level": "1",
  "number of rows": 3,
  "number of columns": 3,
  "rows": [
    {
      "type": "table row",
      "row number": 1,
      "cells": [
        {
          "type": "table cell",
          "page number": 1,
          "bounding box": [left, bottom, right, top],
          "row number": 1,
          "column number": 1,
          "row span": 1,
          "column span": 1,
          "kids": [
            {
              "type": "paragraph",
              "content": "Cell text"
            }
          ]
        }
      ]
    }
  ]
}

Bounding Box Coordinate System Copy Chapter

• OpenDataLoader: [left, bottom, right, top] in PDF points, origin at BOTTOMLEFT
• Docling: {l, t, r, b} with coord_origin: "BOTTOMLEFT" or "TOPLEFT"

Conversion Notes Copy Section

• If docling uses TOPLEFT origin: bottom = page_height - docling_t, top = page_height - docling_b
• If docling uses BOTTOMLEFT origin: direct mapping [l, b, r, t] → [left, bottom, right, top]

Key Java Classes Copy Chapter

From the codebase:

• TableBorder - Table with border-based detection
• TableBorderRow - Table row
• TableBorderCell - Table cell with contents, rowSpan, colSpan
• BoundingBox - PDF coordinates (page, left, bottom, right, top)
• Processors: TextLineProcessor, TableBorderProcessor, HeadingProcessor, ListProcessor