## File: README.md # OpenDataLoader PDF **PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.** [](https://github.com/opendataloader-project/opendataloader-pdf/blob/main/LICENSE) [](https://pypi.org/project/opendataloader-pdf/) [](https://www.npmjs.com/package/@opendataloader/pdf) [](https://search.maven.org/artifact/org.opendataloader/opendataloader-pdf-core) [](https://github.com/opendataloader-project/opendataloader-pdf#java) [](https://trendshift.io/repositories/21917) πŸ” **PDF parser for AI data extraction** β€” Extract Markdown, JSON (with bounding boxes), and HTML from any PDF. #1 in benchmarks (0.907 overall). Deterministic local mode + AI hybrid mode for complex pages. - **How accurate is it?** β€” #1 in benchmarks: 0.907 overall, 0.928 table accuracy across 200 real-world PDFs including multi-column and scientific papers. Deterministic local mode + AI hybrid mode for complex pages ([benchmarks](#extraction-benchmarks)) - **Scanned PDFs and OCR?** β€” Yes. Built-in OCR (80+ languages) in hybrid mode. Works with poor-quality scans at 300 DPI+ ([hybrid mode](#hybrid-mode-1-accuracy-for-complex-pdfs)) - **Tables, formulas, images, charts?** β€” Yes. Complex/borderless tables, LaTeX formulas, and AI-generated picture/chart descriptions all via hybrid mode ([hybrid mode](#hybrid-mode-1-accuracy-for-complex-pdfs)) - **How do I use this for RAG?** β€” `pip install opendataloader-pdf`, convert in 3 lines. Outputs structured Markdown for chunking, JSON with bounding boxes for source citations, and HTML. LangChain integration available. Python, Node.js, Java SDKs ([quick start](#get-started-in-30-seconds) | [LangChain](#langchain-integration)) β™Ώ **PDF accessibility automation** β€” Auto-tag untagged PDFs into screen-reader-ready Tagged PDFs at scale. First open-source tool to generate Tagged PDFs end-to-end. - **What's the problem?** β€” Accessibility regulations are now enforced worldwide. Manual PDF remediation costs $50–200 per document and doesn't scale ([regulations](#pdf-accessibility--pdfua-conversion)) - **What's free?** β€” Layout analysis + auto-tagging (Apache 2.0). Untagged PDF in β†’ Tagged PDF out. No proprietary SDK dependency ([auto-tagging](#auto-tagging)) - **What about PDF/UA compliance?** β€” Converting Tagged PDF to PDF/UA-1 or PDF/UA-2 is an enterprise add-on. Auto-tagging generates the Tagged PDF; PDF/UA export is the final step ([pipeline](#accessibility-pipeline)) - **Why trust this?** β€” Built in collaboration with [Dual Lab](https://duallab.com) ([veraPDF](https://verapdf.org) developers) based on [PDF Association](https://pdfa.org) specifications, best practice guides and expertise of the [PDF Community](https://pdfa.org/community/). Auto-tagging follows the [Well-Tagged PDF specification](https://pdfa.org/wtpdf/), validated with veraPDF ([collaboration](https://opendataloader.org/docs/tagged-pdf-collaboration)) ## Get Started in 30 Seconds **Requires**: Java 11+ and Python 3.10+ ([Node.js](https://opendataloader.org/docs/quick-start-nodejs) | [Java](https://opendataloader.org/docs/quick-start-java) also available) > Before you start: run `java -version`. If not found, install JDK 11+ from [Adoptium](https://adoptium.net/). ```bash pip install -U opendataloader-pdf ``` ```python import opendataloader_pdf # Batch all files in one call β€” each convert() spawns a JVM process, so repeated calls are slow opendataloader_pdf.convert( input_path=["file1.pdf", "file2.pdf", "folder/"], output_dir="output/", format="markdown,json" ) ``` *Annotated PDF output β€” each element (heading, paragraph, table, image) detected with bounding boxes and semantic type.* ## What Problems Does This Solve? | Problem | Solution | Status | |---------|----------|--------| | **PDF structure lost during parsing** β€” wrong reading order, broken tables, no element coordinates | Deterministic local PDF to Markdown/JSON with bounding boxes, XY-Cut++ reading order | Shipped | | **Complex tables, scanned PDFs, formulas, charts** need AI-level understanding | Hybrid mode routes complex pages to AI backend (#1 in benchmarks) | Shipped | | **Manual PDF remediation cost** β€” Accessibility regulations (EAA, ADA, Section 508) demand Tagged PDFs. Manual remediation costs $50–200/doc | Auto-tag untagged PDFs into Tagged PDFs (free, Apache 2.0). Foundation for PDF/UA workflows; full PDF/UA-1/2 export is an enterprise add-on | Auto-tag: Shipped. PDF/UA export: Enterprise | ## Capability Matrix | Capability | Supported | Tier | |------------|-----------|------| | **Data extraction** | | | | Extract text with correct reading order | Yes | Free | | Bounding boxes for every element | Yes | Free | | Table extraction (simple borders) | Yes | Free | | Table extraction (complex/borderless) | Yes | Free (Hybrid) | | Heading hierarchy detection | Yes | Free | | List detection (numbered, bulleted, nested) | Yes | Free | | Image extraction with coordinates | Yes | Free | | AI chart/image description | Yes | Free (Hybrid) | | OCR for scanned PDFs | Yes | Free (Hybrid) | | Formula extraction (LaTeX) | Yes | Free (Hybrid) | | Tagged PDF structure extraction | Yes | Free | | AI safety (prompt injection filtering) | Yes | Free | | Header/footer/watermark filtering | Yes | Free | | **Accessibility** | | | | Auto-tagging β†’ Tagged PDF for untagged PDFs | Yes | Free (Apache 2.0) | | PDF/UA-1, PDF/UA-2 export | πŸ’Ό Available | Enterprise | | Accessibility studio (visual editor) | πŸ’Ό Available | Enterprise | | **Limitations** | | | | Process Word/Excel/PPT | No | β€” | | GPU required | No | β€” | ## Extraction Benchmarks **opendataloader-pdf [hybrid] ranks #1 overall (0.907)** across reading order, table, and heading extraction accuracy. | Engine | Overall | Reading Order | Table | Heading | Speed (s/page) | License | |--------|---------|---------------|-------|---------|----------------|---------| | **opendataloader [hybrid]** | **0.907** | **0.934** | **0.928** | 0.821 | 0.463 | Apache-2.0 | | nutrient | 0.885 | 0.925 | 0.708 | 0.819 | **0.008** | Commercial | | docling | 0.882 | 0.898 | 0.887 | **0.824** | 0.762 | MIT | | marker | 0.861 | 0.890 | 0.808 | 0.796 | 53.932 | GPL-3.0 | | unstructured [hi_res] | 0.841 | 0.904 | 0.588 | 0.749 | 3.008 | Apache-2.0 | | edgeparse | 0.837 | 0.894 | 0.717 | 0.706 | 0.036 | Apache-2.0 | | opendataloader | 0.831 | 0.902 | 0.489 | 0.739 | 0.015 | Apache-2.0 | | mineru | 0.831 | 0.857 | 0.873 | 0.743 | 5.962 | AGPL-3.0 | | pymupdf4llm | 0.732 | 0.885 | 0.401 | 0.412 | 0.091 | AGPL-3.0 | | unstructured | 0.686 | 0.882 | 0.000 | 0.388 | 0.077 | Apache-2.0 | | markitdown | 0.589 | 0.844 | 0.273 | 0.000 | 0.114 | MIT | | liteparse | 0.576 | 0.866 | 0.000 | 0.000 | 1.061 | Apache-2.0 | > Scores normalized to [0, 1]. Higher is better for accuracy; lower is better for speed. **Bold** = best. [Full benchmark details](https://github.com/opendataloader-project/opendataloader-bench) [](https://github.com/opendataloader-project/opendataloader-bench) [](https://github.com/opendataloader-project/opendataloader-bench) ## Which Mode Should I Use? | Your Document | Mode | Install | Server Command | Client Command | |---------------|------|---------|----------------|----------------| | Standard digital PDF | Fast (default) | `pip install opendataloader-pdf` | None needed | `opendataloader-pdf file1.pdf file2.pdf folder/` | | Complex or nested tables | **Hybrid** | `pip install "opendataloader-pdf[hybrid]"` | `opendataloader-pdf-hybrid --port 5002` | `opendataloader-pdf --hybrid docling-fast file1.pdf file2.pdf folder/` | | Scanned / image-based PDF | Hybrid + OCR | `pip install "opendataloader-pdf[hybrid]"` | `opendataloader-pdf-hybrid --port 5002 --force-ocr` | `opendataloader-pdf --hybrid docling-fast file1.pdf file2.pdf folder/` | | Non-English scanned PDF | Hybrid + OCR | `pip install "opendataloader-pdf[hybrid]"` | `opendataloader-pdf-hybrid --port 5002 --force-ocr --ocr-lang "ko,en"` | `opendataloader-pdf --hybrid docling-fast file1.pdf file2.pdf folder/` | | Mathematical formulas | Hybrid + formula | `pip install "opendataloader-pdf[hybrid]"` | `opendataloader-pdf-hybrid --enrich-formula` | `opendataloader-pdf --hybrid docling-fast --hybrid-mode full file1.pdf file2.pdf folder/` | | Charts needing description | Hybrid + picture | `pip install "opendataloader-pdf[hybrid]"` | `opendataloader-pdf-hybrid --enrich-picture-description` | `opendataloader-pdf --hybrid docling-fast --hybrid-mode full file1.pdf file2.pdf folder/` | | Untagged PDFs needing accessibility | Auto-tagging β†’ Tagged PDF | `pip install opendataloader-pdf` | None needed | `opendataloader-pdf --format tagged-pdf file1.pdf file2.pdf folder/` | ## Quick Start ### Python ```bash pip install -U opendataloader-pdf ``` ```python import opendataloader_pdf # Batch all files in one call β€” each convert() spawns a JVM process, so repeated calls are slow opendataloader_pdf.convert( input_path=["file1.pdf", "file2.pdf", "folder/"], output_dir="output/", format="markdown,json" ) ``` ### Node.js ```bash npm install @opendataloader/pdf ``` ```typescript import { convert } from '@opendataloader/pdf'; await convert(['file1.pdf', 'file2.pdf', 'folder/'], { outputDir: 'output/', format: 'markdown,json' }); ``` ### Java ```xml org.opendataloader opendataloader-pdf-core ``` [Python Quick Start](https://opendataloader.org/docs/quick-start-python) | [Node.js Quick Start](https://opendataloader.org/docs/quick-start-nodejs) | [Java Quick Start](https://opendataloader.org/docs/quick-start-java) ## Hybrid Mode: #1 Accuracy for Complex PDFs Hybrid mode combines fast local Java processing with AI backends. Simple pages stay local (0.02s); complex pages route to AI for +90% table accuracy. > **Don't combine with `--use-struct-tree` on tagged PDFs.** `--use-struct-tree` takes precedence, so the hybrid backend is **not** called (a warning is logged). If you want the hybrid backend, drop `--use-struct-tree`. ```bash pip install -U "opendataloader-pdf[hybrid]" ``` **Terminal 1** β€” Start the backend server: ```bash opendataloader-pdf-hybrid --port 5002 ``` **Terminal 2** β€” Process PDFs: ```bash # Batch all files in one call β€” each invocation spawns a JVM process, so repeated calls are slow opendataloader-pdf --hybrid docling-fast file1.pdf file2.pdf folder/ ``` **Python:** ```python # Batch all files in one call β€” each convert() spawns a JVM process, so repeated calls are slow opendataloader_pdf.convert( input_path=["file1.pdf", "file2.pdf", "folder/"], output_dir="output/", hybrid="docling-fast" ) ``` ### OCR for Scanned PDFs Start the backend with `--force-ocr` for image-based PDFs with no selectable text: ```bash opendataloader-pdf-hybrid --port 5002 --force-ocr ``` For non-English documents, specify the language: ```bash opendataloader-pdf-hybrid --port 5002 --force-ocr --ocr-lang "ko,en" ``` Supported languages: `en`, `ko`, `ja`, `ch_sim`, `ch_tra`, `de`, `fr`, `ar`, and more. ### Formula Extraction (LaTeX) Extract mathematical formulas as LaTeX from scientific PDFs: ```bash # Server: enable formula enrichment opendataloader-pdf-hybrid --enrich-formula # Batch all files in one call β€” each invocation spawns a JVM process, so repeated calls are slow opendataloader-pdf --hybrid docling-fast --hybrid-mode full file1.pdf file2.pdf folder/ ``` Output in JSON: ```json { "type": "formula", "page number": 1, "bounding box": [226.2, 144.7, 377.1, 168.7], "content": "\\frac{f(x+h) - f(x)}{h}" } ``` > **Note**: Formula and picture description enrichments require `--hybrid-mode full` on the client side. ### Chart & Image Description Generate AI descriptions for charts and images β€” useful for RAG search and accessibility alt text: ```bash # Server opendataloader-pdf-hybrid --enrich-picture-description # Batch all files in one call β€” each invocation spawns a JVM process, so repeated calls are slow opendataloader-pdf --hybrid docling-fast --hybrid-mode full file1.pdf file2.pdf folder/ ``` Output in JSON: ```json { "type": "picture", "page number": 1, "bounding box": [72.0, 400.0, 540.0, 650.0], "description": "A bar chart showing waste generation by region from 2016 to 2030..." } ``` > Uses SmolVLM (256M), a lightweight vision model. Custom prompts supported via `--picture-description-prompt`. ### Hancom Data Loader Integration β€” Coming Soon Enterprise-grade AI document analysis via [Hancom Data Loader](https://sdk.hancom.com/en/services/1?utm_source=github&utm_medium=readme&utm_campaign=opendataloader-pdf) β€” customer-customized models trained on your domain-specific documents. 30+ element types (tables, charts, formulas, captions, footnotes, etc.), VLM-based image/chart understanding, complex table extraction (merged cells, nested tables), SLA-backed OCR for scanned documents, and native HWP/HWPX support. Supports PDF, DOCX, XLSX, PPTX, HWP, PNG, JPG. [Live demo](https://livedemo.sdk.hancom.com/en/dataloader?utm_source=github&utm_medium=readme&utm_campaign=opendataloader-pdf) [Hybrid Mode Guide](https://opendataloader.org/docs/hybrid-mode) ## Output Formats | Format | Use Case | |--------|----------| | **JSON** | Structured data with bounding boxes, semantic types | | **Markdown** | Clean text for LLM context, RAG chunks | | **HTML** | Web display with styling | | **Annotated PDF** | Visual debugging β€” see detected structures ([sample](https://opendataloader.org/demo/samples/01030000000000)) | | **Text** | Plain text extraction | Combine formats: `format="json,markdown"` ### JSON Output Example ```json { "type": "heading", "id": 42, "level": "Title", "page number": 1, "bounding box": [72.0, 700.0, 540.0, 730.0], "heading level": 1, "font": "Helvetica-Bold", "font size": 24.0, "text color": "[0.0]", "content": "Introduction" } ``` | Field | Description | |-------|-------------| | `type` | Element type: heading, paragraph, table, list, image, caption, formula | | `id` | Unique identifier for cross-referencing | | `page number` | 1-indexed page reference | | `bounding box` | `[left, bottom, right, top]` in PDF points (72pt = 1 inch) | | `heading level` | Heading depth (1+) | | `content` | Extracted text | [Full JSON Schema](https://opendataloader.org/docs/reference/json-schema) ## Advanced Features ### Tagged PDF Support When a PDF has structure tags, OpenDataLoader extracts the **exact layout** the author intended β€” no guessing, no heuristics. Headings, lists, tables, and reading order are preserved from the source. > **Output quality depends on tag quality.** Not all tagged PDFs are well-tagged. For PDFs with sparse or incorrect tags, the default heuristic mode or `--hybrid docling-fast` often produces better results. > **`--use-struct-tree` takes precedence over `--hybrid`.** If both are set on a tagged PDF, the structure tree is used and the hybrid backend is **not** called (a well-tagged PDF already carries reading order and structure). Drop `--use-struct-tree` if you want the hybrid backend instead. ```python # Batch all files in one call β€” each convert() spawns a JVM process, so repeated calls are slow opendataloader_pdf.convert( input_path=["file1.pdf", "file2.pdf", "folder/"], output_dir="output/", use_struct_tree=True # Use native PDF structure tags ) ``` Most PDF parsers ignore structure tags entirely. [Learn more](https://opendataloader.org/docs/tagged-pdf) ### AI Safety: Prompt Injection Protection PDFs can contain hidden prompt injection attacks. OpenDataLoader automatically filters: - Hidden text (transparent, zero-size fonts) - Off-page content - Suspicious invisible layers To sanitize sensitive data (emails, URLs, phone numbers β†’ placeholders), enable it explicitly: ```bash # Batch all files in one call β€” each invocation spawns a JVM process, so repeated calls are slow opendataloader-pdf file1.pdf file2.pdf folder/ --sanitize ``` [AI Safety Guide](https://opendataloader.org/docs/ai-safety) ### LangChain Integration ```bash pip install -U langchain-opendataloader-pdf ``` ```python from langchain_opendataloader_pdf import OpenDataLoaderPDFLoader loader = OpenDataLoaderPDFLoader( file_path=["file1.pdf", "file2.pdf", "folder/"], format="text" ) documents = loader.load() ``` [LangChain Docs](https://docs.langchain.com/oss/python/integrations/document_loaders/opendataloader_pdf) | [GitHub](https://github.com/opendataloader-project/langchain-opendataloader-pdf) | [PyPI](https://pypi.org/project/langchain-opendataloader-pdf/) ### Advanced Options ```python # Batch all files in one call β€” each convert() spawns a JVM process, so repeated calls are slow opendataloader_pdf.convert( input_path=["file1.pdf", "file2.pdf", "folder/"], output_dir="output/", format="json,markdown,pdf", image_output="embedded", # "off", "embedded" (Base64), or "external" (default) image_format="jpeg", # "png" or "jpeg" use_struct_tree=True, # Use native PDF structure ) ``` [Full CLI Options Reference](https://opendataloader.org/docs/reference/cli-options) ## PDF Accessibility & PDF/UA Conversion **Problem**: Millions of existing PDFs lack structure tags, failing accessibility regulations (EAA, ADA/Section 508, Korea Digital Inclusion Act). Manual remediation costs $50–200 per document and doesn't scale. **OpenDataLoader's approach**: Built in collaboration with [PDF Association](https://pdfa.org) and [Dual Lab](https://duallab.com) (developers of [veraPDF](https://verapdf.org), the industry-reference open-source PDF/A and PDF/UA validator). Auto-tagging follows the [Well-Tagged PDF specification](https://pdfa.org/resource/well-tagged-pdf/) and is validated programmatically using veraPDF β€” automated conformance checks against PDF accessibility standards, not manual review. No existing open-source tool generates Tagged PDFs end-to-end β€” most rely on proprietary SDKs for the tag-writing step. OpenDataLoader does it all under Apache 2.0. ([collaboration details](https://opendataloader.org/docs/tagged-pdf-collaboration)) | Regulation | Deadline | Requirement | |------------|----------|-------------| | **European Accessibility Act (EAA)** | June 28, 2025 | Accessible digital products across the EU | | **ADA & Section 508** | In effect | U.S. federal agencies and public accommodations | | **Digital Inclusion Act** | In effect | South Korea digital service accessibility | ### Standards & Validation | Aspect | Detail | |--------|--------| | **Specification** | [Well-Tagged PDF](https://pdfa.org/resource/well-tagged-pdf/) by PDF Association | | **Validation** | [veraPDF](https://verapdf.org) β€” industry-reference open-source PDF/A & PDF/UA validator | | **Collaboration** | PDF Association + [Dual Lab](https://duallab.com) (veraPDF developers) co-develop tagging and validation | | **License** | Auto-tagging β†’ Tagged PDF: Apache 2.0 (free). PDF/UA export: Enterprise | ### Accessibility Pipeline | Step | Feature | Status | Tier | |------|---------|--------|------| | 1. **Audit** | Read existing PDF tags, detect untagged PDFs | Shipped | Free | | 2. **Auto-tag β†’ Tagged PDF** | Generate structure tags for untagged PDFs | Shipped | Free (Apache 2.0) | | 3. **Export PDF/UA** | Convert to PDF/UA-1 or PDF/UA-2 compliant files | πŸ’Ό Available | Enterprise | | 4. **Visual editing** | Accessibility studio β€” review and fix tags | πŸ’Ό Available | Enterprise | > **πŸ’Ό Enterprise features** are available on request. [Contact us](https://opendataloader.org/contact) to get started. ### Auto-Tagging Generate Tagged PDFs from untagged PDFs β€” output is a screen-reader-ready PDF with structure tags (headings, paragraphs, lists, tables, reading order). ```python import opendataloader_pdf # Untagged PDF in β†’ Tagged PDF out opendataloader_pdf.convert( input_path=["file1.pdf", "file2.pdf", "folder/"], output_dir="output/", format="tagged-pdf" ) ``` ```bash # CLI opendataloader-pdf --format tagged-pdf file1.pdf file2.pdf folder/ ``` Combine with other formats: `format="json,tagged-pdf"`. ### End-to-End Compliance Workflow ``` Existing PDFs (untagged) β”‚ β–Ό β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ 1. Audit │───>β”‚ 2. Auto-Tag │───>β”‚ 3. Export │───>β”‚ 4. Studio β”‚ β”‚ (check tags) β”‚ β”‚ (β†’ Tagged PDF) β”‚ β”‚ (PDF/UA) β”‚ β”‚ (visual editor) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β–Ό β–Ό β–Ό β–Ό use_struct_tree format="tagged-pdf" PDF/UA export Accessibility Studio (Available now) (Available, Apache 2.0) (Enterprise) (Enterprise) ``` [PDF Accessibility Guide](https://opendataloader.org/docs/accessibility-compliance) ## Roadmap | Feature | Timeline | Tier | |---------|----------|------| | **[Hancom Data Loader](https://sdk.hancom.com/en/services/1?utm_source=github&utm_medium=readme&utm_campaign=opendataloader-pdf)** β€” Enterprise AI document analysis, customer-customized models, VLM-based chart/image understanding, production-grade OCR | Q2-Q3 2026 | Planned | | **Structure validation** β€” Verify PDF tag trees | Q3 2026 | Planned | [Full Roadmap](https://opendataloader.org/docs/upcoming-roadmap) ## Frequently Asked Questions ### What is the best PDF parser for RAG? For RAG pipelines, you need a parser that preserves document structure, maintains correct reading order, and provides element coordinates for citations. OpenDataLoader is designed specifically for this β€” it outputs structured JSON with bounding boxes, handles multi-column layouts with XY-Cut++, and runs locally without GPU. In hybrid mode, it ranks #1 overall (0.907) in benchmarks. ### What is the best open-source PDF parser? OpenDataLoader PDF is the only open-source parser that combines: rule-based deterministic extraction (no GPU), bounding boxes for every element, XY-Cut++ reading order, built-in AI safety filters, native Tagged PDF support, and hybrid AI mode for complex documents. It ranks #1 in overall accuracy (0.907) while running locally on CPU. ### How do I extract tables from PDF for LLM? OpenDataLoader detects tables using border analysis and text clustering, preserving row/column structure. For complex tables, enable hybrid mode for +90% accuracy improvement (0.489 to 0.928 TEDS score): ```python # Batch all files in one call β€” each convert() spawns a JVM process, so repeated calls are slow opendataloader_pdf.convert( input_path=["file1.pdf", "file2.pdf", "folder/"], output_dir="output/", format="json", hybrid="docling-fast" # For complex tables ) ``` ### How does it compare to docling, marker, or pymupdf4llm? OpenDataLoader [hybrid] ranks #1 overall (0.907) across reading order, table, and heading accuracy. Key differences: docling (0.882) is strong but lacks bounding boxes and AI safety filters. marker (0.861) requires GPU and is 1000x slower (53.932s/page). pymupdf4llm (0.732) is fast but has poor table (0.401) and heading (0.412) accuracy. OpenDataLoader is the only parser that combines deterministic local extraction, bounding boxes for every element, and built-in prompt injection protection. See [full benchmark](https://github.com/opendataloader-project/opendataloader-bench). ### Can I use this without sending data to the cloud? Yes. OpenDataLoader runs 100% locally. No API calls, no data transmission β€” your documents never leave your environment. The hybrid mode backend also runs locally on your machine. Ideal for legal, healthcare, and financial documents. ### Does it support OCR for scanned PDFs? Yes, via hybrid mode. Install with `pip install "opendataloader-pdf[hybrid]"`, start the backend with `--force-ocr`, then process as usual. Supports multiple languages including Korean, Japanese, Chinese, Arabic, and more via `--ocr-lang`. ### Does it work with Korean, Japanese, or Chinese documents? Yes. For digital PDFs, text extraction works out of the box. For scanned PDFs, use hybrid mode with `--force-ocr --ocr-lang "ko,en"` (or `ja`, `ch_sim`, `ch_tra`). Coming soon: [Hancom Data Loader](https://sdk.hancom.com/en/services/1?utm_source=github&utm_medium=readme&utm_campaign=opendataloader-pdf) integration β€” enterprise-grade AI document analysis with built-in production-grade OCR and customer-customized models optimized for your specific document types and workflows. ### How fast is it? Local mode processes 60+ pages per second on CPU (0.02s/page). Hybrid mode processes 2+ pages per second (0.46s/page) with significantly higher accuracy for complex documents. No GPU required. Benchmarked on Apple M4. [Full benchmark details](https://github.com/opendataloader-project/opendataloader-bench). With multi-process batch processing, throughput exceeds 100 pages per second on 8+ core machines. ### Does it handle multi-column layouts? Yes. OpenDataLoader uses XY-Cut++ reading order analysis to correctly sequence text across multi-column pages, sidebars, and mixed layouts. This works in both local and hybrid modes without any configuration. ### What is hybrid mode? Hybrid mode combines fast local Java processing with an AI backend. Simple pages are processed locally (0.02s/page); complex pages (tables, scanned content, formulas, charts) are automatically routed to the AI backend for higher accuracy. The backend runs locally on your machine β€” no cloud required. See [Which Mode Should I Use?](#which-mode-should-i-use) and [Hybrid Mode Guide](https://opendataloader.org/docs/hybrid-mode). ### Does it work with LangChain? Yes. Install `langchain-opendataloader-pdf` for an official LangChain document loader integration. See [LangChain docs](https://docs.langchain.com/oss/python/integrations/document_loaders/opendataloader_pdf). ### How do I chunk PDFs for RAG? OpenDataLoader outputs structured Markdown with headings, tables, and lists preserved β€” ideal input for semantic chunking. Each element in JSON output includes `type`, `heading level`, and `page number`, so you can split by section or page boundary. For most RAG pipelines: parse with `format="markdown"` for text chunks, or `format="json"` when you need element-level control. Pair with LangChain's `RecursiveCharacterTextSplitter` or your own heading-based splitter for best results. ### How do I cite PDF sources in RAG answers? Every element in JSON output includes a `bounding box` (`[left, bottom, right, top]` in PDF points) and `page number`. When your RAG pipeline returns an answer, map the source chunk back to its bounding box to highlight the exact location in the original PDF. This enables "click to source" UX β€” users see which paragraph, table, or figure the answer came from. No other open-source parser provides bounding boxes for every element by default. ### How do I convert PDF to Markdown for LLM? ```python import opendataloader_pdf # Batch all files in one call β€” each convert() spawns a JVM process, so repeated calls are slow opendataloader_pdf.convert( input_path=["file1.pdf", "file2.pdf", "folder/"], output_dir="output/", format="markdown" ) ``` OpenDataLoader preserves heading hierarchy, table structure, and reading order in the Markdown output. For complex documents with borderless tables or scanned pages, use hybrid mode (`hybrid="docling-fast"`) for higher accuracy. The output is clean enough to feed directly into LLM context windows or RAG chunking pipelines. ### Is there an automated PDF accessibility remediation tool? Yes. OpenDataLoader is the first open-source tool that automates PDF accessibility end-to-end. Built in collaboration with [PDF Association](https://pdfa.org) and [Dual Lab](https://duallab.com) (veraPDF developers), auto-tagging follows the Well-Tagged PDF specification and is validated programmatically using veraPDF. The layout analysis engine detects document structure (headings, tables, lists, reading order) and generates accessibility tags automatically. Auto-tagging converts untagged PDFs into Tagged PDFs under Apache 2.0 β€” no proprietary SDK dependency. Use `format="tagged-pdf"` (Python/Node.js) or `--format tagged-pdf` (CLI). For organizations needing full PDF/UA compliance, enterprise add-ons provide PDF/UA export and a visual tag editor. This replaces manual remediation workflows that typically cost $50–200+ per document. ### Is this really the first open-source PDF auto-tagging tool? Yes. Existing tools either depend on proprietary SDKs for writing structure tags, only output non-PDF formats (e.g., Docling outputs Markdown/JSON but cannot produce Tagged PDFs), or require manual intervention. OpenDataLoader is the first to do layout analysis β†’ tag generation β†’ Tagged PDF output entirely under an open-source license (Apache 2.0), with no proprietary dependency. Auto-tagging follows the PDF Association's Well-Tagged PDF specification and is validated using veraPDF, the industry-reference open-source PDF/A and PDF/UA validator. ### How do I convert existing PDFs to PDF/UA? OpenDataLoader provides an end-to-end pipeline: audit existing PDFs for tags (`use_struct_tree=True`), auto-tag untagged PDFs into Tagged PDFs (`format="tagged-pdf"`, free under Apache 2.0), and export as PDF/UA-1 or PDF/UA-2 (enterprise add-on). Auto-tagging follows the PDF Association's Well-Tagged PDF specification and is validated using veraPDF. Auto-tagging generates the Tagged PDF; PDF/UA export is the final step. [Contact us](https://opendataloader.org/contact) for enterprise integration. ### How do I make my PDFs accessible for EAA compliance? The European Accessibility Act requires accessible digital products by June 28, 2025. OpenDataLoader supports the full remediation workflow: audit β†’ auto-tag β†’ Tagged PDF β†’ PDF/UA export. Auto-tagging follows the PDF Association's Well-Tagged PDF specification and is validated using veraPDF, ensuring standards-compliant output. Auto-tagging to Tagged PDF is open-source under Apache 2.0. PDF/UA export and accessibility studio are enterprise add-ons. See our [Accessibility Guide](https://opendataloader.org/docs/accessibility-compliance). ### Is OpenDataLoader PDF free? The core library is **open-source under Apache 2.0** β€” free for commercial use. This includes all extraction features (text, tables, images, OCR, formulas, charts via hybrid mode), AI safety filters, Tagged PDF support, and auto-tagging to Tagged PDF. We are committed to keeping the core accessibility pipeline (layout analysis β†’ auto-tagging β†’ Tagged PDF) free and open-source. Enterprise add-ons (PDF/UA export, accessibility studio) are available for organizations needing end-to-end regulatory compliance. ### Why did the license change from MPL 2.0 to Apache 2.0? MPL 2.0 requires file-level copyleft, which often triggers legal review before enterprise adoption. Apache 2.0 is fully permissive β€” no copyleft obligations, easier to integrate into commercial projects. If you are using a pre-2.0 version, it remains under MPL 2.0 and you can continue using it. Upgrading to 2.0+ means your project follows Apache 2.0 terms, which are strictly more permissive β€” no additional obligations, no action needed on your side. ## Documentation - [Quick Start (Python)](https://opendataloader.org/docs/quick-start-python) - [Quick Start (Node.js)](https://opendataloader.org/docs/quick-start-nodejs) - [Quick Start (Java)](https://opendataloader.org/docs/quick-start-java) - [JSON Schema Reference](https://opendataloader.org/docs/reference/json-schema) - [CLI Options](https://opendataloader.org/docs/reference/cli-options) - [Hybrid Mode Guide](https://opendataloader.org/docs/hybrid-mode) - [Tagged PDF Support](https://opendataloader.org/docs/tagged-pdf) - [AI Safety Features](https://opendataloader.org/docs/ai-safety) - [PDF Accessibility](https://opendataloader.org/docs/accessibility-compliance) ## Contributing We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. ## License [Apache License 2.0](LICENSE) > **Note:** Versions prior to 2.0 are licensed under the [Mozilla Public License 2.0](https://www.mozilla.org/MPL/2.0/). --- **Found this useful?** Give us a star to help others discover OpenDataLoader. --- ## File: docs/superpowers/specs/2026-03-16-cid-font-detection-design.md # CID Font Extraction Failure Detection Issue: [#286](https://github.com/opendataloader-project/opendataloader-pdf/issues/286) ## Problem PDFs with CID-keyed fonts that lack ToUnicode mappings produce no usable text from veraPDF extraction. veraPDF replaces unmappable characters with U+FFFD (replacement character), which `TextProcessor.replaceUndefinedCharacters()` then converts to spaces. The result is empty or whitespace-only output with no indication to the user of what went wrong. Users currently resort to external tools (e.g., pdfplumber) to pre-screen PDFs for CID issues before passing them to opendataloader-pdf. ## Solution Detect pages with high replacement character ratios and: 1. **Always**: emit a WARNING log explaining the problem and suggesting `--hybrid-mode` 2. **When hybrid mode is on**: automatically route affected pages to OCR backend via TriageProcessor No new CLI options. Hybrid mode setting is respected as-is. ## Design ### Detection: Replacement Character Ratio `TextProcessor.measureReplacementCharRatio(List)` counts `\uFFFD` characters across all TextChunks on a page and returns the ratio (0.0–1.0). **Threshold**: 30%. CID-affected pages typically show 90%+ replacement characters. 30% catches real problems while avoiding false positives from PDFs with occasional unmappable glyphs. **Measurement point**: Inside `ContentFilterProcessor.getFilteredContents()`, immediately before `replaceUndefinedCharacters()` is called (line 74). At this point veraPDF has already inserted `\uFFFD` but the characters haven't been replaced with spaces yet, so measurement is accurate. **Safety of measurement point**: The prior processing steps (`mergeCloseTextChunks`, `trimTextChunksWhiteSpaces`, `filterConsecutiveSpaces`, `splitTextChunksByWhiteSpaces`) do not affect U+FFFD characters. U+FFFD is not whitespace, so it is not trimmed, compressed, or used as a split boundary. The count is accurate at this position. **Zero-text pages**: When a page has no TextChunk objects (e.g., image-only pages), the method returns 0.0 to avoid division by zero. This correctly avoids triggering the CID warning on non-text pages. The method uses `ChunkParser.REPLACEMENT_CHARACTER_STRING` constant (not a hardcoded `"\uFFFD"` literal) to stay consistent with `replaceUndefinedCharacters()`. ### Data Flow The measured ratio is stored in `StaticLayoutContainers` per page: ``` ContentFilterProcessor.getFilteredContents() β”‚ β”œβ”€ TextProcessor.measureReplacementCharRatio() β†’ ratio β”œβ”€ StaticLayoutContainers.setReplacementCharRatio(pageNumber, ratio) β”œβ”€ if ratio >= 0.3: LOGGER.warning(...) └─ TextProcessor.replaceUndefinedCharacters() // existing call ``` Note: `StaticLayoutContainers` currently stores global `ThreadLocal` scalars and lists, not per-page maps. Per-page data (e.g., bounding boxes) lives in `DocumentProcessor`. This change introduces a new per-page `Map` pattern to `StaticLayoutContainers`. We place it here rather than `DocumentProcessor` because it is layout-metadata consumed by `TriageProcessor`, keeping the triage data path self-contained. The existing `clearContainers()` method **must** be updated to clear this map to prevent cross-document data leakage in multi-document processing. ### Warning Log Emitted from `ContentFilterProcessor` when ratio >= 0.3: ``` WARNING: Page 3: 94% of characters are replacement characters (U+FFFD). This PDF likely contains CID-keyed fonts without ToUnicode mappings. Text extraction may be incomplete. Consider using --hybrid-mode for OCR fallback. ``` This fires regardless of hybrid mode setting. ### Triage Routing In `TriageProcessor.classifyPage()`, a new **Signal 0** is inserted before all existing signals (before TableBorder check). This signal only fires when hybrid mode is active, since `classifyPage()` is only called from `HybridDocumentProcessor`. In non-hybrid mode, only the warning log (from `ContentFilterProcessor`) is emitted: ```java double replacementRatio = StaticLayoutContainers.getReplacementCharRatio(pageNumber); if (replacementRatio >= 0.3) { return TriageResult.backend(pageNumber, 1.0, signals); } ``` Priority is highest (confidence 1.0) because a page with mostly broken text extraction gains nothing from Java-path processing. ### Behavior Matrix | Hybrid Mode | Ratio >= 30% | Result | |---|---|---| | OFF | Yes | Warning log. Java path produces incomplete text. | | OFF | No | No change. Normal processing. | | ON (auto) | Yes | Warning log + auto-route to BACKEND (OCR). | | ON (auto) | No | No change. Normal triage. | | ON (full) | Yes | Warning log. All pages already go to BACKEND. | | ON (full) | No | No change. All pages already go to BACKEND. | ## Changes ### Modified Files | File | Change | |---|---| | `TextProcessor.java` | Add `measureReplacementCharRatio()` static method | | `ContentFilterProcessor.java` | Call measurement before `replaceUndefinedCharacters()`, store result, emit warning | | `StaticLayoutContainers.java` | Add `replacementCharRatios` map with getter/setter, clear in `clearContainers()` | | `TriageProcessor.java` | Add Signal 0: replacement ratio check before TableBorder signal | ### New Files | File | Purpose | |---|---| | `java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java` | e2e test using synthetic CID PDF | | `java/opendataloader-pdf-core/src/test/resources/cid-font-no-tounicode.pdf` | Pre-generated test fixture (CID font, no ToUnicode) | | `java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py` | Generation script for reference | ### Modified Test Files | File | Change | |---|---| | `TextProcessorTest.java` | 5 unit tests for `measureReplacementCharRatio()` | | `TriageProcessorTest.java` | 3 unit tests for Signal 0 routing | ## Test Plan ### Unit Tests (TextProcessorTest) - `testMeasureReplacementCharRatio_allReplacement` β€” all U+FFFD β†’ 1.0 - `testMeasureReplacementCharRatio_noReplacement` β€” normal text β†’ 0.0 - `testMeasureReplacementCharRatio_mixed` β€” 30% U+FFFD β†’ 0.3 - `testMeasureReplacementCharRatio_emptyContents` β€” empty list β†’ 0.0 - `testMeasureReplacementCharRatio_nonTextChunksIgnored` β€” non-text objects skipped ### Unit Tests (TriageProcessorTest) - `testClassifyPage_highReplacementRatio_routesToBackend` β€” ratio 0.5 β†’ BACKEND - `testClassifyPage_lowReplacementRatio_noEffect` β€” ratio 0.1 β†’ JAVA (default) - `testClassifyPage_exactThreshold_routesToBackend` β€” ratio 0.3 β†’ BACKEND ### Boundary Tests - `testWarningNotEmitted_belowThreshold` β€” ratio 0.29 β†’ no warning log emitted - `testWarningEmitted_atThreshold` β€” ratio 0.30 β†’ warning log emitted ### e2e Test (CidFontDetectionTest) - Load pre-generated `cid-font-no-tounicode.pdf` - Run through `ContentFilterProcessor.getFilteredContents()` - Assert: `StaticLayoutContainers.getReplacementCharRatio(0) >= 0.3` - Assert: warning log contains "replacement characters" ### Benchmark Regression - Existing benchmark PDFs are normal documents with near-zero replacement ratios - New logic does not affect existing test/benchmark results ## Not In Scope - New CLI options (no `--cid-fallback` or similar) - `npm run sync` not required (no CLI option changes) - API signature changes (backward compatible) - Benchmark threshold changes --- ## File: docs/superpowers/specs/2026-04-29-hybrid-hancom-ai-options-design.md # Hybrid hancom-ai Backend Options & CLI Refactoring Issue: not yet filed β€” this design precedes the issue/PRs. Scope: opendataloader-pdf (core CLI) + opendataloader-pdfua (downstream CLI) ## Problem Five hancom-ai backend behaviors are already wired through `HybridConfig` but not exposed on any CLI: | HybridConfig field | Default | Effect when set | |---|---|---| | `regionlistStrategy` | `table-first` | How to handle DLA label 7 (regionlist) | | `ocrStrategy` | `auto` | Stream-only / fallback / OCR-only | | `imageCache` | `memory` | Page image cache backing | | `saveCrops` | `false` | Persist cropped figures (debug) | | `cropOutputDir` | `null` | Output dir for saved crops (debug) | `HancomAISchemaTransformer` and `HancomAIClient` already read these via `HybridConfig`, but `CLIOptions.java` has no flags and `applyHybridOptions()` never sets them. They are reachable only through programmatic `Config` use. A second, structural problem makes adding these flags painful: - **opendataloader-pdf** defines all CLI options in `CLIOptions.OPTION_DEFINITIONS` (private, single source for `options.json`/Python/Node bindings). - **opendataloader-pdfua** defines its own `--hybrid`, `--hybrid-url`, `--hybrid-mode` separately in `Main.java`, with different defaults, then forwards them through `RemediationConfig` (3 hybrid fields, 9 constructor overloads). Adding 5 new flags to both CLIs the current way means duplicating definitions, expanding `RemediationConfig` to 8 hybrid fields, and creating yet more constructor overloads. The two CLIs will drift further apart. ## Solution Two coupled changes, executed in order: 1. **Add the 5 hancom-ai-specific options** to core `CLIOptions` under a `--hybrid-hancom-ai-*` prefix, with a guard that rejects them when `--hybrid` is not `hancom-ai`. 2. **Refactor core `CLIOptions` to be reusable** so opendataloader-pdfua imports the full core option set and adds only its own pdfua-specific options on top. `RemediationConfig` embeds a core `Config` instead of carrying parallel hybrid fields. After this, adding any future core CLI option propagates to pdfua with a rebuild β€” no Main.java edits. ## Design ### Option naming: `--hybrid-hancom-ai-*` (full path) Decision rationale (recorded for future contributors): - `--hybrid-*` alone (e.g. `--hybrid-regionlist-strategy`) was rejected because docling-fast and other backends will never support these knobs; the name would lie about scope. - `--hancom-ai-*` alone (e.g. `--hancom-ai-regionlist-strategy`) was rejected because it breaks the existing `--hybrid-mode/url/timeout/fallback` grouping and gives users two mental models. - `--hybrid-hancom-ai-*` mirrors the `gh pr create` / `git remote add` full-path convention: every option name encodes the context it belongs to. `--help` alphabetic sort keeps all hybrid options in one block. - A single comma-separated `--hybrid-hancom-ai-config` mega-option was rejected: defeats Apache Commons CLI typo detection, breaks `options.json` codegen for Python/Node bindings, complicates Windows path escaping. The five new options: | Long option | Type | Default | Exported | Description | |---|---|---|---|---| | `--hybrid-hancom-ai-regionlist-strategy` | string | `table-first` | yes | DLA label 7 handling. Values: `table-first`, `list-only` | | `--hybrid-hancom-ai-ocr-strategy` | string | `auto` | yes | OCR strategy. Values: `off`, `auto`, `force` | | `--hybrid-hancom-ai-image-cache` | string | `memory` | yes | Page image cache. Values: `memory`, `disk` | | `--hybrid-hancom-ai-save-crops` | boolean | `false` | **no** | Persist cropped figures (debug only) | | `--hybrid-hancom-ai-crop-output-dir` | string | `null` | **no** | Output directory for `--hybrid-hancom-ai-save-crops` | `exported=false` for the two debug options keeps them out of `options.json` and the auto-generated Python/Node bindings, while remaining usable on the Java CLI. Same pattern as existing legacy options at `CLIOptions.java:201-208`. ### Validation Inside `applyHybridOptions()`, after `--hybrid` is parsed, before returning: ```java boolean usesHancomAiOnly = commandLine.hasOption(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION) || commandLine.hasOption(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION) || commandLine.hasOption(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION) || commandLine.hasOption(HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION) || commandLine.hasOption(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION); if (usesHancomAiOnly && !Config.HYBRID_HANCOM_AI.equals(config.getHybrid())) { throw new IllegalArgumentException( "Options --hybrid-hancom-ai-* require --hybrid=hancom-ai"); } ``` Per-value validation (e.g. `regionlistStrategy` must be `table-first`/`list-only`) is already implemented in `HybridConfig` setters and re-thrown as `IllegalArgumentException`. CLI passes the raw value through; HybridConfig is the validation authority. ### Core CLIOptions refactoring Goal: pdfua can register every core option in its own `Options` and ask core to apply them to a `Config`. Two new public static methods on `CLIOptions`: ```java /** Register every core option onto an external Options. Used by downstream CLIs (pdfua). */ public static void addAllTo(Options options) { for (OptionDefinition def : OPTION_DEFINITIONS) { options.addOption(def.toOption()); } } /** Apply parsed core options to a Config. Used by downstream CLIs after parse. */ public static void applyAllTo(Config config, CommandLine commandLine) { // body identical to current createConfigFromCommandLine, // minus `new Config()` and minus the positional-arg output-folder fallback } ``` The existing `defineOptions()` and `createConfigFromCommandLine()` keep their signatures and behavior; internally they delegate to the two new methods. Backward compatibility for any existing callers is preserved. `OPTION_DEFINITIONS` stays private. `OptionDefinition` stays private. We expose only the two operations downstream CLIs actually need. ### pdfua/Main.java refactoring Replace the self-defined hybrid option block (`Main.java:60-62, 99-101`) with: ```java Options options = new Options(); CLIOptions.addAllTo(options); // all core options options.addOption(null, "lang", true, ...); // pdfua-specific only options.addOption(null, "audit-bundle-mode", true, ...); options.addOption(null, "font-embed-mode", true, ...); options.addOption(null, "conformance", true, ...); // ... other pdfua-only options CommandLine cmd = parser.parse(options, args); Config config = new Config(); CLIOptions.applyAllTo(config, cmd); applyPdfuaDefaults(config); // pdfua's hybrid defaults ``` ### pdfua's hybrid defaults pdfua currently hardcodes different defaults from core: `hybrid=hancom-ai`, `hybrid-url=http://localhost:18008`, `hybrid-mode=full` (`Main.java:60-62`). After refactoring, these become an explicit override block: ```java private static void applyPdfuaDefaults(Config config) { if (Config.HYBRID_OFF.equals(config.getHybrid())) { config.setHybrid(Config.HYBRID_HANCOM_AI); } if (config.getHybridConfig().getUrl() == null) { config.getHybridConfig().setUrl("http://localhost:18008"); } if (Config.HYBRID_MODE_AUTO.equals(config.getHybridConfig().getMode())) { config.getHybridConfig().setMode(Config.HYBRID_MODE_FULL); } } ``` This makes pdfua's deviation from core defaults explicit and grep-able. User-supplied flags still win β€” the override only fires when the user did not specify a value. ### RemediationConfig refactoring (Hard break) Current state: 3 flat hybrid fields (`hybrid`, `hybridUrl`, `hybridMode`) and 9 constructor overloads (`RemediationConfig.java:42-105`). Target state: embed core `Config` directly. Keep only pdfua-specific fields. Builder replaces the constructor overloads. ```java public class RemediationConfig { private final Config coreConfig; // ← core options live here // pdfua-only fields private final String input, output, lang; private final AuditBundleMode auditBundleMode; private final FontEmbedMode fontEmbedMode; private final List conformances; private final int threads; private final boolean enrichPictureDescription; private RemediationConfig(Builder b) { ... } public Config getCoreConfig() { return coreConfig; } public String getHybrid() { return coreConfig.getHybrid(); } public HybridConfig getHybridConfig() { return coreConfig.getHybridConfig(); } // ... pdfua-only getters public static Builder builder() { return new Builder(); } public static class Builder { ... } } ``` `getHybridUrl()` and `getHybridMode()` are **removed** (Hard break). All call sites move to `getHybridConfig().getUrl()` / `getHybridConfig().getMode()`. Verified call sites are confined to test code (`AuditBundleEmitterTest`, `CertificateIssuerTest`, `AuditManifestBuilderTest`, `RemediationConfigAuditBundleTest`) and `RemediationProcessor.java:165-170`. No external SDK consumers identified. `RemediationProcessor.java:163-170` simplifies to: ```java Config config = remediationConfig.getCoreConfig(); // (no per-field copy needed β€” Config already carries all hybrid state) ``` ### Files touched **opendataloader-pdf (core)** - `java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java` - Add 5 option constants (long names + descriptions) - Add 5 entries to `OPTION_DEFINITIONS` (3 exported, 2 not) - Extend `applyHybridOptions()`: parse, set on `HybridConfig`, validate hancom-ai gate - Add public `addAllTo(Options)` and `applyAllTo(Config, CommandLine)` - `defineOptions()` and `createConfigFromCommandLine()` delegate to the new methods - `java/opendataloader-pdf-cli/src/test/java/org/opendataloader/pdf/cli/CLIOptionsTest.java` - Tests for parsing each new option - Tests for the hancom-ai gate (using flag without `--hybrid=hancom-ai` throws) - After core changes: run `npm run sync` to regenerate `options.json` + Python/Node bindings **opendataloader-pdfua** - `src/main/java/org/opendataloader/pdf/Main.java` - Replace self-defined hybrid options with `CLIOptions.addAllTo(options)` - Add `applyPdfuaDefaults(config)` helper - Pass `coreConfig` to `RemediationConfig.Builder` instead of individual hybrid strings - `src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java` - Embed `Config coreConfig`, drop `hybrid/hybridUrl/hybridMode` fields - Replace 9 constructors with one `Builder` - Update getters: `getHybrid()` delegates, `getHybridUrl()`/`getHybridMode()` removed - `src/main/java/org/opendataloader/pdf/remediation/RemediationProcessor.java` - Simplify the lines 163-170 hybrid forwarding block - All test files instantiating `RemediationConfig` (5 files identified): switch to `Builder` ### Build & verification Per `opendataloader-pdfua/CLAUDE.md`: 1. Core changes first: `cd opendataloader-pdf/java && mvn install -DskipTests` 2. Then pdfua: `cd opendataloader-pdfua && mvn clean package` 3. Run both test suites 4. Manually verify: - `opendataloader-pdf input.pdf --hybrid=hancom-ai --hybrid-hancom-ai-regionlist-strategy=list-only` works - `opendataloader-pdf input.pdf --hybrid-hancom-ai-regionlist-strategy=list-only` (no `--hybrid`) fails with clear error - `opendataloader-pdfua input.pdf --hybrid-hancom-ai-ocr-strategy=force` works (inherits core option, pdfua defaults still apply) ### Phasing The work splits cleanly into independent commits/PRs: 1. **PR 1 (core)**: Add 5 options + gate validation + tests + `npm run sync`. Self-contained, mergeable alone. 2. **PR 2 (core)**: Extract `addAllTo` / `applyAllTo` public API. Pure refactoring, no behavior change. 3. **PR 3 (pdfua)**: Switch `Main.java` to use core's API; refactor `RemediationConfig` to Builder + embedded `Config`; update tests. PR 1 unblocks immediate user value. PRs 2 & 3 are coupled (PR 3 depends on PR 2 merge β†’ core artifact rebuild) but neither blocks PR 1. ## Out of scope - Migrating other parts of pdfua to a Builder-style config (only `RemediationConfig` is touched). - Adding flags to docling-fast or other hybrid backends. Spec is hancom-ai-specific. - Changing `HybridConfig` setter validation (already strict). - Renaming or repackaging existing `--hybrid-*` options. - Documentation site updates β€” `opendataloader.org` reference docs are CI-generated from `options.json` after core merge. --- ## File: docs/superpowers/plans/2026-03-16-cid-font-detection.md # CID Font Extraction Failure Detection β€” Implementation Plan > **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Detect pages where CID font extraction failed (high U+FFFD ratio), emit warning logs, and auto-route to OCR backend in hybrid mode. **Architecture:** Measure replacement character ratio in ContentFilterProcessor before replacement, store in StaticLayoutContainers, consume in TriageProcessor as highest-priority signal. Warning log fires regardless of hybrid mode. **Tech Stack:** Java 11+, JUnit Jupiter, veraPDF API (`ChunkParser.REPLACEMENT_CHARACTER_STRING`) --- ## File Structure | File | Responsibility | |---|---| | `TextProcessor.java` | New `measureReplacementCharRatio()` method | | `StaticLayoutContainers.java` | Per-page replacement ratio storage | | `ContentFilterProcessor.java` | Measure + warn + store before replacement | | `TriageProcessor.java` | Signal 0: route high-ratio pages to BACKEND | | `TextProcessorTest.java` | Unit tests for measurement | | `TriageProcessorTest.java` | Unit tests for Signal 0 routing | | `CidFontDetectionTest.java` (new) | e2e test with synthetic PDF | | `test/resources/cid-font-no-tounicode.pdf` (new) | Test fixture | | `test/resources/generate-cid-test-pdf.py` (new) | Generation script (reference) | All paths below are relative to `java/opendataloader-pdf-core/src/`. --- ## Chunk 1: Measurement + Storage ### Task 1: Add per-page ratio storage to StaticLayoutContainers **Files:** - Modify: `main/java/org/opendataloader/pdf/containers/StaticLayoutContainers.java` - [ ] **Step 1: Add ThreadLocal map field and imports** Add after line 40 (`imageFormat` field): ```java private static final ThreadLocal replacementCharRatios = ThreadLocal.withInitial(HashMap::new); ``` Add to imports: ```java import java.util.HashMap; import java.util.Map; ``` - [ ] **Step 2: Add getter and setter** Add after `setImageFormat()` (after line 145): ```java public static void setReplacementCharRatio(int pageNumber, double ratio) { replacementCharRatios.get().put(pageNumber, ratio); } public static double getReplacementCharRatio(int pageNumber) { return replacementCharRatios.get().getOrDefault(pageNumber, 0.0); } ``` - [ ] **Step 3: Clear in clearContainers()** Add inside `clearContainers()` method, after line 51 (`imageFormat.set(...)`): ```java replacementCharRatios.get().clear(); ``` - [ ] **Step 4: Compile check** Run: `cd java && mvn compile -pl opendataloader-pdf-core -q` Expected: BUILD SUCCESS - [ ] **Step 5: Commit** ```bash git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/containers/StaticLayoutContainers.java git commit -m "feat: add per-page replacement char ratio storage to StaticLayoutContainers" ``` ### Task 2: Add measureReplacementCharRatio to TextProcessor **Files:** - Modify: `main/java/org/opendataloader/pdf/processors/TextProcessor.java` - Test: `test/java/org/opendataloader/pdf/processors/TextProcessorTest.java` - [ ] **Step 1: Write failing tests** Add to `TextProcessorTest.java` after the last test method (before closing `}`): ``` /* Detailed source-code truncated for AI context efficiency. */ ``` - [ ] **Step 2: Run tests to verify they fail** Run: `cd java && mvn test -pl opendataloader-pdf-core -Dtest=TextProcessorTest#testMeasureReplacementCharRatio_allReplacement -q` Expected: FAIL β€” `measureReplacementCharRatio` method not found - [ ] **Step 3: Implement measureReplacementCharRatio** Add to `TextProcessor.java` after `replaceUndefinedCharacters()` method (after line 53): ```java public static double measureReplacementCharRatio(List contents) { char replacementChar = ChunkParser.REPLACEMENT_CHARACTER_STRING.charAt(0); int totalChars = 0; int replacementChars = 0; for (IObject object : contents) { if (object instanceof TextChunk) { String value = ((TextChunk) object).getValue(); totalChars += value.length(); for (int i = 0; i < value.length(); i++) { if (value.charAt(i) == replacementChar) { replacementChars++; } } } } if (totalChars == 0) { return 0.0; } return (double) replacementChars / totalChars; } ``` - [ ] **Step 4: Run tests to verify they pass** Run: `cd java && mvn test -pl opendataloader-pdf-core -Dtest=TextProcessorTest -q` Expected: All tests PASS - [ ] **Step 5: Commit** ```bash git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/TextProcessor.java git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/TextProcessorTest.java git commit -m "feat: add measureReplacementCharRatio to TextProcessor Counts U+FFFD replacement characters across TextChunks and returns the ratio. Returns 0.0 for empty contents or pages with no text." ``` --- ## Chunk 2: Warning Log + Triage Routing ### Task 3: Add warning log in ContentFilterProcessor **Files:** - Modify: `main/java/org/opendataloader/pdf/processors/ContentFilterProcessor.java` - [ ] **Step 1: Add import for StaticLayoutContainers** Add to imports: ```java import org.opendataloader.pdf.containers.StaticLayoutContainers; ``` - [ ] **Step 2: Add measurement + warning before replaceUndefinedCharacters** Insert immediately before line 74 (`TextProcessor.replaceUndefinedCharacters(...)`) in `getFilteredContents()`: ```java double replacementCharRatio = TextProcessor.measureReplacementCharRatio(pageContents); StaticLayoutContainers.setReplacementCharRatio(pageNumber, replacementCharRatio); if (replacementCharRatio >= 0.3) { LOGGER.log(Level.WARNING, "Page {0}: {1,number,#.#%} of characters are replacement characters (U+FFFD). " + "This PDF likely contains CID-keyed fonts without ToUnicode mappings. " + "Text extraction may be incomplete. Consider using --hybrid-mode for OCR fallback.", new Object[]{pageNumber + 1, replacementCharRatio}); } ``` - [ ] **Step 3: Compile check** Run: `cd java && mvn compile -pl opendataloader-pdf-core -q` Expected: BUILD SUCCESS - [ ] **Step 4: Commit** ```bash git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/processors/ContentFilterProcessor.java git commit -m "feat: detect CID font extraction failure and emit warning log Measures U+FFFD ratio before replacement. Warns when >= 30% of characters are replacement characters, suggesting hybrid mode." ``` ### Task 4: Add Signal 0 to TriageProcessor **Files:** - Modify: `main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java` - Test: `test/java/org/opendataloader/pdf/hybrid/TriageProcessorTest.java` - [ ] **Step 1: Write failing tests** Add to `TriageProcessorTest.java` before the `// Helper methods` comment: ```java @Test public void testClassifyPage_highReplacementRatio_routesToBackend() { StaticLayoutContainers.clearContainers(); StaticLayoutContainers.setReplacementCharRatio(0, 0.5); List contents = new ArrayList<>(); contents.add(createTextChunk(10, 100, 200, 120, "text")); TriageResult result = TriageProcessor.classifyPage(contents, 0, new HybridConfig()); Assertions.assertEquals(TriageDecision.BACKEND, result.getDecision()); Assertions.assertEquals(1.0, result.getConfidence(), 0.001); } @Test public void testClassifyPage_lowReplacementRatio_noEffect() { StaticLayoutContainers.clearContainers(); StaticLayoutContainers.setReplacementCharRatio(0, 0.1); List contents = new ArrayList<>(); contents.add(createTextChunk(10, 100, 200, 120, "normal text")); TriageResult result = TriageProcessor.classifyPage(contents, 0, new HybridConfig()); Assertions.assertEquals(TriageDecision.JAVA, result.getDecision()); } @Test public void testClassifyPage_exactThreshold_routesToBackend() { StaticLayoutContainers.clearContainers(); StaticLayoutContainers.setReplacementCharRatio(0, 0.3); List contents = new ArrayList<>(); contents.add(createTextChunk(10, 100, 200, 120, "text")); TriageResult result = TriageProcessor.classifyPage(contents, 0, new HybridConfig()); Assertions.assertEquals(TriageDecision.BACKEND, result.getDecision()); Assertions.assertEquals(1.0, result.getConfidence(), 0.001); } ``` - [ ] **Step 2: Run tests to verify they fail** Run: `cd java && mvn test -pl opendataloader-pdf-core -Dtest=TriageProcessorTest#testClassifyPage_highReplacementRatio_routesToBackend -q` Expected: FAIL β€” returns JAVA instead of BACKEND - [ ] **Step 3: Add Signal 0 to classifyPage** In `TriageProcessor.java`, in the `classifyPage()` method with `TriageThresholds` parameter, insert before the TableBorder check (before `// Signal 1: TableBorder presence`): ```java // Signal 0: CID font extraction failure (highest priority) // Only fires in hybrid mode (classifyPage is only called from HybridDocumentProcessor) double replacementRatio = StaticLayoutContainers.getReplacementCharRatio(pageNumber); if (replacementRatio >= 0.3) { return TriageResult.backend(pageNumber, 1.0, signals); } ``` Add import at top of file: ```java import org.opendataloader.pdf.containers.StaticLayoutContainers; ``` Also update the `classifyPage()` Javadoc (around line 617) to list Signal 0: ``` * Signal priority (highest to lowest): *
    *
  1. CID font extraction failure (replacement char ratio >= 30%)
    2. *
  2. TableBorder presence
    3. * ... ``` - [ ] **Step 4: Run tests to verify they pass** Run: `cd java && mvn test -pl opendataloader-pdf-core -Dtest=TriageProcessorTest -q` Expected: All tests PASS - [ ] **Step 5: Run full test suite** Run: `cd java && mvn test -pl opendataloader-pdf-core -q` Expected: All tests PASS β€” no regressions - [ ] **Step 6: Commit** ```bash git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/TriageProcessorTest.java git commit -m "feat: add CID font detection signal to TriageProcessor Signal 0 (highest priority): routes pages with >= 30% replacement characters to backend for OCR fallback in hybrid mode." ``` --- ## Chunk 3: e2e Test + Test Fixture ### Task 5: Generate synthetic CID PDF test fixture **Files:** - Create: `java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py` - Create: `java/opendataloader-pdf-core/src/test/resources/cid-font-no-tounicode.pdf` - [ ] **Step 1: Create test resources directory** ```bash mkdir -p java/opendataloader-pdf-core/src/test/resources ``` - [ ] **Step 2: Write the generation script** Create `java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py`. This script generates a minimal PDF with a Type0 (CID) font that has no ToUnicode CMap. The PDF must cause veraPDF to emit `\uFFFD` for the majority of text characters when parsed. Approach: Use raw PDF syntax to embed a CID font referencing CID values without a ToUnicode mapping. Alternatively, use `reportlab` which provides low-level CID font control. The implementer should: 1. Generate the PDF 2. Verify it with opendataloader-pdf: `cd java && mvn exec:java -pl opendataloader-pdf-cli -Dexec.mainClass="org.opendataloader.pdf.cli.CLIMain" -Dexec.args="../src/test/resources/cid-font-no-tounicode.pdf -f text" 2>&1` 3. Confirm output is mostly empty/whitespace (indicating `\uFFFD` β†’ space replacement) 4. Confirm WARNING log about replacement characters appears If generating a proper CID PDF proves difficult, search `odl-test-fixtures` for an existing PDF with CID font issues, or use a known CID-problematic PDF from the Korean pharmaceutical fixtures (pdf-003 through pdf-007). - [ ] **Step 3: Commit fixture** ```bash git add java/opendataloader-pdf-core/src/test/resources/cid-font-no-tounicode.pdf git add java/opendataloader-pdf-core/src/test/resources/generate-cid-test-pdf.py git commit -m "test: add synthetic CID font PDF test fixture PDF with CID-keyed font without ToUnicode mapping for testing replacement character detection. Generation script included." ``` ### Task 6: Write e2e integration test **Files:** - Create: `test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java` This test follows the pattern from `TriageProcessorIntegrationTest.java`: `DocumentProcessor.preprocessing()` β†’ `ContentFilterProcessor.getFilteredContents()` β†’ assert ratio and warning. - [ ] **Step 1: Write the integration test** Create `java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java`: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` - [ ] **Step 2: Run tests** Run: `cd java && mvn test -pl opendataloader-pdf-core -Dtest=CidFontDetectionTest -q` Expected: Integration tests pass (or skip if fixture not yet generated). Boundary tests always pass. - [ ] **Step 3: Run full test suite** Run: `cd java && mvn test -pl opendataloader-pdf-core -q` Expected: All tests PASS - [ ] **Step 4: Commit** ```bash git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/processors/CidFontDetectionTest.java git commit -m "test: add integration + boundary tests for CID font detection Integration tests load cid-font-no-tounicode.pdf through full pipeline. Boundary tests verify 29%/30% threshold behavior." ``` --- ## Chunk 4: Final Verification ### Task 7: Full test suite + benchmark regression check - [ ] **Step 1: Run full Java test suite** Run: `cd java && mvn test -q` Expected: All tests PASS across both core and cli modules - [ ] **Step 2: Check benchmark regression (if benchmark script exists)** Run: `./scripts/bench.sh --check-regression 2>/dev/null || echo "No benchmark script β€” skip"` Expected: PASS or skip - [ ] **Step 3: Review all changes** Run: `git log --oneline main..HEAD` Expected commits (newest first): 1. `test: add e2e tests for CID font detection pipeline` 2. `test: add synthetic CID font PDF test fixture` 3. `feat: add CID font detection signal to TriageProcessor` 4. `feat: detect CID font extraction failure and emit warning log` 5. `feat: add measureReplacementCharRatio to TextProcessor` 6. `feat: add per-page replacement char ratio storage to StaticLayoutContainers` - [ ] **Step 4: Update issue #286 comment** Update the previously posted enhancement comment with the actual fix status. The earlier `enhancement` label should be removed since this is now a fix. ```bash gh issue edit 286 --repo opendataloader-project/opendataloader-pdf --remove-label enhancement ``` --- ## File: docs/superpowers/plans/2026-04-18-hancom-ai-mock-server.md # Hancom AI Mock Server Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Hancom AI μ„œλ²„ λΆ€μž¬ μ‹œ transformer 개발/디버그 사이클을 μœ μ§€ν•˜κΈ° μœ„ν•œ fixture-replay HTTP mock μ„œλ²„ κ΅¬ν˜„ + ν΄λΌμ΄μ–ΈνŠΈ(`HancomAIClient.java`) REQUEST_ID κ·œμ•½ 패치. **Architecture:** Python(FastAPI) 둜컬 μ„œλ²„κ°€ 200개 벀치 PDF의 SHA256을 μΈλ±μ‹±ν•˜κ³ , PDF μž…λ ₯은 SHA256 룩업, 이미지 μž…λ ₯은 ν΄λΌμ΄μ–ΈνŠΈκ°€ REQUEST_ID에 μΈμ½”λ”©ν•œ `(sha_short, page, obj, module)`둜 룩업, pdf2imgλŠ” PyMuPDF둜 동적 300DPI λ Œλ”λ§ + base64 응닡. ν΄λΌμ΄μ–ΈνŠΈλŠ” `convert()` μ§„μž… μ‹œ PDF SHA256을 μΊμ‹œν•˜κ³  λͺ¨λ“  ν˜ΈμΆœμ— REQUEST_ID κ·œμ•½ 적용. **Tech Stack:** - Mock μ„œλ²„: Python 3.10+, FastAPI, uvicorn, PyMuPDF, pytest, httpx (TestClient) - Client 패치: Java 8+, OkHttp, Jackson, Maven, JUnit 5 - 데이터: `bundolee/kb-odl/raw/4-기술/2026-04-16_Q2-기술-ctx_hancom-ai-a11y_좜λ ₯데이터-μŠ€ν‚€λ§ˆ/` - PDFs: `opendataloader-project/opendataloader-bench/pdfs/` **Spec:** `bundolee/kb-odl/raw/4-기술/2026-04-18_Q2-DEV-02-Code_hancom-ai-mock-server-design.md` (commit `dfe0671`) --- ## File Structure ### Mock μ„œλ²„ (μ‹ κ·œ) `opendataloader-project/opendataloader-pdfua/scripts/mock_server/` | 파일 | μ±…μž„ | |---|---| | `mock_server/__init__.py` | 빈 νŒ¨ν‚€μ§€ 마컀 | | `mock_server/__main__.py` | CLI μ—”νŠΈλ¦¬ν¬μΈνŠΈ (`python -m mock_server ...`) | | `mock_server/index.py` | λΆ€νŒ… μ‹œ `--pdf-dir` μŠ€μΊ” β†’ `{sha256: (basename, path)}` 인덱슀 | | `mock_server/request_id.py` | REQUEST_ID μ •κ·œμ‹ νŒŒμ‹± (`odl-{sha12}-p{n}-o{n}-{module}`) | | `mock_server/lookup.py` | λͺ¨λ“ˆλͺ…/μž…λ ₯νƒ€μž…μ— λ”°λ₯Έ fixture 경둜 κ²°μ • + JSON λ‘œλ“œ. `FixtureMiss` μ˜ˆμ™Έ. | | `mock_server/pdf_render.py` | PyMuPDF둜 PDF νŽ˜μ΄μ§€ β†’ PNG 300DPI bytes | | `mock_server/server.py` | FastAPI app: `/ping`, `/hocr/sdk`, `/support/pdf2img` (각각 `/api/v1/` λ³€ν˜• 별칭) | | `mock_server/tests/__init__.py` | | | `mock_server/tests/conftest.py` | ν…ŒμŠ€νŠΈ fixture (μƒ˜ν”Œ PDF dir, μƒ˜ν”Œ fixture dir) | | `mock_server/tests/test_index.py` | | | `mock_server/tests/test_request_id.py` | | | `mock_server/tests/test_lookup.py` | | | `mock_server/tests/test_pdf_render.py` | | | `mock_server/tests/test_server.py` | | | `mock_server/pyproject.toml` | μ˜μ‘΄μ„± + pytest μ„€μ • | | `mock_server/README.md` | 싀행법 + 디버깅 + client 패치 λ©”λͺ¨ | ### Client 패치 (μˆ˜μ •) `opendataloader-project/opendataloader-pdf/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/HancomAIClient.java` `opendataloader-project/opendataloader-pdf/java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/HancomAIClientRequestIdTest.java` (μ‹ κ·œ) --- ## Phase 1 β€” Mock Server (10 tasks) ### Task 1: ν”„λ‘œμ νŠΈ μŠ€μΊν΄λ”© **Files:** - Create: `opendataloader-project/opendataloader-pdfua/scripts/mock_server/pyproject.toml` - Create: `opendataloader-project/opendataloader-pdfua/scripts/mock_server/__init__.py` - Create: `opendataloader-project/opendataloader-pdfua/scripts/mock_server/tests/__init__.py` - Create: `opendataloader-project/opendataloader-pdfua/scripts/mock_server/README.md` - [ ] **Step 1: pyproject.toml μž‘μ„±** ```toml [project] name = "hancom-ai-mock-server" version = "0.1.0" requires-python = ">=3.10" dependencies = [ "fastapi>=0.110", "uvicorn>=0.27", "pymupdf>=1.24", "python-multipart>=0.0.9", ] [project.optional-dependencies] dev = [ "pytest>=8.0", "httpx>=0.27", ] [tool.pytest.ini_options] testpaths = ["tests"] ``` - [ ] **Step 2: 빈 νŒ¨ν‚€μ§€ 마컀 + README μ΄ˆμ•ˆ** `mock_server/__init__.py`: ```python """Hancom AI HOCR SDK fixture-replay mock server.""" ``` `mock_server/tests/__init__.py`: ```python ``` `mock_server/README.md`: ```markdown # Hancom AI Mock Server Fixture-replay mock for `HancomAIClient`. See spec at `bundolee/kb-odl/raw/4-기술/2026-04-18_Q2-DEV-02-Code_hancom-ai-mock-server-design.md`. ## Run ```bash cd opendataloader-pdfua/scripts/mock_server pip install -e ".[dev]" python -m mock_server \ --pdf-dir /path/to/opendataloader-bench/pdfs \ --fixture-dir /path/to/kb-odl/raw/4-기술/2026-04-16_Q2-기술-ctx_hancom-ai-a11y_좜λ ₯데이터-μŠ€ν‚€λ§ˆ \ --port 18008 ``` ## Tests ```bash pytest -v ``` ``` - [ ] **Step 3: 디렉토리 ꡬ쑰 확인** Run: `ls opendataloader-project/opendataloader-pdfua/scripts/mock_server/` Expected: `pyproject.toml README.md __init__.py tests/` - [ ] **Step 4: Commit** ```bash cd opendataloader-project/opendataloader-pdfua git add scripts/mock_server/ git commit -m "feat(mock-server): scaffold project structure" ``` --- ### Task 2: REQUEST_ID νŒŒμ„œ (TDD) **Files:** - Test: `opendataloader-project/opendataloader-pdfua/scripts/mock_server/tests/test_request_id.py` - Create: `opendataloader-project/opendataloader-pdfua/scripts/mock_server/mock_server/request_id.py` - [ ] **Step 1: μ‹€νŒ¨ν•˜λŠ” ν…ŒμŠ€νŠΈ μž‘μ„±** `tests/test_request_id.py`: ```python import pytest from mock_server.request_id import parse_request_id, RequestIdParts def test_parse_caption(): parts = parse_request_id("odl-a3f1c9d2e7b8-p0-o5-caption") assert parts == RequestIdParts(sha_short="a3f1c9d2e7b8", page=0, obj=5, module="caption") def test_parse_chart(): parts = parse_request_id("odl-deadbeef0000-p3-o12-chart") assert parts == RequestIdParts(sha_short="deadbeef0000", page=3, obj=12, module="chart") def test_parse_tsr(): parts = parse_request_id("odl-aabbccddeeff-p0-o0-tsr") assert parts.module == "tsr" def test_parse_invalid_returns_none(): assert parse_request_id("odl-DOCUMENT_LAYOUT_WITH_OCR") is None assert parse_request_id("garbage") is None assert parse_request_id("") is None assert parse_request_id("odl-a3f1-p0-o0-unknown") is None # bad module def test_parse_pdf_module_request_id_returns_none(): # PDF modules use REQUEST_ID for tracing only; mock matches via SHA256 of bytes. # Parser returns None for these so caller falls back to PDF lookup path. assert parse_request_id("odl-a3f1c9d2e7b8-dla-ocr") is None ``` - [ ] **Step 2: ν…ŒμŠ€νŠΈ μ‹€ν–‰ (μ‹€νŒ¨ 확인)** Run: `cd opendataloader-project/opendataloader-pdfua/scripts/mock_server && pytest tests/test_request_id.py -v` Expected: FAIL β€” `ModuleNotFoundError: No module named 'mock_server.request_id'` - [ ] **Step 3: μ΅œμ†Œ κ΅¬ν˜„** `mock_server/request_id.py`: ```python """Parse client REQUEST_ID for image-input module lookup.""" import re from dataclasses import dataclass from typing import Optional _PATTERN = re.compile( r"^odl-(?P[0-9a-f]{12})-p(?P\d+)-o(?P\d+)-(?Pcaption|chart|tsr)$" ) @dataclass(frozen=True) class RequestIdParts: sha_short: str page: int obj: int module: str def parse_request_id(request_id: str) -> Optional[RequestIdParts]: if not request_id: return None m = _PATTERN.match(request_id) if not m: return None return RequestIdParts( sha_short=m.group("sha"), page=int(m.group("page")), obj=int(m.group("obj")), module=m.group("module"), ) ``` - [ ] **Step 4: ν…ŒμŠ€νŠΈ 톡과 확인** Run: `pytest tests/test_request_id.py -v` Expected: 5 passed - [ ] **Step 5: Commit** ```bash git add scripts/mock_server/mock_server/request_id.py scripts/mock_server/tests/test_request_id.py git commit -m "feat(mock-server): REQUEST_ID parser for image module lookup" ``` --- ### Task 3: SHA256 인덱슀 λΉŒλ” (TDD) **Files:** - Test: `scripts/mock_server/tests/test_index.py` - Test: `scripts/mock_server/tests/conftest.py` - Create: `scripts/mock_server/mock_server/index.py` - [ ] **Step 1: conftest fixture μž‘μ„±** `tests/conftest.py`: ```python import hashlib import shutil from pathlib import Path import pytest @pytest.fixture def sample_pdf_dir(tmp_path): """Create a tiny PDF dir with 3 fake PDFs (just bytes, not parsed by index).""" d = tmp_path / "pdfs" d.mkdir() for i in range(1, 4): (d / f"pdf{i:03d}.pdf").write_bytes(f"FAKE_PDF_CONTENT_{i}".encode()) return d def sha256_hex(b: bytes) -> str: return hashlib.sha256(b).hexdigest() ``` - [ ] **Step 2: μ‹€νŒ¨ν•˜λŠ” ν…ŒμŠ€νŠΈ μž‘μ„±** `tests/test_index.py`: ```python from mock_server.index import build_pdf_index, PdfIndexEntry from .conftest import sha256_hex def test_build_index_three_pdfs(sample_pdf_dir): index = build_pdf_index(sample_pdf_dir) assert len(index) == 3 expected_sha = sha256_hex(b"FAKE_PDF_CONTENT_1") assert expected_sha in index entry = index[expected_sha] assert entry.basename == "pdf001" assert entry.path == sample_pdf_dir / "pdf001.pdf" def test_build_index_empty_dir(tmp_path): index = build_pdf_index(tmp_path) assert index == {} def test_build_index_sha_short_lookup(sample_pdf_dir): index = build_pdf_index(sample_pdf_dir) full_sha = next(iter(index)) short = full_sha[:12] entry = index.find_by_short(short) assert entry is not None assert entry.basename in {"pdf001", "pdf002", "pdf003"} ``` - [ ] **Step 3: ν…ŒμŠ€νŠΈ μ‹€ν–‰ (μ‹€νŒ¨ 확인)** Run: `pytest tests/test_index.py -v` Expected: FAIL β€” module not found - [ ] **Step 4: μ΅œμ†Œ κ΅¬ν˜„** `mock_server/index.py`: ```python """Build SHA256 index of benchmark PDFs at server boot.""" import hashlib from dataclasses import dataclass from pathlib import Path from typing import Dict, Optional @dataclass(frozen=True) class PdfIndexEntry: basename: str path: Path class PdfIndex(dict): """Dict {full_sha256: PdfIndexEntry} with helper for short-prefix lookup.""" def find_by_short(self, sha_short: str) -> Optional[PdfIndexEntry]: for full_sha, entry in self.items(): if full_sha.startswith(sha_short): return entry return None def build_pdf_index(pdf_dir: Path) -> PdfIndex: index = PdfIndex() for path in sorted(Path(pdf_dir).glob("*.pdf")): sha = hashlib.sha256(path.read_bytes()).hexdigest() index[sha] = PdfIndexEntry(basename=path.stem, path=path) return index ``` - [ ] **Step 5: ν…ŒμŠ€νŠΈ 톡과 확인** Run: `pytest tests/test_index.py -v` Expected: 3 passed - [ ] **Step 6: Commit** ```bash git add scripts/mock_server/ git commit -m "feat(mock-server): SHA256 PDF index with short-prefix lookup" ``` --- ### Task 4: Lookup 둜직 (TDD) **Files:** - Test: `scripts/mock_server/tests/test_lookup.py` - Test: `scripts/mock_server/tests/conftest.py` (ν™•μž₯) - Create: `scripts/mock_server/mock_server/lookup.py` - [ ] **Step 1: conftest ν™•μž₯ β€” κ°€μ§œ fixture dir** `tests/conftest.py` 끝에 μΆ”κ°€: ```python import json @pytest.fixture def sample_fixture_dir(tmp_path): """Mimic the recorded data layout with one entry per module.""" root = tmp_path / "fixtures" for sub in ["DLA_OCR", "DLA", "OCR", "TSR", "TSR_regionlist", "FIGURE"]: (root / sub).mkdir(parents=True) (root / "DLA_OCR" / "pdf001.json").write_text(json.dumps({"module": "DLA_OCR", "id": "pdf001"})) (root / "DLA" / "pdf001.json").write_text(json.dumps({"module": "DLA"})) (root / "OCR" / "pdf001.json").write_text(json.dumps({"module": "OCR"})) (root / "TSR" / "pdf002_p0_o0.json").write_text(json.dumps({"module": "TSR", "table": True})) (root / "TSR_regionlist" / "pdf003_p0_o0.json").write_text(json.dumps({"module": "TSR_regionlist"})) (root / "FIGURE" / "pdf001_p0_o5_caption.json").write_text(json.dumps({"caption": "test"})) (root / "FIGURE" / "pdf001_p0_o5_chart.json").write_text(json.dumps({"understanding": "TITLE | <0x0A> 1 | 2"})) return root ``` - [ ] **Step 2: μ‹€νŒ¨ ν…ŒμŠ€νŠΈ μž‘μ„±** `tests/test_lookup.py`: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` - [ ] **Step 3: μ‹€νŒ¨ 확인** Run: `pytest tests/test_lookup.py -v` Expected: FAIL β€” module not found - [ ] **Step 4: κ΅¬ν˜„** `mock_server/lookup.py`: ```python """Map (module, key) β†’ fixture JSON path and load. Raises FixtureMiss on miss.""" import json from pathlib import Path from typing import Any class FixtureMiss(Exception): """Raised when no fixture file matches the request.""" MODULE_TO_DIR = { "DOCUMENT_LAYOUT_WITH_OCR": "DLA_OCR", "DOCUMENT_LAYOUT_ANALYSIS": "DLA", "TEXT_RECOGNITION": "OCR", "TABLE_STRUCTURE_RECOGNITION": "TSR", } def lookup_pdf_module(fixture_dir: Path, module_name: str, basename: str) -> Any: sub = MODULE_TO_DIR.get(module_name) if sub is None: raise FixtureMiss(f"unknown PDF-input module: {module_name}") path = Path(fixture_dir) / sub / f"{basename}.json" if not path.exists(): raise FixtureMiss(f"no fixture at {sub}/{basename}.json") return json.loads(path.read_text()) def lookup_image_module( fixture_dir: Path, module_short: str, basename: str, page: int, obj: int ) -> Any: if module_short in {"caption", "chart"}: path = Path(fixture_dir) / "FIGURE" / f"{basename}_p{page}_o{obj}_{module_short}.json" if not path.exists(): raise FixtureMiss(f"no FIGURE fixture: {path.name}") return json.loads(path.read_text()) if module_short == "tsr": primary = Path(fixture_dir) / "TSR" / f"{basename}_p{page}_o{obj}.json" fallback = Path(fixture_dir) / "TSR_regionlist" / f"{basename}_p{page}_o{obj}.json" for candidate in (primary, fallback): if candidate.exists(): return json.loads(candidate.read_text()) raise FixtureMiss(f"no TSR fixture: {basename}_p{page}_o{obj}.json (TSR/ or TSR_regionlist/)") raise FixtureMiss(f"unknown image module: {module_short}") ``` - [ ] **Step 5: ν…ŒμŠ€νŠΈ 톡과 확인** Run: `pytest tests/test_lookup.py -v` Expected: 10 passed - [ ] **Step 6: Commit** ```bash git add scripts/mock_server/ git commit -m "feat(mock-server): lookup with TSR_regionlist fallback" ``` --- ### Task 5: PDF Renderer (TDD) **Files:** - Test: `scripts/mock_server/tests/test_pdf_render.py` - Create: `scripts/mock_server/mock_server/pdf_render.py` - [ ] **Step 1: μ‹€νŒ¨ ν…ŒμŠ€νŠΈ** `tests/test_pdf_render.py`: ```python import io import pytest import fitz # PyMuPDF from mock_server.pdf_render import render_page_png @pytest.fixture def real_pdf(tmp_path): """Create a minimal 2-page PDF using PyMuPDF itself.""" doc = fitz.open() doc.new_page(width=612, height=792) # 1 page doc.new_page(width=612, height=792) # 2 pages p = tmp_path / "minimal.pdf" doc.save(p) doc.close() return p def test_render_first_page_returns_png_bytes(real_pdf): png_bytes = render_page_png(real_pdf, page_index=0, dpi=300) assert png_bytes[:8] == b"\x89PNG\r\n\x1a\n" def test_render_dimensions_300dpi(real_pdf): from PIL import Image png_bytes = render_page_png(real_pdf, page_index=0, dpi=300) img = Image.open(io.BytesIO(png_bytes)) # 612pt at 300dpi / 72pt-per-inch = 2550px (Β±1 for rounding) assert abs(img.width - 2550) <= 2 assert abs(img.height - 3300) <= 2 def test_render_out_of_range_raises(real_pdf): with pytest.raises(IndexError): render_page_png(real_pdf, page_index=99) ``` Note: Pillow is a transitive dep of PyMuPDF; if not, add to dev deps. - [ ] **Step 2: μ‹€νŒ¨ 확인** Run: `pytest tests/test_pdf_render.py -v` Expected: FAIL β€” module not found - [ ] **Step 3: κ΅¬ν˜„** `mock_server/pdf_render.py`: ```python """Render a PDF page to PNG bytes using PyMuPDF.""" from pathlib import Path import fitz def render_page_png(pdf_path: Path, page_index: int, dpi: int = 300) -> bytes: doc = fitz.open(str(pdf_path)) try: if page_index < 0 or page_index >= doc.page_count: raise IndexError(f"page_index {page_index} out of range (0..{doc.page_count - 1})") page = doc.load_page(page_index) zoom = dpi / 72.0 matrix = fitz.Matrix(zoom, zoom) pixmap = page.get_pixmap(matrix=matrix, alpha=False) return pixmap.tobytes("png") finally: doc.close() ``` - [ ] **Step 4: pyproject.toml dev deps에 Pillow μΆ”κ°€ (ν…ŒμŠ€νŠΈμš©)** Edit `scripts/mock_server/pyproject.toml`, replace `[project.optional-dependencies]` block with: ```toml [project.optional-dependencies] dev = [ "pytest>=8.0", "httpx>=0.27", "pillow>=10.0", ] ``` Reinstall: `pip install -e ".[dev]"` - [ ] **Step 5: ν…ŒμŠ€νŠΈ 톡과 확인** Run: `pytest tests/test_pdf_render.py -v` Expected: 3 passed - [ ] **Step 6: Commit** ```bash git add scripts/mock_server/ git commit -m "feat(mock-server): PyMuPDF page renderer at 300dpi" ``` --- ### Task 6: FastAPI Server β€” /ping + /hocr/sdk PDF λΆ„κΈ° (TDD) **Files:** - Test: `scripts/mock_server/tests/test_server.py` - Create: `scripts/mock_server/mock_server/server.py` - [ ] **Step 1: μ‹€νŒ¨ ν…ŒμŠ€νŠΈ (ping + DLA_OCR happy path + unknown PDF)** `tests/test_server.py`: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` - [ ] **Step 2: μ‹€νŒ¨ 확인** Run: `pytest tests/test_server.py -v` Expected: FAIL β€” module not found - [ ] **Step 3: κ΅¬ν˜„** `mock_server/server.py`: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` - [ ] **Step 4: ν…ŒμŠ€νŠΈ 톡과 확인** Run: `pytest tests/test_server.py -v` Expected: 5 passed (ping x2, DLA_OCR happy x2, unknown PDF) - [ ] **Step 5: Commit** ```bash git add scripts/mock_server/ git commit -m "feat(mock-server): /ping + /hocr/sdk PDF branch with /api/v1 alias" ``` --- ### Task 7: 이미지 λΆ„κΈ° μΆ”κ°€ (TDD) **Files:** - Modify: `scripts/mock_server/mock_server/server.py` - Modify: `scripts/mock_server/tests/test_server.py` - [ ] **Step 1: μΆ”κ°€ ν…ŒμŠ€νŠΈ μž‘μ„±** Append to `tests/test_server.py`: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` - [ ] **Step 2: μ‹€νŒ¨ 확인** Run: `pytest tests/test_server.py -v -k image` Expected: 4 failed (image branch returns FIXTURE_MISS placeholder) - [ ] **Step 3: 이미지 λΆ„κΈ° κ΅¬ν˜„** In `mock_server/server.py`, replace the image branch placeholder line `# image branch (Task 7)` and the line below it with: ```python # image branch parts = parse_request_id(REQUEST_ID) if parts is None: hint = f"REQUEST_ID does not match odl--p-o-" LOGGER.warning("FIXTURE_MISS bad REQUEST_ID: %s", REQUEST_ID) return _envelope(REQUEST_ID, False, "FIXTURE_MISS", [], hint) entry = pdf_index.find_by_short(parts.sha_short) if entry is None: hint = f"sha_short={parts.sha_short} not in {len(pdf_index)}-PDF index" LOGGER.warning("FIXTURE_MISS unknown sha_short: %s", parts.sha_short) return _envelope(REQUEST_ID, False, "FIXTURE_MISS", [], hint) try: data = lookup_image_module( Path(fixture_dir), parts.module, entry.basename, parts.page, parts.obj ) except FixtureMiss as e: LOGGER.warning("FIXTURE_MISS image-lookup: %s", e) return _envelope(REQUEST_ID, False, "FIXTURE_MISS", [], str(e)) return _envelope(REQUEST_ID, True, "SUCCESS", data) ``` - [ ] **Step 4: ν…ŒμŠ€νŠΈ 톡과 확인** Run: `pytest tests/test_server.py -v` Expected: 9 passed (5 from Task 6 + 4 new) - [ ] **Step 5: Commit** ```bash git add scripts/mock_server/ git commit -m "feat(mock-server): image module branch with REQUEST_ID parsing" ``` --- ### Task 8: pdf2img μ—”λ“œν¬μΈνŠΈ (TDD) **Files:** - Modify: `scripts/mock_server/mock_server/server.py` - Modify: `scripts/mock_server/tests/test_server.py` - Modify: `scripts/mock_server/tests/conftest.py` (real PDF fixture) - [ ] **Step 1: conftest에 real PDF μΆ”κ°€ + μΈλ±μŠ€μ— 포함** Edit `tests/conftest.py`, replace `sample_pdf_dir` with: ```python @pytest.fixture def sample_pdf_dir(tmp_path): """Create a tiny PDF dir: 3 fake byte files + 1 real PDF for pdf2img tests.""" import fitz d = tmp_path / "pdfs" d.mkdir() for i in range(1, 4): (d / f"pdf{i:03d}.pdf").write_bytes(f"FAKE_PDF_CONTENT_{i}".encode()) # real 2-page PDF named pdf004.pdf doc = fitz.open() doc.new_page(width=612, height=792) doc.new_page(width=612, height=792) doc.save(d / "pdf004.pdf") doc.close() return d ``` - [ ] **Step 2: μ‹€νŒ¨ ν…ŒμŠ€νŠΈ μΆ”κ°€** Append to `tests/test_server.py`: ```python import base64 def test_pdf2img_returns_base64_png(client, sample_pdf_dir): pdf_bytes = (sample_pdf_dir / "pdf004.pdf").read_bytes() r = client.post( "/support/pdf2img", data={"REQUEST_ID": "odl-pdf2img-0", "PAGE_INDEX": "0"}, files={"FILE": ("document.pdf", pdf_bytes, "application/pdf")}, ) assert r.status_code == 200 body = r.json() assert body["SUCCESS"] is True png_b64 = body["RESULT"][0]["RESULT"]["PAGE_PNG_DATA"] assert png_b64 assert base64.b64decode(png_b64)[:8] == b"\x89PNG\r\n\x1a\n" def test_pdf2img_via_v1_alias(client, sample_pdf_dir): pdf_bytes = (sample_pdf_dir / "pdf004.pdf").read_bytes() r = client.post( "/api/v1/support/pdf2img", data={"REQUEST_ID": "x", "PAGE_INDEX": "1"}, files={"FILE": ("d.pdf", pdf_bytes, "application/pdf")}, ) assert r.json()["SUCCESS"] is True def test_pdf2img_unknown_pdf(client): r = client.post( "/support/pdf2img", data={"REQUEST_ID": "x", "PAGE_INDEX": "0"}, files={"FILE": ("u.pdf", b"NOT_INDEXED", "application/pdf")}, ) body = r.json() assert body["SUCCESS"] is False assert body["MSG"] == "FIXTURE_MISS" def test_pdf2img_out_of_range(client, sample_pdf_dir): pdf_bytes = (sample_pdf_dir / "pdf004.pdf").read_bytes() r = client.post( "/support/pdf2img", data={"REQUEST_ID": "x", "PAGE_INDEX": "99"}, files={"FILE": ("d.pdf", pdf_bytes, "application/pdf")}, ) body = r.json() assert body["SUCCESS"] is False assert "page_index" in body["_mock_hint"] ``` - [ ] **Step 3: μ‹€νŒ¨ 확인** Run: `pytest tests/test_server.py -v -k pdf2img` Expected: 4 failed (404 from FastAPI β€” endpoint not registered) - [ ] **Step 4: pdf2img 라우트 μΆ”κ°€** In `mock_server/server.py`, inside `create_app` after the `/hocr/sdk` route, add: ```python @router.post("/support/pdf2img") async def pdf2img( REQUEST_ID: str = Form(""), PAGE_INDEX: int = Form(...), FILE: UploadFile = File(...), ): body = await FILE.read() sha = hashlib.sha256(body).hexdigest() entry = pdf_index.get(sha) if entry is None: hint = f"sha256={sha[:12]}... not in {len(pdf_index)}-PDF index" return _envelope(REQUEST_ID, False, "FIXTURE_MISS", [], hint) try: png_bytes = render_page_png(entry.path, PAGE_INDEX, dpi=300) except IndexError as e: return _envelope(REQUEST_ID, False, "FIXTURE_MISS", [], str(e)) import base64 b64 = base64.b64encode(png_bytes).decode("ascii") return _envelope( REQUEST_ID, True, "SUCCESS", {"RESULT": {"PAGE_PNG_DATA": b64}}, ) ``` - [ ] **Step 5: ν…ŒμŠ€νŠΈ 톡과 확인** Run: `pytest tests/test_server.py -v` Expected: 13 passed - [ ] **Step 6: Commit** ```bash git add scripts/mock_server/ git commit -m "feat(mock-server): /support/pdf2img with PyMuPDF dynamic rendering" ``` --- ### Task 9: CLI μ—”νŠΈλ¦¬ν¬μΈνŠΈ **Files:** - Create: `scripts/mock_server/mock_server/__main__.py` - [ ] **Step 1: κ΅¬ν˜„** `mock_server/__main__.py`: ```python """CLI: python -m mock_server --pdf-dir ... --fixture-dir ... --port 18008""" import argparse import logging import sys from pathlib import Path import uvicorn from .server import create_app def main(argv=None): parser = argparse.ArgumentParser(prog="mock_server") parser.add_argument("--pdf-dir", required=True, type=Path) parser.add_argument("--fixture-dir", required=True, type=Path) parser.add_argument("--port", type=int, default=18008) parser.add_argument("--host", default="127.0.0.1") args = parser.parse_args(argv) logging.basicConfig( level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s", ) if not args.pdf_dir.is_dir(): print(f"--pdf-dir not found: {args.pdf_dir}", file=sys.stderr) sys.exit(2) if not args.fixture_dir.is_dir(): print(f"--fixture-dir not found: {args.fixture_dir}", file=sys.stderr) sys.exit(2) app = create_app(pdf_dir=args.pdf_dir, fixture_dir=args.fixture_dir) uvicorn.run(app, host=args.host, port=args.port, log_level="info") if __name__ == "__main__": main() ``` - [ ] **Step 2: smoke test β€” CLI 인자 검증** Run: ```bash cd opendataloader-project/opendataloader-pdfua/scripts/mock_server python -m mock_server --pdf-dir /nonexistent --fixture-dir /nonexistent ``` Expected: exit code 2, stderr `--pdf-dir not found: /nonexistent` - [ ] **Step 3: μ‹€μ œ λ°μ΄ν„°λ‘œ λΆ€νŒ… smoke test** Run (in a separate terminal or background, then kill): ```bash python -m mock_server \ --pdf-dir /Users/benedict/Workspace/opendataloader-project/opendataloader-bench/pdfs \ --fixture-dir /Users/benedict/Workspace/bundolee/kb-odl/raw/4-기술/2026-04-16_Q2-기술-ctx_hancom-ai-a11y_좜λ ₯데이터-μŠ€ν‚€λ§ˆ \ --port 18008 & sleep 2 curl -s http://127.0.0.1:18008/ping kill %1 ``` Expected: log `indexed 200 PDFs`, curl returns `{"status":"ok"}`. - [ ] **Step 4: Commit** ```bash git add scripts/mock_server/ git commit -m "feat(mock-server): CLI entrypoint with arg validation" ``` --- ### Task 10: README 마무리 + E2E 절차 기둝 **Files:** - Modify: `scripts/mock_server/README.md` - [ ] **Step 1: README ν™•μž₯** Replace `mock_server/README.md` with: ```markdown # Hancom AI Mock Server Fixture-replay mock for [HancomAIClient.java](../../../opendataloader-pdf/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/HancomAIClient.java). Spec: `bundolee/kb-odl/raw/4-기술/2026-04-18_Q2-DEV-02-Code_hancom-ai-mock-server-design.md` ## Setup ```bash cd opendataloader-pdfua/scripts/mock_server pip install -e ".[dev]" ``` ## Run ```bash python -m mock_server \ --pdf-dir /Users/benedict/Workspace/opendataloader-project/opendataloader-bench/pdfs \ --fixture-dir /Users/benedict/Workspace/bundolee/kb-odl/raw/4-기술/2026-04-16_Q2-기술-ctx_hancom-ai-a11y_좜λ ₯데이터-μŠ€ν‚€λ§ˆ \ --port 18008 ``` Expect log: `indexed 200 PDFs, listening on :18008`. ## Endpoints | Method | Path (and `/api/v1/` alias) | Purpose | |---|---|---| | GET | `/ping` | Health check (HTTP 200) | | POST | `/hocr/sdk` | All 6 modules (DATA_FORMAT distinguishes pdf vs image) | | POST | `/support/pdf2img` | 300dpi PNG, base64 encoded | ## REQUEST_ID Convention Image-input modules (TSR / IMAGE_CAPTIONING / CHART_IMAGE_UNDERSTANDING) require: ``` odl-{sha_short}-p{page}-o{obj}-{module_short} ``` - `sha_short` = first 12 hex chars of source PDF SHA256 - `module_short` ∈ {`caption`, `chart`, `tsr`} PDF-input modules (DLA_OCR / DLA / OCR / TSR-pdf / pdf2img) match via SHA256 of FILE bytes; REQUEST_ID is informational. The Java client patch (Phase 2 of the plan) builds these IDs automatically. ## End-to-End Smoke Test After both Phase 1 (server) and Phase 2 (client patch) are merged: ```bash # Terminal 1 python -m mock_server --pdf-dir ... --fixture-dir ... --port 18008 # Terminal 2 β€” pdfua against the mock cd opendataloader-project/opendataloader-pdfua mvn package -q java -jar target/*.jar \ --input /path/to/opendataloader-bench/pdfs/01030000000001.pdf \ --output /tmp/out \ --hybrid hancom-ai \ --hybrid-url http://localhost:18008 ``` Expected: tagged PDF appears in `/tmp/out`, server log shows DLA_OCR + pdf2img + TSR + IMAGE_CAPTIONING calls all returning SUCCESS. ## Tests ```bash pytest -v ``` Expected: ~22 tests pass. ## FIXTURE_MISS Debugging When `SUCCESS:false`, response includes `_mock_hint`: - `sha256=... not in 200-PDF index` β†’ input PDF is not in the recorded benchmark set; either add it or use a benchmark PDF. - `REQUEST_ID does not match odl--p-o-<...>` β†’ client did not apply the REQUEST_ID convention; check Phase 2 patch. - `no FIXTURE_<...>` β†’ the recorded set lacks this specific page/object; this can happen for new combinations. ``` - [ ] **Step 2: Commit** ```bash git add scripts/mock_server/README.md git commit -m "docs(mock-server): README with endpoints, REQUEST_ID convention, E2E smoke" ``` --- ## Phase 2 β€” Client Patch (4 tasks) ### Task 11: Test for REQUEST_ID building (TDD) **Files:** - Create: `opendataloader-project/opendataloader-pdf/java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/HancomAIClientRequestIdTest.java` - [ ] **Step 1: μ‹€νŒ¨ ν…ŒμŠ€νŠΈ μž‘μ„±** `HancomAIClientRequestIdTest.java`: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` - [ ] **Step 2: pom.xml에 mockwebserver μ˜μ‘΄μ„± 확인** Run: `grep -n "mockwebserver" opendataloader-project/opendataloader-pdf/java/opendataloader-pdf-core/pom.xml` Expected: dependency line exists. If not, add to `pom.xml` ``: ```xml com.squareup.okhttp3 mockwebserver 4.12.0 test ``` - [ ] **Step 3: μ‹€νŒ¨ 확인** Run: ```bash cd opendataloader-project/opendataloader-pdf/java mvn -pl opendataloader-pdf-core test -Dtest=HancomAIClientRequestIdTest ``` Expected: COMPILE FAIL β€” `invokeCallModule`, `invokeCallImageCaptioning`, `setSourcePdfShaShort` not defined. - [ ] **Step 4: Commit (test only, expecting fail)** ```bash cd opendataloader-project/opendataloader-pdf git add java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/HancomAIClientRequestIdTest.java java/opendataloader-pdf-core/pom.xml git commit -m "test(hybrid): failing test for HancomAIClient REQUEST_ID convention" ``` --- ### Task 12: HancomAIClient 패치 β€” REQUEST_ID λΉŒλ“œ **Files:** - Modify: `opendataloader-project/opendataloader-pdf/java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/HancomAIClient.java` - [ ] **Step 1: SHA256 헬퍼 + μΈμŠ€ν„΄μŠ€ ν•„λ“œ + λͺ¨λ“ˆ 단좕λͺ… 맡 μΆ”κ°€** In `HancomAIClient.java` after the existing instance field declarations (after `private final HybridConfig config;`), add: ```java private String sourcePdfShaShort = "unknown"; private static final java.util.Map MODULE_SHORT; static { java.util.Map m = new java.util.HashMap<>(); m.put("DOCUMENT_LAYOUT_WITH_OCR", "dla-ocr"); m.put("DOCUMENT_LAYOUT_ANALYSIS", "dla"); m.put("TEXT_RECOGNITION", "ocr"); m.put("TABLE_STRUCTURE_RECOGNITION", "tsr"); m.put("IMAGE_CAPTIONING", "caption"); m.put("CHART_IMAGE_UNDERSTANDING", "chart"); MODULE_SHORT = java.util.Collections.unmodifiableMap(m); } private static String sha256ShortHex(byte[] data) { try { java.security.MessageDigest md = java.security.MessageDigest.getInstance("SHA-256"); byte[] hash = md.digest(data); StringBuilder sb = new StringBuilder(12); for (int i = 0; i < 6; i++) sb.append(String.format("%02x", hash[i])); return sb.toString(); } catch (java.security.NoSuchAlgorithmException e) { return "nohash000000"; } } // Test hook void setSourcePdfShaShort(String s) { this.sourcePdfShaShort = s; } ``` - [ ] **Step 2: `convert(HybridRequest)` μ§„μž…μ—μ„œ SHA 계산** In `convert(...)` method, immediately after `byte[] pdfBytes = request.getPdfBytes();`, add: ```java this.sourcePdfShaShort = sha256ShortHex(pdfBytes); ``` - [ ] **Step 3: `callModule` REQUEST_ID λ³€κ²½** In `callModule(byte[] pdfBytes, String moduleName)` method, replace the line: ```java .addFormDataPart("REQUEST_ID", "odl-" + moduleName) ``` with: ```java .addFormDataPart("REQUEST_ID", "odl-" + sourcePdfShaShort + "-" + MODULE_SHORT.getOrDefault(moduleName, moduleName)) ``` - [ ] **Step 4: `callModuleImage` μ‹œκ·Έλ‹ˆμ²˜ + REQUEST_ID λ³€κ²½** Replace the entire `callModuleImage` method with: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Update the single existing call site of `callModuleImage` (line ~410 in original, inside `recognizeTableStructures`) β€” find: ```java JsonNode tsrResult = callModuleImage(cropPng, "TABLE_STRUCTURE_RECOGNITION"); ``` and replace with: ```java int objId = obj.has("object_id") ? obj.get("object_id").asInt() : -1; JsonNode tsrResult = callModuleImage(cropPng, "TABLE_STRUCTURE_RECOGNITION", pageNum, objId); ``` - [ ] **Step 5: `callImageCaptioning` μ‹œκ·Έλ‹ˆμ²˜ + REQUEST_ID λ³€κ²½** Replace the `callImageCaptioning` method signature and body with: ```java private String callImageCaptioning(byte[] pngBytes, int pageNum, int objectId) throws IOException { String requestId = "odl-" + sourcePdfShaShort + "-p" + pageNum + "-o" + objectId + "-caption"; MultipartBody body = new MultipartBody.Builder() .setType(MultipartBody.FORM) .addFormDataPart("REQUEST_ID", requestId) .addFormDataPart("OPEN_API_NAME", "IMAGE_CAPTIONING") .addFormDataPart("DATA_FORMAT", "image") .addFormDataPart("FILE", "figure.png", RequestBody.create(pngBytes, MEDIA_TYPE_PNG)) .build(); Request httpRequest = new Request.Builder() .url(baseUrl + SDK_ENDPOINT) .post(body) .build(); try (Response response = httpClient.newCall(httpRequest).execute()) { if (!response.isSuccessful()) return null; ResponseBody respBody = response.body(); if (respBody == null) return null; JsonNode root = objectMapper.readTree(respBody.string()); if (!root.has("SUCCESS") || !root.get("SUCCESS").asBoolean()) return null; JsonNode result = root.get("RESULT"); if (result == null || !result.isArray() || result.size() == 0) return null; JsonNode page = result.get(0); if (page.isArray() && page.size() > 0) page = page.get(0); return page.has("caption") ? page.get("caption").asText("") : null; } } ``` Update the single existing call site (inside `captionFigures`, line ~294 in original) β€” find: ```java String caption = callImageCaptioning(croppedPng); ``` and replace with: ```java int objIdForCaption = fig.has("object_id") ? fig.get("object_id").asInt() : -1; String caption = callImageCaptioning(croppedPng, pageNum, objIdForCaption); ``` - [ ] **Step 6: `fetchPageImage` REQUEST_ID κ°•ν™” (mock 맀칭 무관, 디버그 일관성)** In `fetchPageImage(...)`, replace: ```java .addFormDataPart("REQUEST_ID", "odl-pdf2img-" + pageIndex) ``` with: ```java .addFormDataPart("REQUEST_ID", "odl-" + sourcePdfShaShort + "-pdf2img-p" + pageIndex) ``` - [ ] **Step 7: ν…ŒμŠ€νŠΈ hook λ©”μ„œλ“œ μΆ”κ°€ (package-private)** At the very bottom of the class (before the closing `}`), add: ```java // --- Test hooks (package-private) --- void invokeCallModule(byte[] pdfBytes, String moduleName) throws IOException { this.sourcePdfShaShort = sha256ShortHex(pdfBytes); callModule(pdfBytes, moduleName); } void invokeCallImageCaptioning(byte[] pngBytes, int pageNum, int objectId) throws IOException { callImageCaptioning(pngBytes, pageNum, objectId); } ``` - [ ] **Step 8: 컴파일 + μƒˆ ν…ŒμŠ€νŠΈ 톡과 확인** Run: ```bash cd opendataloader-project/opendataloader-pdf/java mvn -pl opendataloader-pdf-core test -Dtest=HancomAIClientRequestIdTest ``` Expected: 2 passed. - [ ] **Step 9: κΈ°μ‘΄ ν…ŒμŠ€νŠΈλ„ νšŒκ·€ μ—†λŠ”μ§€ 확인** Run: ```bash mvn -pl opendataloader-pdf-core test ``` Expected: BUILD SUCCESS, all tests pass. - [ ] **Step 10: Commit** ```bash cd opendataloader-project/opendataloader-pdf git add java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/HancomAIClient.java git commit -m "feat(hybrid): HancomAIClient REQUEST_ID convention with sha-short, page, obj For mock-server fixture lookup. Real Hancom AI server treats REQUEST_ID as opaque client-defined string; no behavior change there. Spec: bundolee/kb-odl/.../2026-04-18_Q2-DEV-02-Code_hancom-ai-mock-server-design.md" ``` --- ### Task 13: 톡합 smoke β€” pdfua β†’ mock 전체 호좜 체인 **Files:** (μ½”λ“œ λ³€κ²½ μ—†μŒ, κ²€μ¦λ§Œ) - [ ] **Step 1: pdf-core 둜컬 install** Run: ```bash cd opendataloader-project/opendataloader-pdf/java mvn -pl opendataloader-pdf-core install -DskipTests ``` Expected: BUILD SUCCESS, artifact installed to `~/.m2`. - [ ] **Step 2: pdfua λΉŒλ“œ** Run: ```bash cd opendataloader-project/opendataloader-pdfua mvn package -DskipTests ``` Expected: BUILD SUCCESS, jar at `target/`. - [ ] **Step 3: mock μ„œλ²„ λ°±κ·ΈλΌμš΄λ“œ 기동** Run: ```bash cd opendataloader-project/opendataloader-pdfua/scripts/mock_server python -m mock_server \ --pdf-dir /Users/benedict/Workspace/opendataloader-project/opendataloader-bench/pdfs \ --fixture-dir /Users/benedict/Workspace/bundolee/kb-odl/raw/4-기술/2026-04-16_Q2-기술-ctx_hancom-ai-a11y_좜λ ₯데이터-μŠ€ν‚€λ§ˆ \ --port 18008 > /tmp/mock-server.log 2>&1 & sleep 2 curl -sf http://127.0.0.1:18008/ping ``` Expected: `{"status":"ok"}`, log contains `indexed 200 PDFs`. - [ ] **Step 4: pdfua μ‹€ν–‰** Run: ```bash mkdir -p /tmp/mock-out cd opendataloader-project/opendataloader-pdfua java -jar target/*.jar \ --input /Users/benedict/Workspace/opendataloader-project/opendataloader-bench/pdfs/01030000000001.pdf \ --output /tmp/mock-out \ --hybrid hancom-ai \ --hybrid-url http://localhost:18008 ``` Expected: exit 0, output PDF in `/tmp/mock-out/`, server log shows DLA_OCR + pdf2img + (TSR or IMAGE_CAPTIONING based on document) all SUCCESS. - [ ] **Step 5: FIXTURE_MISS μ˜λ„μ  트리거 β€” μ™ΈλΆ€ PDF** Run: ```bash echo "%PDF-1.4 fake" > /tmp/notindexed.pdf java -jar opendataloader-project/opendataloader-pdfua/target/*.jar \ --input /tmp/notindexed.pdf \ --output /tmp/mock-out2 \ --hybrid hancom-ai \ --hybrid-url http://localhost:18008 ``` Expected: server log contains `FIXTURE_MISS pdf` warning with sha prefix; pdfua may either fail loud or fall back gracefully β€” record actual behavior in commit message. - [ ] **Step 6: μ„œλ²„ μ’…λ£Œ + 둜그 보쑴** Run: ```bash kill %1 || pkill -f "python -m mock_server" cp /tmp/mock-server.log opendataloader-project/opendataloader-pdfua/scripts/mock_server/e2e-smoke.log ``` - [ ] **Step 7: κ²°κ³Ό κ²€ν†  + commit** Run: ```bash ls -la /tmp/mock-out/ head -50 opendataloader-project/opendataloader-pdfua/scripts/mock_server/e2e-smoke.log ``` ```bash cd opendataloader-project/opendataloader-pdfua git add scripts/mock_server/e2e-smoke.log git commit -m "chore(mock-server): record e2e smoke log against benchmark PDF #01030000000001" ``` --- ### Task 14: νšŒμ˜λ‘μ— μ™„λ£Œ 보고 μΆ”κ°€ **Files:** - Modify: `bundolee/kb-odl/raw/운영/회의둝/2026-04-16_hancom-ai-a11y_2차회의-λ§€ν•‘ν‘œ-리뷰.md` (λ‹€μŒ μ•‘μ…˜ ν‘œ μ—…λ°μ΄νŠΈ) λ˜λŠ” 별도 진행 보고 - [ ] **Step 1: 회의둝 λ‹€μŒ μ•‘μ…˜ ν‘œ κ°±μ‹ ** In the meeting notes' "λ‹€μŒ μ•‘μ…˜" table (around line 63), find rows referencing the blocked development and append a status note. If unsure where, add a new row below the existing table: ```markdown | 8 | hancom-ai mock μ„œλ²„ + client REQUEST_ID 패치 μ™„λ£Œ | 기술 | 2026-04-18 β€” transformer μž¬μ„€κ³„ λ§‰νž˜ ν•΄μ†Œ | ``` - [ ] **Step 2: Commit (in kb-odl)** ```bash cd /Users/benedict/Workspace/bundolee/kb-odl git add "raw/운영/회의둝/2026-04-16_hancom-ai-a11y_2차회의-λ§€ν•‘ν‘œ-리뷰.md" git commit -m "회의둝 κ°±μ‹ : hancom-ai mock μ„œλ²„ ꡬ좕 μ™„λ£Œ, transformer μž¬μ„€κ³„ unblock" ``` --- ## Self-Review Checklist (μž‘μ„±μž 자체 검증) **Spec coverage:** - βœ… 6개 λͺ¨λ“ˆ (DLA_OCR/DLA/OCR/TSR/IMAGE_CAPTIONING/CHART) β€” Task 4 lookup, Task 6/7 server - βœ… pdf2img 동적 λ³€ν™˜ β€” Task 5 renderer, Task 8 endpoint - βœ… /ping μ—”λ“œν¬μΈνŠΈ β€” Task 6 - βœ… /api/v1 alias β€” Task 6 - βœ… DATA_FORMAT λΆ„κΈ° β€” Task 6/7 - βœ… TSR_regionlist 폴백 β€” Task 4 - βœ… Unknown PDF FIXTURE_MISS β€” Task 6 - βœ… Client 패치 (REQUEST_ID κ·œμ•½ 6개 λͺ¨λ“ˆ 단좕λͺ…, sha_short, page, obj) β€” Task 11/12 - βœ… E2E smoke β€” Task 13 **Placeholder scan:** TBD/TODO/"κ΅¬ν˜„ λ‚˜μ€‘μ—" μ—†μŒ. λͺ¨λ“  μ½”λ“œ step에 μ™„μ „ν•œ μ½”λ“œ 블둝 첨뢀. **Type consistency:** `RequestIdParts(sha_short, page, obj, module)` 일관, `MODULE_TO_DIR`(server-side mapping) vs `MODULE_SHORT`(client-side mapping) 뢄리 λͺ…ν™•. `lookup_image_module(..., module_short, basename, page, obj)` μ‹œκ·Έλ‹ˆμ²˜ 일관. **Java 컴파일 μœ„ν—˜μš”μ†Œ:** - `HexFormat`은 Java 17+. λ§Œμ•½ λΉŒλ“œ 타깃이 Java 8/11이면 ν…ŒμŠ€νŠΈμ—μ„œ 직접 hex λ³€ν™˜ ν—¬νΌλ‘œ λŒ€μ²΄ ν•„μš”. Task 11 Step 1 컴파일 μ‹€νŒ¨ μ‹œ μ¦‰μ‹œ 발견. - mockwebserver μ˜μ‘΄μ„± μΆ”κ°€ ν•„μš” (Task 11 Step 2μ—μ„œ 검증). --- ## Execution Handoff Plan complete and saved to `opendataloader-project/opendataloader-pdf/docs/superpowers/plans/2026-04-18-hancom-ai-mock-server.md`. Two execution options: 1. **Subagent-Driven (recommended)** β€” fresh subagent per task with two-stage review between tasks. Best for this plan since it spans two repos (Python + Java) and has a clear TDD rhythm. 2. **Inline Execution** β€” execute tasks in this session with checkpoints. Faster turnaround but heavier on the main context. Which approach? --- ## File: docs/superpowers/plans/2026-04-29-hybrid-hancom-ai-options.md # Hybrid hancom-ai Options & CLI Refactoring Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Expose 5 hancom-ai-specific knobs as `--hybrid-hancom-ai-*` CLI options, then refactor `CLIOptions` so opendataloader-pdfua reuses the entire core option set instead of redefining its own. **Architecture:** Three independent commits. (1) Add 5 options + gate validation in core. (2) Extract `addAllTo` / `applyAllTo` public API on `CLIOptions`. (3) Refactor pdfua's `Main.java` and `RemediationConfig` to consume that API; collapse 9 `RemediationConfig` constructors into a Builder; embed `Config` directly. Hard-break removed getters since all call sites are internal/test. **Tech Stack:** Java 11, Apache Commons CLI, JUnit 5, Maven multi-module (`opendataloader-pdf-core`, `opendataloader-pdf-cli`, `opendataloader-pdfua`). --- ## File Structure **opendataloader-pdf (core)** | File | Action | Responsibility | |---|---|---| | `java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java` | Modify | Add 5 option constants, 5 OPTION_DEFINITIONS entries, parsing in `applyHybridOptions`, hancom-ai gate. Add public `addAllTo(Options)` and `applyAllTo(Config, CommandLine)`. | | `java/opendataloader-pdf-cli/src/test/java/org/opendataloader/pdf/cli/CLIOptionsTest.java` | Modify | Tests for each new option's parsing + gate enforcement. | | `options.json` (root) | Regenerate | Auto-generated from `CLIOptions` via `npm run sync`. | | Python/Node bindings | Regenerate | Auto-generated by `npm run sync`. | **opendataloader-pdfua** | File | Action | Responsibility | |---|---|---| | `src/main/java/org/opendataloader/pdf/Main.java` | Modify | Replace self-defined hybrid options with `CLIOptions.addAllTo()`. Add `applyPdfuaDefaults()`. Use new `RemediationConfig.Builder`. | | `src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java` | Modify | Embed core `Config`. Replace 9 constructors with one Builder. Remove `getHybridUrl()` / `getHybridMode()` (delegate via `getHybridConfig()`). | | `src/main/java/org/opendataloader/pdf/remediation/RemediationProcessor.java` | Modify | Replace per-field hybrid forwarding (lines 163-170) with `config = remediationConfig.getCoreConfig()`. | | `src/test/java/.../AuditBundleEmitterTest.java` | Modify | Switch to Builder. | | `src/test/java/.../CertificateIssuerTest.java` | Modify | Switch to Builder (4 instantiations). | | `src/test/java/.../AuditManifestBuilderTest.java` | Modify | Switch to Builder (4 instantiations). | | `src/test/java/.../RemediationConfigAuditBundleTest.java` | Modify | Switch to Builder. | --- ## Phase 1 β€” Add 5 hybrid-hancom-ai-* options to core ### Task 1.1: Add option name + description constants **Files:** - Modify: `opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java` (insert after line 137, the existing `HYBRID_FALLBACK_DESC`) - [ ] **Step 1: Add the 5 long-option constants and descriptions** Insert after the `HYBRID_FALLBACK_DESC` block (currently around line 137) and before the `// ===== Stdout Output =====` section header: ```java // ===== Hybrid hancom-ai backend-specific ===== private static final String HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION = "hybrid-hancom-ai-regionlist-strategy"; private static final String HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_DESC = "DLA label 7 (regionlist) handling. Requires --hybrid=hancom-ai. " + "Values: table-first (default; check TSR overlap), list-only (skip TSR, always treat as list)"; private static final String HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION = "hybrid-hancom-ai-ocr-strategy"; private static final String HYBRID_HANCOM_AI_OCR_STRATEGY_DESC = "OCR strategy. Requires --hybrid=hancom-ai. " + "Values: off (stream-only), auto (default; stream first, OCR fallback), force (OCR-only)"; private static final String HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION = "hybrid-hancom-ai-image-cache"; private static final String HYBRID_HANCOM_AI_IMAGE_CACHE_DESC = "Page image cache backing. Requires --hybrid=hancom-ai. " + "Values: memory (default), disk"; private static final String HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION = "hybrid-hancom-ai-save-crops"; private static final String HYBRID_HANCOM_AI_SAVE_CROPS_DESC = "Persist cropped figure images to disk for debugging. Requires --hybrid=hancom-ai"; private static final String HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION = "hybrid-hancom-ai-crop-output-dir"; private static final String HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_DESC = "Output directory for --hybrid-hancom-ai-save-crops. Requires --hybrid=hancom-ai"; ``` - [ ] **Step 2: Verify file still compiles** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli compile -DskipTests -q` Expected: BUILD SUCCESS (constants added but unused β€” no warning since `private static final` is fine). - [ ] **Step 3: Commit** ```bash cd opendataloader-pdf && git add java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java git commit -m "$(cat <<'EOF' feat(cli): add --hybrid-hancom-ai-* option name constants Defines long-name and description constants for 5 hancom-ai-specific flags. Wiring into OPTION_DEFINITIONS and parser comes in subsequent commits. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 1.2: Register options in OPTION_DEFINITIONS **Files:** - Modify: `opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java` (the `OPTION_DEFINITIONS` list, currently lines 166-208) - [ ] **Step 1: Add 3 exported entries to OPTION_DEFINITIONS (after `HYBRID_FALLBACK_LONG_OPTION` line, around line 196)** Insert these three lines immediately after the `new OptionDefinition(HYBRID_FALLBACK_LONG_OPTION, ...)` line and before the `new OptionDefinition(TO_STDOUT_LONG_OPTION, ...)` line: ```java new OptionDefinition(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION, null, "string", "table-first", HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_DESC, true), new OptionDefinition(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION, null, "string", "auto", HYBRID_HANCOM_AI_OCR_STRATEGY_DESC, true), new OptionDefinition(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION, null, "string", "memory", HYBRID_HANCOM_AI_IMAGE_CACHE_DESC, true), ``` - [ ] **Step 2: Add 2 hidden entries (exported=false) after the legacy options block (around line 208)** Insert after the `NO_JSON_REPORT_LONG_OPTION` line (the last legacy entry). Note the trailing `)`/`;` of the list β€” adjust so the new lines come **before** the closing `)`: ```java new OptionDefinition(HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION, null, "boolean", false, HYBRID_HANCOM_AI_SAVE_CROPS_DESC, false), new OptionDefinition(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION, null, "string", null, HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_DESC, false), ``` - [ ] **Step 3: Add a "options registered" sanity test** Modify `opendataloader-pdf/java/opendataloader-pdf-cli/src/test/java/org/opendataloader/pdf/cli/CLIOptionsTest.java` β€” append the following test methods at the end of the class (before the closing `}`): ```java @Test void testDefineOptions_containsHybridHancomAiRegionlistStrategy() { assertTrue(options.hasOption("hybrid-hancom-ai-regionlist-strategy")); } @Test void testDefineOptions_containsHybridHancomAiOcrStrategy() { assertTrue(options.hasOption("hybrid-hancom-ai-ocr-strategy")); } @Test void testDefineOptions_containsHybridHancomAiImageCache() { assertTrue(options.hasOption("hybrid-hancom-ai-image-cache")); } @Test void testDefineOptions_containsHybridHancomAiSaveCrops() { assertTrue(options.hasOption("hybrid-hancom-ai-save-crops")); } @Test void testDefineOptions_containsHybridHancomAiCropOutputDir() { assertTrue(options.hasOption("hybrid-hancom-ai-crop-output-dir")); } ``` - [ ] **Step 4: Run the sanity tests** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest -q` Expected: All 5 new tests PASS. (No parsing yet β€” these only verify Apache Commons CLI registration.) - [ ] **Step 5: Commit** ```bash cd opendataloader-pdf && git add java/opendataloader-pdf-cli git commit -m "$(cat <<'EOF' feat(cli): register --hybrid-hancom-ai-* in OPTION_DEFINITIONS Three options (regionlist-strategy, ocr-strategy, image-cache) are exported to options.json and wrapper bindings; two debug options (save-crops, crop-output-dir) stay hidden via exported=false. Sanity tests verify Apache Commons CLI registration. Parsing logic and Config wiring come next. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 1.3: Parse new options into HybridConfig **Files:** - Modify: `opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java` β€” `applyHybridOptions()` method (around lines 463-557) - [ ] **Step 1: Write failing test for `--hybrid-hancom-ai-regionlist-strategy=list-only`** Append to `CLIOptionsTest.java`: ```java @Test void testCreateConfig_withHybridHancomAiRegionlistStrategy() throws ParseException { String[] args = {"--hybrid", "hancom-ai", "--hybrid-hancom-ai-regionlist-strategy", "list-only", testPdf.getAbsolutePath()}; CommandLine cmd = parser.parse(options, args); Config config = CLIOptions.createConfigFromCommandLine(cmd); assertEquals("list-only", config.getHybridConfig().getRegionlistStrategy()); } ``` - [ ] **Step 2: Run, expect FAIL** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest#testCreateConfig_withHybridHancomAiRegionlistStrategy -q` Expected: FAIL β€” value still default `table-first` because parsing hasn't been added yet. - [ ] **Step 3: Add parsing for all 5 new options** Insert this block inside `applyHybridOptions()`, immediately after the existing `if (commandLine.hasOption(HYBRID_FALLBACK_LONG_OPTION)) { config.getHybridConfig().setFallbackToJava(true); }` line and before `if (commandLine.hasOption(TO_STDOUT_LONG_OPTION)) { ... }`: ```java if (commandLine.hasOption(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION)) { String value = commandLine.getOptionValue(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION); if (value != null && !value.trim().isEmpty()) { config.getHybridConfig().setRegionlistStrategy(value.trim().toLowerCase(Locale.ROOT)); } } if (commandLine.hasOption(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION)) { String value = commandLine.getOptionValue(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION); if (value != null && !value.trim().isEmpty()) { config.getHybridConfig().setOcrStrategy(value.trim().toLowerCase(Locale.ROOT)); } } if (commandLine.hasOption(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION)) { String value = commandLine.getOptionValue(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION); if (value != null && !value.trim().isEmpty()) { config.getHybridConfig().setImageCache(value.trim().toLowerCase(Locale.ROOT)); } } if (commandLine.hasOption(HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION)) { config.getHybridConfig().setSaveCrops(true); } if (commandLine.hasOption(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION)) { String value = commandLine.getOptionValue(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION); if (value != null && !value.trim().isEmpty()) { config.getHybridConfig().setCropOutputDir(value.trim()); } } ``` - [ ] **Step 4: Run the test, expect PASS** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest#testCreateConfig_withHybridHancomAiRegionlistStrategy -q` Expected: PASS. - [ ] **Step 5: Add coverage tests for the other four options** Append to `CLIOptionsTest.java`: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` - [ ] **Step 6: Run all new tests, expect PASS** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest -q` Expected: All tests PASS. - [ ] **Step 7: Commit** ```bash cd opendataloader-pdf && git add java/opendataloader-pdf-cli git commit -m "$(cat <<'EOF' feat(cli): parse --hybrid-hancom-ai-* into HybridConfig Wires --hybrid-hancom-ai-{regionlist-strategy,ocr-strategy,image-cache, save-crops,crop-output-dir} through applyHybridOptions(). HybridConfig setters validate enum values and throw IllegalArgumentException on invalid input. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 1.4: Add hancom-ai gate validation **Files:** - Modify: `opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java` β€” `applyHybridOptions()` - [ ] **Step 1: Write failing test β€” using the option without `--hybrid=hancom-ai` should throw** Append to `CLIOptionsTest.java`: ```java @Test void testCreateConfig_hybridHancomAiOption_withoutHancomAi_throws() { String[] args = {"--hybrid-hancom-ai-regionlist-strategy", "list-only", testPdf.getAbsolutePath()}; // No --hybrid set, defaults to off. IllegalArgumentException ex = assertThrows(IllegalArgumentException.class, () -> { CommandLine cmd = parser.parse(options, args); CLIOptions.createConfigFromCommandLine(cmd); }); assertTrue(ex.getMessage().contains("--hybrid-hancom-ai-"), "Error should mention the offending prefix, got: " + ex.getMessage()); assertTrue(ex.getMessage().contains("hancom-ai"), "Error should mention required backend, got: " + ex.getMessage()); } @Test void testCreateConfig_hybridHancomAiOption_withDoclingFast_throws() { String[] args = {"--hybrid", "docling-fast", "--hybrid-hancom-ai-ocr-strategy", "force", testPdf.getAbsolutePath()}; assertThrows(IllegalArgumentException.class, () -> { CommandLine cmd = parser.parse(options, args); CLIOptions.createConfigFromCommandLine(cmd); }); } ``` - [ ] **Step 2: Run, expect FAIL** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest#testCreateConfig_hybridHancomAiOption_withoutHancomAi_throws -q` Expected: FAIL β€” currently no validation, so the call succeeds silently. - [ ] **Step 3: Add the gate at the end of `applyHybridOptions()`** Insert this block at the very end of `applyHybridOptions()` (after the `--to-stdout` handling, before the closing `}`): ```java boolean usesHancomAiOnly = commandLine.hasOption(HYBRID_HANCOM_AI_REGIONLIST_STRATEGY_LONG_OPTION) || commandLine.hasOption(HYBRID_HANCOM_AI_OCR_STRATEGY_LONG_OPTION) || commandLine.hasOption(HYBRID_HANCOM_AI_IMAGE_CACHE_LONG_OPTION) || commandLine.hasOption(HYBRID_HANCOM_AI_SAVE_CROPS_LONG_OPTION) || commandLine.hasOption(HYBRID_HANCOM_AI_CROP_OUTPUT_DIR_LONG_OPTION); if (usesHancomAiOnly && !Config.HYBRID_HANCOM_AI.equals(config.getHybrid())) { throw new IllegalArgumentException( "Options --hybrid-hancom-ai-* require --hybrid=hancom-ai (got --hybrid=" + config.getHybrid() + ")"); } ``` - [ ] **Step 4: Run both gate tests, expect PASS** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest -q` Expected: All tests PASS, including both new gate tests AND all earlier ones (which use `--hybrid hancom-ai`). - [ ] **Step 5: Commit** ```bash cd opendataloader-pdf && git add java/opendataloader-pdf-cli git commit -m "$(cat <<'EOF' feat(cli): reject --hybrid-hancom-ai-* without --hybrid=hancom-ai Without this guard, these flags would silently no-op when the user selects a different backend. Fail fast with a message naming the offending prefix and required backend. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 1.5: Regenerate options.json and wrapper bindings **Files:** - Modify: `opendataloader-pdf/options.json` - Modify: `opendataloader-pdf/python/...`, `opendataloader-pdf/node/...` (auto-generated) - [ ] **Step 1: Run npm sync** Run: `cd opendataloader-pdf && npm run sync` Expected: Command completes; `git status` shows changes to `options.json` and python/node binding files. The 3 exported new options (regionlist-strategy, ocr-strategy, image-cache) appear; the 2 hidden ones (save-crops, crop-output-dir) do **not**. - [ ] **Step 2: Spot-check options.json** Verify the diff contains the 3 new options with correct defaults: ```bash cd opendataloader-pdf && grep -A 2 "hybrid-hancom-ai" options.json ``` Expected: 3 entries for `hybrid-hancom-ai-regionlist-strategy`, `hybrid-hancom-ai-ocr-strategy`, `hybrid-hancom-ai-image-cache` with their defaults `table-first` / `auto` / `memory`. No entry for `save-crops` or `crop-output-dir`. - [ ] **Step 3: Commit** ```bash cd opendataloader-pdf && git add options.json python node git commit -m "$(cat <<'EOF' chore: regenerate bindings for --hybrid-hancom-ai-* options Auto-generated by npm run sync. Adds the 3 exported hancom-ai options to options.json, Python bindings, and Node bindings. The 2 hidden debug options (save-crops, crop-output-dir) stay out of the public surface. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ## Phase 2 β€” Extract reusable CLIOptions API This phase is a **pure refactor** β€” behavior unchanged. ### Task 2.1: Add public `addAllTo(Options)` method **Files:** - Modify: `opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java` β€” `defineOptions()` (around line 210) - [ ] **Step 1: Write a test that exercises `addAllTo` against a fresh Options instance** Append to `CLIOptionsTest.java`: ```java @Test void testAddAllTo_registersAllOptions() { Options ext = new Options(); CLIOptions.addAllTo(ext); assertTrue(ext.hasOption("hybrid")); assertTrue(ext.hasOption("hybrid-mode")); assertTrue(ext.hasOption("hybrid-hancom-ai-regionlist-strategy")); assertTrue(ext.hasOption("format")); assertTrue(ext.hasOption("threads")); } ``` - [ ] **Step 2: Run, expect FAIL (compile error β€” method does not exist)** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest#testAddAllTo_registersAllOptions -q` Expected: COMPILATION FAILURE β€” `cannot find symbol: addAllTo`. - [ ] **Step 3: Add `addAllTo` and refactor `defineOptions` to use it** Replace the existing `defineOptions` method (currently lines ~210-217) with: ```java public static Options defineOptions() { Options options = new Options(); addAllTo(options); return options; } /** * Registers every core CLI option onto an external {@link Options} instance. * Used by downstream CLIs (e.g. opendataloader-pdfua) that want to inherit * the entire core option set and add their own options on top. * * @param options the Options instance to populate */ public static void addAllTo(Options options) { for (OptionDefinition def : OPTION_DEFINITIONS) { options.addOption(def.toOption()); } } ``` - [ ] **Step 4: Run, expect PASS** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest -q` Expected: All tests PASS β€” `addAllTo` works and all existing tests still work via `defineOptions`. - [ ] **Step 5: Commit** ```bash cd opendataloader-pdf && git add java/opendataloader-pdf-cli git commit -m "$(cat <<'EOF' refactor(cli): extract CLIOptions.addAllTo(Options) public API Splits Options registration out of defineOptions() so downstream CLIs can register every core option onto their own Options instance and add their own options on top. defineOptions() now delegates. Pure refactor β€” no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 2.2: Add public `applyAllTo(Config, CommandLine)` method **Files:** - Modify: `opendataloader-pdf/java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIOptions.java` β€” `createConfigFromCommandLine()` (around lines 219-287) - [ ] **Step 1: Write a test that exercises `applyAllTo` directly** Append to `CLIOptionsTest.java`: ```java @Test void testApplyAllTo_appliesCoreOptions() throws ParseException { Options ext = new Options(); CLIOptions.addAllTo(ext); ext.addOption(null, "downstream-only", true, "Downstream-specific"); String[] args = {"--hybrid", "hancom-ai", "--threads", "2", "--downstream-only", "value", testPdf.getAbsolutePath()}; CommandLine cmd = parser.parse(ext, args); Config config = new Config(); config.setOutputFolder(tempDir.toString()); CLIOptions.applyAllTo(config, cmd); assertEquals("hancom-ai", config.getHybrid()); assertEquals(2, config.getThreads()); // downstream-only is not consumed by applyAllTo β€” that is the caller's job. assertEquals("value", cmd.getOptionValue("downstream-only")); } ``` - [ ] **Step 2: Run, expect FAIL (compile error)** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -Dtest=CLIOptionsTest#testApplyAllTo_appliesCoreOptions -q` Expected: COMPILATION FAILURE β€” `cannot find symbol: applyAllTo`. - [ ] **Step 3: Refactor `createConfigFromCommandLine` to delegate to `applyAllTo`** Replace the existing `createConfigFromCommandLine` (around lines 219-287) with this two-method form: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The output-folder positional handling stays in `createConfigFromCommandLine` only β€” pdfua does not have positional input args, so it should not pay for that branch. - [ ] **Step 4: Run all CLI tests, expect PASS** Run: `cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli test -q` Expected: All tests PASS β€” both the new `applyAllTo` test and every existing test (the latter still go through `createConfigFromCommandLine`, which now delegates). - [ ] **Step 5: Commit** ```bash cd opendataloader-pdf && git add java/opendataloader-pdf-cli git commit -m "$(cat <<'EOF' refactor(cli): extract CLIOptions.applyAllTo(Config, CommandLine) Splits per-option Config wiring out of createConfigFromCommandLine() so downstream CLIs (opendataloader-pdfua) can apply core options to their own Config without inheriting the positional-arg output-folder fallback. createConfigFromCommandLine() delegates. Pure refactor β€” no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 2.3: Install core to local Maven for pdfua Per `opendataloader-pdfua/CLAUDE.md`, pdfua consumes core via Maven; cross-repo changes require `mvn install` on core first. - [ ] **Step 1: Install core to local Maven** Run: `cd opendataloader-pdf/java && mvn install -DskipTests -q` Expected: BUILD SUCCESS. The new core artifact (with `addAllTo` / `applyAllTo`) is now available to pdfua. --- ## Phase 3 β€” Refactor pdfua to consume core CLIOptions ### Task 3.1: Add Builder to RemediationConfig **Files:** - Modify: `opendataloader-pdfua/src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java` This task **adds** a Builder without removing the existing constructors yet β€” this lets us migrate call sites incrementally while keeping the test suite green. - [ ] **Step 1: Add `coreConfig` field, the Builder class, and a builder-based constructor** After the existing `private final MockMode mockMode;` line (around line 52), add: ```java private final org.opendataloader.pdf.api.Config coreConfig; ``` After the last existing constructor (around line 125, ending `this.mockMode = ...;`), add: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Also: in **each** existing constructor (the 5 of them, ending at line 125), add this line after the existing `this.mockMode = ...;` assignment: ```java this.coreConfig = buildCoreConfigForLegacyConstructor(); ``` And add this private helper after the constructors (before `public String getInput()`): ```java private org.opendataloader.pdf.api.Config buildCoreConfigForLegacyConstructor() { org.opendataloader.pdf.api.Config c = new org.opendataloader.pdf.api.Config(); if (this.hybrid != null) { c.setHybrid(this.hybrid); } if (this.hybridUrl != null) { c.getHybridConfig().setUrl(this.hybridUrl); } if (this.hybridMode != null) { c.getHybridConfig().setMode(this.hybridMode); } return c; } ``` This keeps the existing 5 constructors source-compatible during migration. They will be removed in Task 3.5. - [ ] **Step 2: Verify pdfua still compiles and tests still pass** Run: `cd opendataloader-pdfua && mvn test -q` Expected: BUILD SUCCESS, all tests pass. (We added members; we did not change existing behavior.) - [ ] **Step 3: Commit** ```bash cd opendataloader-pdfua && git add src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java git commit -m "$(cat <<'EOF' feat(remediation): add RemediationConfig.Builder + embedded coreConfig Adds a Builder API and a coreConfig field carrying the full core opendataloader-pdf Config. Existing 5 constructors keep working β€” they construct a synthetic coreConfig from the flat hybrid fields to maintain semantics. Migration of call sites and removal of the legacy constructors follows. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 3.2: Refactor pdfua/Main.java to use core CLIOptions **Files:** - Modify: `opendataloader-pdfua/src/main/java/org/opendataloader/pdf/Main.java` - [ ] **Step 1: Replace `buildOptions()` body β€” remove hybrid options, add `addAllTo`** Replace `buildOptions()` (lines 94-125) with: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Note: `--threads` is **removed** here because it now comes from core CLIOptions. The pdfua-specific note about HancomAIClient thread-safety becomes a runtime warning instead β€” that is handled in BatchProcessor and is out of scope for this plan. (If the warning text was important, file a separate issue.) - [ ] **Step 2: Replace `runWithOptions` body** Replace `runWithOptions` (lines 52-92) with: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` - [ ] **Step 3: Run pdfua tests** Run: `cd opendataloader-pdfua && mvn test -q` Expected: PASS. `MainAuditBundleCliTest` exercises CLI parsing and should now route through core CLIOptions transparently. - [ ] **Step 4: Manual smoke test of inherited core options** Run: `cd opendataloader-pdfua && mvn -q exec:java -Dexec.mainClass=org.opendataloader.pdf.Main -Dexec.args="--input src/test/resources --output /tmp/pdfua-test --hybrid-hancom-ai-regionlist-strategy list-only --hybrid-hancom-ai-ocr-strategy auto --audit-bundle none --mock off" 2>&1 | head -20` If the project doesn't have `exec:java` configured, **skip this step** β€” the test suite covers wiring. Expected: Either runs without "Unrecognized option" errors for `--hybrid-hancom-ai-*`, or fails on a downstream concern (network/file). The point is that argument parsing accepts the flags. - [ ] **Step 5: Commit** ```bash cd opendataloader-pdfua && git add src/main/java/org/opendataloader/pdf/Main.java git commit -m "$(cat <<'EOF' refactor(pdfua): inherit core CLIOptions instead of redefining Replace pdfua's hand-rolled --hybrid/--hybrid-url/--hybrid-mode/ --threads definitions with org.opendataloader.pdf.cli.CLIOptions. addAllTo(). pdfua now automatically inherits every core option, including the new --hybrid-hancom-ai-* flags. pdfua's deviating defaults (hybrid=hancom-ai, url=localhost:18008, mode=full) are now in an explicit applyPdfuaDefaults() helper instead of buried in cmd.getOptionValue() defaults. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 3.3: Simplify RemediationProcessor's hybrid forwarding **Files:** - Modify: `opendataloader-pdfua/src/main/java/org/opendataloader/pdf/remediation/RemediationProcessor.java` (lines 163-170) - [ ] **Step 1: Read the current hybrid block to confirm context** Run: `cd opendataloader-pdfua && sed -n '160,175p' src/main/java/org/opendataloader/pdf/remediation/RemediationProcessor.java` Expected output (approximately): ```java Config config = ...; // may need to inspect actual surrounding code ... if (remediationConfig.getHybridUrl() != null || !"docling-fast".equals(remediationConfig.getHybrid())) { config.setHybrid(remediationConfig.getHybrid()); if (remediationConfig.getHybridUrl() != null) { config.getHybridConfig().setUrl(remediationConfig.getHybridUrl()); } config.getHybridConfig().setMode(remediationConfig.getHybridMode()); } ``` - [ ] **Step 2: Replace the hybrid block with coreConfig copy** Replace those lines (currently around 163-170) with: ```java // RemediationConfig now carries the full core Config. Adopt it directly. config = remediationConfig.getCoreConfig(); ``` If the local `Config config` variable is constructed earlier in the method with other settings, you must merge them β€” read the full method first to verify. The simplest correct rewrite is: 1. Find where the local `Config` is first assigned in `processSingle` or whichever method contains lines 163-170. 2. Replace `Config config = new Config();` with `Config config = remediationConfig.getCoreConfig();` 3. Delete the lines 163-170 hybrid block. 4. Keep any other `config.setXxx()` calls in the method β€” they continue to work on the borrowed Config instance. If `coreConfig` is mutated by other settings inside `processSingle` and that mutation should not leak back to the caller, clone it first. But based on the spec's intent ("RemediationConfig embeds Config directly"), in-place mutation is acceptable since `RemediationConfig` is per-document and short-lived. - [ ] **Step 3: Run pdfua tests** Run: `cd opendataloader-pdfua && mvn test -q` Expected: PASS. - [ ] **Step 4: Commit** ```bash cd opendataloader-pdfua && git add src/main/java/org/opendataloader/pdf/remediation/RemediationProcessor.java git commit -m "$(cat <<'EOF' refactor(remediation): forward coreConfig directly to RemediationProcessor Replace the per-field hybrid forwarding block (setHybrid + setUrl + setMode) with config = remediationConfig.getCoreConfig(). Now that RemediationConfig embeds the full core Config, no manual copy is needed. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 3.4: Migrate test call sites to Builder **Files:** - Modify: `opendataloader-pdfua/src/test/java/org/opendataloader/pdf/audit/AuditBundleEmitterTest.java` - Modify: `opendataloader-pdfua/src/test/java/org/opendataloader/pdf/audit/certificate/CertificateIssuerTest.java` - Modify: `opendataloader-pdfua/src/test/java/org/opendataloader/pdf/audit/manifest/AuditManifestBuilderTest.java` - Modify: `opendataloader-pdfua/src/test/java/org/opendataloader/pdf/remediation/RemediationConfigAuditBundleTest.java` - [ ] **Step 1: Find every `new RemediationConfig(` call site** Run: ```bash cd opendataloader-pdfua && grep -rn "new RemediationConfig(" src/test ``` Record each file:line and inspect the constructor arguments. - [ ] **Step 2: For each call site, replace with Builder** For a call like: ```java RemediationConfig cfg = new RemediationConfig( "input.pdf", "/tmp/out", "en", "hancom-ai", "http://localhost:18008", "full", true, 1, EnumSet.of(ConformanceLevel.UA1), ReportLevel.NONE); ``` Rewrite as: ```java org.opendataloader.pdf.api.Config core = new org.opendataloader.pdf.api.Config(); core.setHybrid("hancom-ai"); core.getHybridConfig().setUrl("http://localhost:18008"); core.getHybridConfig().setMode("full"); RemediationConfig cfg = RemediationConfig.builder() .input("input.pdf") .output("/tmp/out") .lang("en") .coreConfig(core) .enrichPictureDescription(true) .threads(1) .conformances(EnumSet.of(ConformanceLevel.UA1)) .reportLevel(ReportLevel.NONE) .build(); ``` If the test only cares about a subset (e.g. doesn't pass hybrid args), use the relevant builder methods only β€” Builder defaults handle the rest. - [ ] **Step 3: Run pdfua tests** Run: `cd opendataloader-pdfua && mvn test -q` Expected: PASS. - [ ] **Step 4: Commit** ```bash cd opendataloader-pdfua && git add src/test git commit -m "$(cat <<'EOF' test(remediation): migrate RemediationConfig instantiations to Builder Switches all 4 test files (AuditBundleEmitterTest, CertificateIssuerTest, AuditManifestBuilderTest, RemediationConfigAuditBundleTest) from positional constructors to RemediationConfig.builder() + .build(). Prepares for removal of the legacy constructors in the next commit. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 3.5: Remove legacy constructors and `getHybridUrl`/`getHybridMode` **Files:** - Modify: `opendataloader-pdfua/src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java` - [ ] **Step 1: Verify no production code references the legacy constructors** Run: ```bash cd opendataloader-pdfua && grep -rn "new RemediationConfig(" src/main ``` Expected: No output. (Main.java should now use `builder()`.) If any matches appear, migrate them to Builder before continuing. - [ ] **Step 2: Verify no code references `getHybridUrl` / `getHybridMode`** Run: ```bash cd opendataloader-pdfua && grep -rn "getHybridUrl\|getHybridMode" src ``` Expected: Only matches in `RemediationConfig.java` itself (the methods being removed). If any other call sites appear, replace with `getCoreConfig().getHybridConfig().getUrl()` / `.getMode()`. - [ ] **Step 3: Remove the 5 legacy constructors and the `buildCoreConfigForLegacyConstructor` helper** In `RemediationConfig.java`, delete: - All 5 public constructors (currently lines 54-125) - The `buildCoreConfigForLegacyConstructor()` helper added in Task 3.1 Keep the private `RemediationConfig(Builder b)` constructor. - [ ] **Step 4: Remove `getHybridUrl()` and `getHybridMode()` getters; redirect `getHybrid()` to delegate** Replace the three lines (currently around 130-132): ```java public String getHybrid() { return hybrid; } public String getHybridUrl() { return hybridUrl; } public String getHybridMode() { return hybridMode; } ``` with: ```java public String getHybrid() { return coreConfig.getHybrid(); } public org.opendataloader.pdf.hybrid.HybridConfig getHybridConfig() { return coreConfig.getHybridConfig(); } ``` Also remove the now-unused fields `hybrid`, `hybridUrl`, `hybridMode` (lines ~42-44) and their assignments from the Builder constructor body. Delegate via `coreConfig` everywhere. - [ ] **Step 5: Verify compile and tests** Run: `cd opendataloader-pdfua && mvn compile -q && mvn test -q` Expected: BUILD SUCCESS and all tests PASS. - [ ] **Step 6: Commit** ```bash cd opendataloader-pdfua && git add src/main/java/org/opendataloader/pdf/remediation/RemediationConfig.java git commit -m "$(cat <<'EOF' refactor(remediation)!: remove legacy RemediationConfig constructors BREAKING: Removes the 5 positional constructors and the flat hybrid/hybridUrl/hybridMode fields. RemediationConfig now consists of pdfua-specific fields plus an embedded core Config; hybrid state is read via getCoreConfig().getHybrid() / getHybridConfig(). getHybridUrl() and getHybridMode() are removed β€” call sites use getHybridConfig().getUrl() / .getMode() instead. Verified that no production or test code outside RemediationConfig itself referenced those getters. Co-Authored-By: Claude Opus 4.7 (1M context) EOF )" ``` --- ### Task 3.6: Final integration check - [ ] **Step 1: Full build + test on both repos** Run: ```bash cd opendataloader-pdf/java && mvn install -DskipTests -q cd opendataloader-pdfua && mvn clean package -q ``` Expected: Both BUILD SUCCESS. - [ ] **Step 2: Run the full test suite for both** Run: ```bash cd opendataloader-pdf/java && mvn test -q cd opendataloader-pdfua && mvn test -q ``` Expected: All tests pass in both repos. - [ ] **Step 3: Verify --help output for both CLIs lists the new options** ```bash cd opendataloader-pdf/java && mvn -pl opendataloader-pdf-cli exec:java \ -Dexec.mainClass=org.opendataloader.pdf.cli.CLIMain -Dexec.args="--help" 2>&1 | \ grep "hybrid-hancom-ai" ``` Expected: 3 lines (regionlist-strategy, ocr-strategy, image-cache). ```bash cd opendataloader-pdfua && java -jar target/*-jar-with-dependencies.jar --input dummy --output dummy --help 2>&1 | grep "hybrid-hancom-ai" ``` Expected: 3 lines (same options inherited). Note: pdfua's `--help` output requires `--input` / `--output` to be supplied since they're `addRequiredOption`. If this is awkward, this verification step can be done by inspecting `Options` programmatically in a quick `MainTest`. - [ ] **Step 4: No final commit needed β€” all phase-3 commits already pushed.** --- ## Self-Review **Spec coverage check:** | Spec section | Tasks | |---|---| | 5 new options with `--hybrid-hancom-ai-*` prefix | Task 1.1, 1.2 | | Validation gate (require --hybrid=hancom-ai) | Task 1.4 | | `addAllTo(Options)` public API | Task 2.1 | | `applyAllTo(Config, CommandLine)` public API | Task 2.2 | | pdfua/Main inherits core options | Task 3.2 | | `applyPdfuaDefaults` explicit override block | Task 3.2 | | `RemediationConfig` Builder + embedded `Config` | Task 3.1, 3.5 | | Hard break: remove `getHybridUrl/Mode`, 9 constructors | Task 3.5 | | `RemediationProcessor` simplified | Task 3.3 | | All test call sites migrated | Task 3.4 | | `npm run sync` to regenerate bindings | Task 1.5 | | `mvn install -DskipTests` (core) before pdfua build | Task 2.3, 3.6 | All spec items mapped to at least one task. **Placeholder check:** No "TBD", "TODO", "fill in details", or stub steps. Every code step contains the actual code. Test code is complete (assertions, args). Commit messages are pre-written. **Type consistency:** `coreConfig` field name used consistently. `getCoreConfig()` getter name used consistently. Builder method names match field names. `applyPdfuaDefaults` signature matches between definition (Task 3.2) and call site (also Task 3.2). --- ## Plan complete and saved to `docs/superpowers/plans/2026-04-29-hybrid-hancom-ai-options.md`. Two execution options: **1. Subagent-Driven (recommended)** - dispatch a fresh subagent per task, review between tasks, fast iteration **2. Inline Execution** - execute tasks in this session using executing-plans, batch execution with checkpoints **Which approach?** --- ## File: docs/hybrid/experiments/triage/triage-experiments.md --- name: triage-lab --- # Triage Lab - Experiment Records This skill manages experiment records and optimization history for triage logic. ## Current Implementation **File**: `java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java` ### Signal Priority (classifyPage method) 1. `hasTableBorder` - TableBorder presence (confidence: 1.0) 2. `hasVectorTableSignal` - Grid lines, border lines, line art (confidence: 0.95) 3. `hasTextTablePattern` - Text patterns with consecutive validation (confidence: 0.9) 4. `hasSuspiciousPattern` - Y-overlap or large gap detection (confidence: 0.85) 5. `lineToTextRatio > 0.3` - High line chunk ratio (confidence: 0.8) 6. `alignedLineGroups >= 5` - Aligned baseline groups (confidence: 0.7) ### Key Thresholds | Parameter | Value | Location | |-----------|-------|----------| | LINE_RATIO_THRESHOLD | 0.3 | TriageProcessor:41 | | ALIGNED_LINE_GROUPS_THRESHOLD | 5 | TriageProcessor:46 | | GRID_GAP_MULTIPLIER | 3.0 | TriageProcessor:49 | | MIN_LINE_COUNT_FOR_TABLE | 8 | TriageProcessor:57 | | MIN_GRID_LINES | 3 | TriageProcessor:60 | | MIN_CONSECUTIVE_PATTERNS | 2 | TriageProcessor:79 | --- ## Experiment History ### Experiment 001 (2026-01-03): FP Cause Analysis **Goal**: Identify root causes of high False Positive rate **Baseline**: - Documents: 200 (42 with tables) - TP: 41, TN: 48, FP: 110, FN: 1 - Precision: 27.15%, Recall: 97.62%, F1: 42.49% **FP by Signal**: | Signal | Count | % | |--------|-------|---| | hasSuspiciousPattern | 65 | 59.1% | | hasVectorTableSignal | 23 | 20.9% | | hasTableBorder | 14 | 12.7% | | hasTextTablePattern | 5 | 4.5% | | alignedLineGroups | 2 | 1.8% | | highLineRatio | 1 | 0.9% | **Root Cause**: Y-overlap check in `hasSuspiciousPattern` is too sensitive - Condition `previous.getTopY() < current.getBottomY()` triggers on normal multi-column layouts **Experiments**: | Config | Precision | Recall | F1 | FP | FN | |--------|-----------|--------|-----|-----|-----| | Baseline | 27.15% | 97.62% | 42.49% | 110 | 1 | | Disable Y-overlap | 36.28% | 97.62% | 52.90% | ~69 | 1 | | Only Reliable Signals | 50.67% | 90.48% | 64.96% | ~38 | 4 | | Disable SuspiciousPattern | 39.22% | 95.24% | 55.56% | ~64 | 2 | | Require 3+ patterns | 37.38% | 95.24% | 53.69% | ~67 | 2 | **Recommendation**: - To maintain recall: Remove Y-overlap check (Precision +9%, Recall unchanged) - To optimize F1: Use only reliable signals (F1 +22%, Recall -7%) **FN Documents**: - `01030000000110`: Missed by all experiments (needs investigation) - `01030000000122`, `01030000000116`, `01030000000117`: Only detected by SuspiciousPattern **Applied**: Y-overlap check removed (2026-01-03) --- ### Experiment 002 (2026-01-03): Further FP Reduction **Goal**: Reduce remaining 72 FPs after Y-overlap removal **Current FP by Signal** (after Experiment 001): | Signal | Count | % | |--------|-------|---| | hasSuspiciousPattern | 21 | 29.2% | | hasTableBorder | 14 | 19.4% | | hasVectorTableSignal | 13 | 18.1% | | alignedLineGroups | 10 | 13.9% | | unknown | 8 | 11.1% | | hasTextTablePattern | 5 | 6.9% | | highLineRatio | 1 | 1.4% | **Experiment 2A: Gap Multiplier** (hasSuspiciousPattern) | Gap | Precision | Recall | F1 | FP | FN | |-----|-----------|--------|-----|-----|-----| | 3.0 (current) | 37.86% | 92.86% | 53.79% | 64 | 3 | | 4.0 | 37.86% | 92.86% | 53.79% | 64 | 3 | | 5.0 | 37.86% | 92.86% | 53.79% | 64 | 3 | | 6.0 | 37.86% | 92.86% | 53.79% | 64 | 3 | β†’ No effect (Y-overlap removal already optimized this signal) **Experiment 2B: AlignedLineGroups Threshold** | Threshold | Precision | Recall | F1 | FP | FN | |-----------|-----------|--------|-----|-----|-----| | 3 (current) | 37.86% | 92.86% | 53.79% | 64 | 3 | | 4 | 39.39% | 92.86% | 55.32% | 60 | 3 | | **5** | **39.80%** | **92.86%** | **55.71%** | **59** | **3** | | 6 | 39.80% | 92.86% | 55.71% | 59 | 3 | β†’ **Recommended**: Threshold 5 (FP -5, Recall maintained) **Experiment 2C: Vector Signal Criteria** | LineCount | GridLines | Precision | Recall | F1 | FP | FN | |-----------|-----------|-----------|--------|-----|-----|-----| | 8, 3 (current) | | 37.86% | 92.86% | 53.79% | 64 | 3 | | 10, 4 | | 38.24% | 92.86% | 54.17% | 63 | 3 | | 12, 4 | | 37.62% | 90.48% | 53.15% | 63 | 4 | β†’ Minimal effect (FP -1, higher values reduce Recall) **Recommendation**: - Apply `alignedLineGroups` threshold 3 β†’ 5 - Expected: FP 64 β†’ 59 (-5), Recall 92.86% (maintained), F1 +1.92% **Applied**: alignedLineGroups threshold 3 β†’ 5 (2026-01-03) **Actual Results**: | Metric | Before (Exp 001) | After (Exp 002) | Change | |--------|------------------|-----------------|--------| | FP | 72 | 67 | -5 | | FN | 1 | 1 | 0 | | Precision | 36.28% | 37.96% | +1.68% | | Recall | 97.62% | 97.62% | 0 | | F1 | 52.90% | 54.67% | +1.77% | **Next Steps**: - Investigate `hasTableBorder` FPs (14 cases, external library) - Investigate `unknown` FPs (8 cases) --- ### Experiment 003 (2026-01-03): VectorTableSignal & SuspiciousPattern Analysis **Goal**: Further reduce FP 67 while maintaining high Recall **Current FP by Signal** (after Experiment 002): | Signal | Count | % | |--------|-------|---| | hasVectorTableSignal | 23 | 34.3% | | hasSuspiciousPattern | 19 | 28.4% | | hasTableBorder | 14 | 20.9% | | hasTextTablePattern | 5 | 7.5% | | alignedLineGroups | 5 | 7.5% | | highLineRatio | 1 | 1.5% | **VectorTableSignal Sub-signal Analysis** (23 FPs): | Sub-signal | Count | |------------|-------| | hasAlignedShortLines | 30 | | hasTableBorderLines | 22 | | hasGridLines | 16 | | lineArt>=8 | 13 | | hasRowSeparatorPattern | 12 | β†’ `hasAlignedShortLines` is the primary cause of VectorSignal FPs **Experiments**: | Config | Precision | Recall | F1 | FP | FN | |--------|-----------|--------|-----|-----|-----| | Current (Exp 002) | 37.96% | 97.62% | 54.67% | 67 | 1 | | 003B: Disable VectorSignal | 40.00% | 90.48% | 55.47% | 57 | 4 | | 003C: Grid OR BorderLines only | 40.21% | 92.86% | 56.12% | 58 | 3 | | **003D: Disable SuspiciousPattern** | **42.11%** | **95.24%** | **58.39%** | **55** | **2** | | 003F: Only Reliable Signals | 56.25% | 85.71% | 67.92% | 28 | 6 | | 003G: Disable AlignedShortLines | 40.00% | 95.24% | 56.34% | 60 | 2 | | 003I: Combined (NoAlign+NoSusp) | 44.71% | 90.48% | 59.84% | 47 | 4 | **Analysis**: - `hasTableBorder` (14 FPs): External library, cannot modify - `hasVectorTableSignal` (23 FPs): `hasAlignedShortLines` too aggressive - `hasSuspiciousPattern` (19 FPs): Gap detection catches non-table layouts **Recommendation**: - **Best for Recall**: 003D (Disable SuspiciousPattern) - FP: 67 β†’ 55 (-12), FN: 1 β†’ 2 (+1), Recall: 95.24% - **Best for F1**: 003I (Combined) - FP: 67 β†’ 47 (-20), FN: 1 β†’ 4 (+3), F1: 59.84% **FN Documents** (003D): - `01030000000122`: Only detected by SuspiciousPattern (gap-based) - `01030000000110`: Never detected (needs separate investigation) **Applied**: 003D - Disabled hasSuspiciousPattern (2026-01-03) **Actual Results**: | Metric | Before (Exp 002) | After (Exp 003) | Change | |--------|------------------|-----------------|--------| | FP | 67 | 55 | -12 | | FN | 1 | 2 | +1 | | Precision | 37.96% | 42.11% | +4.15% | | Recall | 97.62% | 95.24% | -2.38% | | F1 | 54.67% | 58.39% | +3.72% | --- ### Experiment 004 (2026-01-03): AlignedLineGroups Signal Analysis **Goal**: Further reduce FP 55 while maintaining Recall 95.24% **Current FP by Signal** (after Experiment 003): | Signal | Count | % | |--------|-------|---| | hasVectorTableSignal | 23 | 41.8% | | hasTableBorder | 14 | 25.5% | | alignedLineGroups | 12 | 21.8% | | hasTextTablePattern | 5 | 9.1% | | highLineRatio | 1 | 1.8% | **VectorTableSignal Sub-signal Analysis** (23 FPs): | Sub-signal | Count | |------------|-------| | hasAlignedShortLines | 16 | | hasTableBorderLines | 10 | | lineArt>=8 | 8 | | hasRowSeparatorPattern | 7 | | hasGridLines | 5 | **Experiments**: | Config | Precision | Recall | F1 | FP | FN | |--------|-----------|--------|-----|-----|-----| | Current (Exp 003) | 42.11% | 95.24% | 58.39% | 55 | 2 | | 004A: NoAlignedShortLines | 44.71% | 90.48% | 59.84% | 47 | 4 | | 004B: Grid+BorderLines only | 45.12% | 88.10% | 59.68% | 45 | 5 | | **004D: No alignedLineGroups** | **48.19%** | **95.24%** | **64.00%** | **43** | **2** | | 004E: alignedLineGroups>=7 | 44.94% | 95.24% | 61.07% | 49 | 2 | | 004G: NoAlignShort+Groups>=7 | 48.10% | 90.48% | 62.81% | 41 | 4 | | 004I: Reliable Only | 54.41% | 88.10% | 67.27% | 31 | 5 | **Analysis**: - `alignedLineGroups` signal caused 12 FPs but detected no additional true tables - Disabling it removes all 12 FPs without any FN increase - Best option for maintaining Recall while improving Precision **Applied**: 004D - Disabled alignedLineGroups signal (2026-01-03) **Actual Results**: | Metric | Before (Exp 003) | After (Exp 004) | Change | |--------|------------------|-----------------|--------| | FP | 55 | 43 | -12 | | FN | 2 | 2 | 0 | | Precision | 42.11% | 48.19% | +6.08% | | Recall | 95.24% | 95.24% | 0 | | F1 | 58.39% | 64.00% | +5.61% | **Next Steps**: - Investigate `hasVectorTableSignal` FPs (23 remaining) - hasAlignedShortLines main cause - Investigate `hasTableBorder` FPs (14 cases, external library limitation) --- ### Experiment 005 (2026-01-03): Large Image Signal for FN Reduction **Goal**: Reduce FN by detecting pages with large images (potential table/chart images) **Background**: - FN documents `01030000000110` and `01030000000122` contain images with tables - 110: 28.64% page area (graph image) - 122: 11.73% page area (table image) **Implementation**: - Added `hasLargeImage` signal to TriageProcessor - Detects ImageChunk objects and calculates max image area / page area ratio **Experiments**: | Threshold | Precision | Recall | F1 | FP | FN | |-----------|-----------|--------|-----|-----|-----| | Baseline (no image) | 48.19% | 95.24% | 64.00% | 43 | 2 | | 10% | 33.07% | **100%** | 49.70% | 85 | 0 | | **11%** | **33.60%** | **100%** | **50.30%** | **83** | **0** | | 15% | 35.96% | 97.62% | 52.56% | 73 | 1 | **Analysis**: - 11% threshold achieves **100% Recall** (all 42 table documents detected) - Trade-off: FP increases from 43 β†’ 83 (+40), Precision drops from 48% β†’ 34% - F1 decreases from 64% β†’ 50% due to high FP increase - Many FPs are documents with decorative images, diagrams, photos **Experiment 005B: Adding Aspect Ratio Condition** Observation: FN documents have wide images (ratio 1.79, 3.68), while FP documents often have square/tall images (ratio 0.6~1.5). | Config | Precision | Recall | F1 | FP | FN | |--------|-----------|--------|-----|-----|-----| | Baseline (no image) | 48.19% | 95.24% | 64.00% | 43 | 2 | | 11% only | 33.60% | 100% | 50.30% | 83 | 0 | | 11% + ratio 1.7 | 42.86% | 100% | 60.00% | 56 | 0 | | **11% + ratio 1.75** | **43.30%** | **100%** | **60.43%** | **55** | **0** | | 11% + ratio 2.0 | 46.59% | 97.62% | 63.08% | 47 | 1 | **Final Configuration**: - Image area >= 11% of page area - Image aspect ratio (width/height) >= 1.75 **Trade-off**: - Achieves **100% Recall** (all 42 table documents detected) - FP increases from 43 β†’ 55 (+12) - F1 decreases from 64% β†’ 60.43% (-3.57%) **Applied**: 11% + aspect ratio 1.75 (2026-01-03) --- ## Template for New Experiments ```markdown ### Experiment XXX (YYYY-MM-DD): [Title] **Goal**: [What are you trying to improve?] **Changes**: [What did you modify?] **Results**: | Config | Precision | Recall | F1 | FP | FN | |--------|-----------|--------|-----|-----|-----| | Before | | | | | | | After | | | | | | **Conclusion**: [What did you learn? Should this be applied?] **Next Steps**: [What to try next?] ``` --- ## How to Run Experiments ```bash # Run triage accuracy test ./scripts/test-java.sh -Dtest=TriageProcessorIntegrationTest#testTriageAccuracyOnBenchmarkPDFs # Debug specific document ./scripts/bench.sh --doc-id 01030000000110 ``` ## Related Files - [TriageProcessor.java](../../java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/hybrid/TriageProcessor.java) - [TriageProcessorIntegrationTest.java](../../java/opendataloader-pdf-core/src/test/java/org/opendataloader/pdf/hybrid/TriageProcessorIntegrationTest.java) --- ## File: docs/hybrid/experiments/speed/speed-experiment-2026-01-03.md # Docling Speed Experiment Results **Date**: 2026-01-03 14:31:43 ## Summary | Approach | Description | Avg (s/doc) | Target | Status | Speedup | |----------|-------------|-------------|--------|--------|---------| | baseline | docling-serve HTTP | 2.283 | - | - | - | | fastapi | FastAPI + SDK singleton | 0.685 | 0.8 | PASS | 3.3x | | subprocess | Persistent subprocess | 0.661 | 1.0 | PASS | 3.5x | ## Decision **Phase 0 PASSED** - FastAPI approach meets the < 0.8s threshold. Proceed to Phase 1 implementation: - [x] Task 1.1: docling_subprocess_worker.py (skipped - FastAPI only) - [x] Task 1.2: hybrid_server.py (opendataloader-pdf-hybrid CLI) - [x] Task 2.1: DoclingSubprocessClient.java (skipped - FastAPI only) - [x] Task 2.2: DoclingFastServerClient.java - [x] Task 2.3: HybridClientFactory modification - [x] Task 3: Benchmark integration - [x] Task 4: Final validation Subprocess approach also passed - both approaches available for implementation. ## Detailed Statistics ### Baseline - **Description**: docling-serve HTTP API - **Timestamp**: 2026-01-03 14:23:41 - **Total documents**: 200 - **Successful**: 200 - **Failed**: 0 - **Total elapsed**: 456.6s - **Average per doc**: 2.2825s - **Min**: 2.0045s - **Max**: 8.0182s ### Fastapi - **Description**: FastAPI server with docling SDK singleton - **Timestamp**: 2026-01-03 14:27:18 - **Total documents**: 200 - **Successful**: 200 - **Failed**: 0 - **Total elapsed**: 137.1s - **Average per doc**: 0.6855s - **Min**: 0.1912s - **Max**: 4.2420s ### Subprocess - **Description**: Persistent Python subprocess with docling SDK - **Timestamp**: 2026-01-03 14:30:50 - **Total documents**: 200 - **Successful**: 200 - **Failed**: 0 - **Total elapsed**: 132.4s - **Average per doc**: 0.6612s - **Min**: 0.1908s - **Max**: 4.2498s --- ## File: docs/hybrid/research/comparison-summary.md # Docling vs OpenDataLoader Output Comparison ## Test Document - File: `01030000000045.pdf` (1 page with table) ## Element Count Comparison | Category | Docling | OpenDataLoader | |----------|---------|----------------| | Tables | 1 | 1 | | Text elements | 5 | 4 paragraphs | | Images | 0 | 1 | | Headings | (N/A - uses labels) | 1 | ## Text Element Labels (Docling) | Label | Count | |-------|-------| | caption | 1 | | footnote | 1 | | page_footer | 1 | | page_header | 1 | | text | 1 | ## Table Structure Comparison | Property | Docling | OpenDataLoader | |----------|---------|----------------| | Rows | 9 | 3 | | Columns | 3 | 3 | | Total cells | 26 | 9 | **Note**: Docling detects more rows in the table structure. This may be due to: - Different table detection algorithms - OpenDataLoader may have merged some rows - Different handling of header rows ## Bounding Box Comparison (Table) | System | l/left | t/top | r/right | b/bottom | Origin | |--------|--------|-------|---------|----------|--------| | Docling | 53.22 | 439.98 | 373.94 | 234.74 | BOTTOMLEFT | | OpenDataLoader | 54.0 | 234.44 | 372.73 | 440.21 | BOTTOMLEFT | **Coordinate mapping**: Both use BOTTOMLEFT origin. - Docling: `{l, t, r, b}` where t=top, b=bottom - OpenDataLoader: `[left, bottom, right, top]` So the actual coordinates match closely: - Left: 53.22 β‰ˆ 54.0 - Bottom: 234.74 β‰ˆ 234.44 - Right: 373.94 β‰ˆ 372.73 - Top: 439.98 β‰ˆ 440.21 ## Schema Mapping Summary | Docling Type | OpenDataLoader Type | |--------------|---------------------| | texts (label: text) | paragraph | | texts (label: section_header) | heading | | tables | table | | pictures | image | | texts (label: page_header) | paragraph (filtered as header) | | texts (label: page_footer) | paragraph (filtered as footer) | | texts (label: caption) | paragraph | | texts (label: footnote) | paragraph | ## Key Differences 1. **Type naming**: Docling uses `label` field for text types, OpenDataLoader uses `type` 2. **Table structure**: Docling detects more detailed row structure 3. **Coordinate format**: Same origin but different field order 4. **Heading detection**: Docling uses `SectionHeaderItem` with `level`, OpenDataLoader uses `heading` type with `level` --- ## File: docs/hybrid/research/iobject-structure.md # IObject Class Structure ## Overview IObject is imported from `org.verapdf.wcag.algorithms.entities.IObject` (external verapdf-wcag-algs library). ## JSON Output Types Based on sample response analysis, OpenDataLoader produces the following element types: ### Element Types | Type | JSON `type` field | Description | |------|-------------------|-------------| | Paragraph | `paragraph` | Text paragraph with font info | | Heading | `heading` | Section heading with level | | Table | `table` | Table with rows and cells | | Image | `image` | Image/figure element | | List | `list` | Bulleted or numbered list | ### Common Fields (all types) ```json { "type": "paragraph", "id": 17, "page number": 1, "bounding box": [left, bottom, right, top] // PDF points, origin at bottom-left } ``` ### Paragraph Fields ```json { "type": "paragraph", "font": "ArialMT", "font size": 8.0, "text color": "[0.0, 0.0, 0.0, 0.7]", "content": "Text content here" } ``` ### Heading Fields ```json { "type": "heading", "level": "1", "content": "Heading text" } ``` ### Table Structure ```json { "type": "table", "level": "1", "number of rows": 3, "number of columns": 3, "rows": [ { "type": "table row", "row number": 1, "cells": [ { "type": "table cell", "page number": 1, "bounding box": [left, bottom, right, top], "row number": 1, "column number": 1, "row span": 1, "column span": 1, "kids": [ { "type": "paragraph", "content": "Cell text" } ] } ] } ] } ``` ## Bounding Box Coordinate System - **OpenDataLoader**: `[left, bottom, right, top]` in PDF points, origin at BOTTOMLEFT - **Docling**: `{l, t, r, b}` with `coord_origin: "BOTTOMLEFT"` or `"TOPLEFT"` ### Conversion Notes - If docling uses TOPLEFT origin: `bottom = page_height - docling_t`, `top = page_height - docling_b` - If docling uses BOTTOMLEFT origin: direct mapping `[l, b, r, t]` β†’ `[left, bottom, right, top]` ## Key Java Classes From the codebase: - `TableBorder` - Table with border-based detection - `TableBorderRow` - Table row - `TableBorderCell` - Table cell with contents, rowSpan, colSpan - `BoundingBox` - PDF coordinates (page, left, bottom, right, top) - Processors: `TextLineProcessor`, `TableBorderProcessor`, `HeadingProcessor`, `ListProcessor`