## File: README.md

PDF scientific paper translation and bilingual comparison library.

- **Online Service**: Beta version launched [Immersive Translate - BabelDOC](https://app.immersivetranslate.com/babel-doc/) Free usage quota is available; please refer to the FAQ section on the page for details.
- **Self-deployment**: [PDFMathTranslate-next](https://github.com/PDFMathTranslate-next/PDFMathTranslate-next) support for BabelDOC, available for self-deployment + WebUI with more translation services.
- Provides a simple [command line interface](#getting-started).
- Provides a [Python API](#python-api).
- Mainly designed to be embedded into other programs, but can also be used directly for simple translation tasks.

> [!TIP]
>
> How to use BabelDOC in Zotero
>
> 1. Immersive Translate Pro members can use the [immersive-translate/zotero-immersivetranslate](https://github.com/immersive-translate/zotero-immersivetranslate) plugin
>
> 2. PDFMathTranslate self-deployed users can use the [guaguastandup/zotero-pdf2zh](https://github.com/guaguastandup/zotero-pdf2zh) plugin

[Supported Language](https://funstory-ai.github.io/BabelDOC/supported_languages/)

## Preview



## We are hiring

See details: [EN](https://github.com/funstory-ai/jobs) | [ZH](https://github.com/funstory-ai/jobs/blob/main/README_ZH.md)

## Getting Started

### Install from PyPI

We recommend using the Tool feature of [uv](https://github.com/astral-sh/uv) to install BabelDOC.

1. First, you need to refer to [uv installation](https://github.com/astral-sh/uv#installation) to install uv and set up the `PATH` environment variable as prompted.

2. Use the following command to install BabelDOC:

```bash
uv tool install --python 3.12 BabelDOC

babeldoc --help
```

3. Use the `babeldoc` command. For example:

```bash
babeldoc --openai --openai-model "gpt-4o-mini" --openai-base-url "https://api.openai.com/v1" --openai-api-key "your-api-key-here"  --files example.pdf

# multiple files
babeldoc --openai --openai-model "gpt-4o-mini" --openai-base-url "https://api.openai.com/v1" --openai-api-key "your-api-key-here"  --files example1.pdf --files example2.pdf
```

### Install from Source

We still recommend using [uv](https://github.com/astral-sh/uv) to manage virtual environments.

1. First, you need to refer to [uv installation](https://github.com/astral-sh/uv#installation) to install uv and set up the `PATH` environment variable as prompted.

2. Use the following command to install BabelDOC:

```bash
# clone the project
git clone https://github.com/funstory-ai/BabelDOC

# enter the project directory
cd BabelDOC

# install dependencies and run babeldoc
uv run babeldoc --help
```

3. Use the `uv run babeldoc` command. For example:

```bash
uv run babeldoc --files example.pdf --openai --openai-model "gpt-4o-mini" --openai-base-url "https://api.openai.com/v1" --openai-api-key "your-api-key-here"

# multiple files
uv run babeldoc --files example.pdf --files example2.pdf --openai --openai-model "gpt-4o-mini" --openai-base-url "https://api.openai.com/v1" --openai-api-key "your-api-key-here"
```

> [!TIP]
> The absolute path is recommended.

## Advanced Options

> [!NOTE]
> This CLI is mainly for debugging purposes. Although end users can use this CLI to translate files, we do not provide any technical support for this purpose.
>
> End users should directly use **Online Service**: Beta version launched [Immersive Translate - BabelDOC](https://app.immersivetranslate.com/babel-doc/) 1000 free pages per month.
>
> End users who need self-deployment should use [PDFMathTranslate 2.0](https://github.com/PDFMathTranslate/PDFMathTranslate-next)
> 
> If you find that an option is not listed below, it means that this option is a debugging option for maintainers. Please do not use these options.


### Language Options

- `--lang-in`, `-li`: Source language code (default: en)
- `--lang-out`, `-lo`: Target language code (default: zh)

> [!TIP]
> Currently, this project mainly focuses on English-to-Chinese translation, and other scenarios have not been tested yet.
> 
> (2025.3.1 update): Basic English target language support has been added, primarily to minimize line breaks within words([0-9A-Za-z]+).
> 
> [HELP WANTED: Collecting word regular expressions for more languages](https://github.com/funstory-ai/BabelDOC/issues/129)

### PDF Processing Options

- `--files`: One or more file paths to input PDF documents.
- `--pages`, `-p`: Specify pages to translate (e.g., "1,2,1-,-3,3-5"). If not set, translate all pages
- `--split-short-lines`: Force split short lines into different paragraphs (may cause poor typesetting & bugs)
- `--short-line-split-factor`: Split threshold factor (default: 0.8). The actual threshold is the median length of all lines on the current page \* this factor
- `--skip-clean`: Skip PDF cleaning step
- `--dual-translate-first`: Put translated pages first in dual PDF mode (default: original pages first)
- `--disable-rich-text-translate`: Disable rich text translation (may help improve compatibility with some PDFs)
- `--enhance-compatibility`: Enable all compatibility enhancement options (equivalent to --skip-clean --dual-translate-first --disable-rich-text-translate)
- `--use-alternating-pages-dual`: Use alternating pages mode for dual PDF. When enabled, original and translated pages are arranged in alternate order. When disabled (default), original and translated pages are shown side by side on the same page.
- `--watermark-output-mode`: Control watermark output mode: 'watermarked' (default) adds watermark to translated PDF, 'no_watermark' doesn't add watermark, 'both' outputs both versions.
- `--max-pages-per-part`: Maximum number of pages per part for split translation. If not set, no splitting will be performed.
- `--no-watermark`: [DEPRECATED] Use --watermark-output-mode=no_watermark instead.
- `--translate-table-text`: Translate table text (experimental, default: False)
- `--formular-font-pattern`: Font pattern to identify formula text (default: None)
- `--formular-char-pattern`: Character pattern to identify formula text (default: None)
- `--show-char-box`: Show character bounding boxes (debug only, default: False)
- `--skip-scanned-detection`: Skip scanned document detection (default: False). When using split translation, only the first part performs detection if not skipped.
- `--ocr-workaround`: Use OCR workaround (default: False). Only suitable for documents with black text on white background. When enabled, white rectangular blocks will be added below the translation to cover the original text content, and all text will be forced to black color.
- `--auto-enable-ocr-workaround`: Enable automatic OCR workaround (default: False). If a document is detected as heavily scanned, this will attempt to enable OCR processing and skip further scan detection. See "Important Interaction Note" below for crucial details on how this interacts with `--ocr-workaround` and `--skip-scanned-detection`.
- `--primary-font-family`: Override primary font family for translated text. Choices: 'serif' for serif fonts, 'sans-serif' for sans-serif fonts, 'script' for script/italic fonts. If not specified, uses automatic font selection based on original text properties.
- `--only-include-translated-page`: Only include translated pages in the output PDF. This option is only effective when `--pages` is used. (default: False)
- `--merge-alternating-line-numbers`: Enable post-processing to merge alternating line-number layouts (keep the number paragraph as an independent paragraph b; merge adjacent text paragraphs a and c across it when `layout_id` and `xobj_id` match, digits are ASCII and spaces only). Default: off.
- `--skip-form-render`: Skip form rendering (default: False). When enabled, PDF forms will not be rendered in the output.
- `--skip-curve-render`: Skip curve rendering (default: False). When enabled, PDF curves will not be rendered in the output.
- `--only-parse-generate-pdf`: Only parse PDF and generate output PDF without translation (default: False). This skips all translation-related processing including layout analysis, paragraph finding, style processing, and translation itself. Useful for testing PDF parsing and reconstruction functionality.
- `--remove-non-formula-lines`: Remove non-formula lines from paragraph areas (default: False). This removes decorative lines that are not part of formulas, while protecting lines in figure/table areas. Useful for cleaning up documents with decorative elements that interfere with text flow.
- `--non-formula-line-iou-threshold`: IoU threshold for detecting paragraph overlap when removing non-formula lines (default: 0.9). Higher values are more conservative and will remove fewer lines.
- `--figure-table-protection-threshold`: IoU threshold for protecting lines in figure/table areas when removing non-formula lines (default: 0.9). Higher values provide more protection for structural elements in figures and tables.

- `--rpc-doclayout`: RPC service host address for document layout analysis (default: None)
- `--working-dir`: Working directory for translation. If not set, use temp directory.
- `--no-auto-extract-glossary`: Disable automatic term extraction. If this flag is present, the step is skipped. Defaults to enabled.
- `--save-auto-extracted-glossary`: Save automatically extracted glossary to the specified file. If not set, the glossary will not be saved.

> [!TIP]
> - Both `--skip-clean` and `--dual-translate-first` may help improve compatibility with some PDF readers
> - `--disable-rich-text-translate` can also help with compatibility by simplifying translation input
> - However, using `--skip-clean` will result in larger file sizes
> - If you encounter any compatibility issues, try using `--enhance-compatibility` first
> - Use `--max-pages-per-part` for large documents to split them into smaller parts for translation and automatically merge them back.
> - Use `--skip-scanned-detection` to speed up processing when you know your document is not a scanned PDF.
> - Use `--ocr-workaround` to fill background for scanned PDF. (Current assumption: background is pure white, text is pure black, this option will also auto enable `--skip-scanned-detection`)

### Translation Service Options

- `--qps`: QPS (Queries Per Second) limit for translation service (default: 4)
- `--ignore-cache`: Ignore translation cache and force retranslation
- `--no-dual`: Do not output bilingual PDF files
- `--no-mono`: Do not output monolingual PDF files
- `--min-text-length`: Minimum text length to translate (default: 5)
- `--openai`: Use OpenAI for translation (default: False)
- `--custom-system-prompt`: Custom system prompt for translation.
- `--add-formula-placehold-hint`: Add formula placeholder hint for translation. (Currently not recommended, it may affect translation quality, default: False)
- `--disable-same-text-fallback`: Disable fallback translation when LLM output matches input text. (default: False)
- `--pool-max-workers`: Maximum number of worker threads for internal task processing pools. If not specified, defaults to QPS value. This parameter directly sets the worker count, replacing previous QPS-based dynamic calculations.
- `--no-auto-extract-glossary`: Disable automatic term extraction. If this flag is present, the step is skipped. Defaults to enabled.

> [!TIP]
>
> 1. Currently, only OpenAI-compatible LLM is supported. For more translator support, please use [PDFMathTranslate 2.0](https://github.com/PDFMathTranslate/PDFMathTranslate-next).
> 2. It is recommended to use models with strong compatibility with OpenAI, such as: `glm-4-flash`, `deepseek-chat`, etc.
> 3. Currently, it has not been optimized for traditional translation engines like Bing/Google, it is recommended to use LLMs.
> 4. You can use [litellm](https://github.com/BerriAI/litellm) to access multiple models.
> 5. `--custom-system-prompt`: It is mainly used to add the `/no_think` instruction of Qwen 3 in the prompt. For example: `--custom-system-prompt "/no_think You are a professional, authentic machine translation engine."`

### OpenAI Specific Options

- `--openai-model`: OpenAI model to use (default: gpt-4o-mini)
- `--openai-base-url`: Base URL for OpenAI API
- `--openai-api-key`: API key for OpenAI service
- `--enable-json-mode-if-requested`: Enable JSON mode for OpenAI requests (default: False)
- `--term-pool-max-workers`: Maximum number of worker threads dedicated to automatic term extraction. If not specified, this defaults to the value of `--pool-max-workers`, which itself defaults to the QPS value when unset.

> [!TIP]
>
> 1. This tool supports any OpenAI-compatible API endpoints. Just set the correct base URL and API key. (e.g. `https://xxx.custom.xxx/v1`)
> 2. For local models like Ollama, you can use any value as the API key (e.g. `--openai-api-key a`).

### Glossary Options

- `--glossary-files`: Comma-separated paths to glossary CSV files.
  - Each CSV file should have the columns: `source`, `target`, and an optional `tgt_lng`.
  - The `source` column contains the term in the original language.
  - The `target` column contains the term in the target language.
  - The `tgt_lng` column (optional) specifies the target language for that specific entry (e.g., "zh-CN", "en-US").
    - If `tgt_lng` is provided for an entry, that entry will only be loaded and used if its (normalized) `tgt_lng` matches the (normalized) overall target language specified by `--lang-out`. Normalization involves lowercasing and replacing hyphens (`-`) with underscores (`_`).
    - If `tgt_lng` is omitted for an entry, that entry is considered applicable for any `--lang-out`.
  - The name of each glossary (used in LLM prompts) is derived from its filename (without the .csv extension).
  - During translation, the system will check the input text against the loaded glossaries. If terms from a glossary are found in the current text segment, that glossary (with the relevant terms) will be included in the prompt to the language model, along with an instruction to adhere to it.

### Output Control

- `--output`, `-o`: Output directory for translated files. If not set, use current working directory.
- `--debug`: Enable debug logging level and export detailed intermediate results in `~/.cache/babeldoc/working`.
- `--report-interval`: Progress report interval in seconds (default: 0.1).

### General Options

- `--warmup`: Only download and verify required assets then exit (default: False)

### Offline Assets Management

- `--generate-offline-assets`: Generate an offline assets package in the specified directory. This creates a zip file containing all required models and fonts.
- `--restore-offline-assets`: Restore an offline assets package from the specified file. This extracts models and fonts from a previously generated package.

> [!TIP]
> 
> 1. Offline assets packages are useful for environments without internet access or to speed up installation on multiple machines.
> 2. Generate a package once with `babeldoc --generate-offline-assets /path/to/output/dir` and then distribute it.
> 3. Restore the package on target machines with `babeldoc --restore-offline-assets /path/to/offline_assets_*.zip`.
> 4. The offline assets package name cannot be modified because the file list hash is encoded in the name.
> 5. If you provide a directory path to `--restore-offline-assets`, the tool will automatically look for the correct offline assets package file in that directory.
> 6. The package contains all necessary fonts and models required for document processing, ensuring consistent results across different environments.
> 7. The integrity of all assets is verified using SHA3-256 hashes during both packaging and restoration.
> 8. If you're deploying in an air-gapped environment, make sure to generate the package on a machine with internet access first.

### Configuration File

- `--config`, `-c`: Configuration file path. Use the TOML format.

Example Configuration:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Python API

The current recommended way to call BabelDOC in Python is to call the `high_level.do_translate_async_stream` function of [pdf2zh next](https://pdf2zh-next.com/).

> [!WARNING]
> **All APIs of BabelDOC should be considered as internal APIs, and any direct use of BabelDOC is not supported.**

## Background

There are a lot projects and teams working on to make document editing and translating easier like:

- [mathpix](https://mathpix.com/)
- [Doc2X](https://doc2x.noedgeai.com/)
- [minerU](https://github.com/opendatalab/MinerU)
- [PDFMathTranslate](https://github.com/PDFMathTranslate/PDFMathTranslate)

There are also some solutions to solve specific parts of the problem like:

- [layoutreader](https://github.com/microsoft/unilm/tree/master/layoutreader): the read order of the text block in a pdf
- [Surya](https://github.com/surya-is/surya): the structure of the pdf

This project hopes to promote a standard pipeline and interface to solve the problem.

In fact, there are two main stages of a PDF parser or translator:

- **Parsing**: A stage of parsing means to get the structure of the pdf such as text blocks, images, tables, etc.
- **Rendering**: A stage of rendering means to render the structure into a new pdf or other format.

For a service like mathpix, it will parse the pdf into a structure may be in a XML format, and then render them using a single column reader order as [layoutreader](https://github.com/microsoft/unilm/tree/master/layoutreader) does. The bad news is that the original structure lost.

Some people will use Adobe PDF Parser because it will generate a Word document and it keeps the original structure. But it is somewhat expensive.
And you know, a pdf or word document is not a good format for reading in mobile devices.

We offer an intermediate representation of the results from parser and can be rendered into a new pdf or other format. The pipeline is also a plugin-based system which everybody can add their new model, ocr, renderer, etc.

## Roadmap

- [ ] Add line support
- [ ] Add table support
- [ ] Add cross-page/cross-column paragraph support
- [ ] More advanced typesetting features
- [ ] Outline support
- [ ] ...

Our first 1.0 version goal is to finish a translation from [PDF Reference, Version 1.7](https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/pdfreference1.7old.pdf) to the following language version:

- Simplified Chinese
- Traditional Chinese
- Japanese
- Spanish

And meet the following requirements:

- layout error less than 1%
- content loss less than 1%

## Version Number Explanation

This project uses a combination of [Semantic Versioning](https://semver.org/) and [Pride Versioning](https://pridever.org/). The version number format is: "0.MAJOR.MINOR".

> [!NOTE]
>
> The API compatibility here mainly refers to the compatibility with [pdf2zh_next](https://pdf2zh-next.com/).


- MAJOR: Incremented by 1 when API incompatible changes are made or when proud improvements are implemented.

- MINOR: Incremented by 1 when any API compatible changes are made.

## Known Issues

1. Parsing errors in the author and reference sections; they get merged into one paragraph after translation.
2. Lines are not supported.
3. Does not support drop caps.
4. Large pages will be skipped.

## How to Contribute

BabelDOC is currently developed in a maintainer-led mode. Bug reports,
reproducible PDFs, documentation fixes, and small compatibility fixes are
welcome. For changes to parsing, rendering, translation, or service integration
behavior, please open an issue for discussion before submitting a pull request.

Everyone interacting in BabelDOC and its sub-projects' codebases, issue trackers, chat rooms, and mailing lists is expected to follow the BabelDOC [Code of Conduct](https://github.com/funstory-ai/BabelDOC/blob/main/docs/CODE_OF_CONDUCT.md).

[Immersive Translation](https://immersivetranslate.com) sponsors monthly Pro membership redemption codes for active contributors to this project, see details at: [CONTRIBUTOR_REWARD.md](https://github.com/funstory-ai/BabelDOC/blob/main/docs/CONTRIBUTOR_REWARD.md)

## Acknowledgements

BabelDOC is still at an early stage of development, and many parts are not as
polished as we would like yet. We sincerely appreciate every bug report,
criticism, suggestion, reproducible PDF, downstream integration experience, and
contribution from people working on BabelDOC and related upstream/downstream
projects. We will keep iterating on BabelDOC, fixing bugs, and making it better
step by step.

- [PDFMathTranslate](https://github.com/PDFMathTranslate/PDFMathTranslate)
- [DocLayout-YOLO](https://github.com/opendatalab/DocLayout-YOLO)
- [pdfminer](https://github.com/pdfminer/pdfminer.six)
- [PyMuPDF](https://github.com/pymupdf/PyMuPDF)
- [Asynchronize](https://github.com/multimeric/Asynchronize/tree/master?tab=readme-ov-file)
- [PriorityThreadPoolExecutor](https://github.com/oleglpts/PriorityThreadPoolExecutor)

<h2 id="star_hist">Star History</h2>

<a href="https://star-history.com/#funstory-ai/babeldoc&Date">
 
   
   
   
 
</a>

> [!WARNING]
> **Important Interaction Note for `--auto-enable-ocr-workaround`:**
>
> When `--auto-enable-ocr-workaround` is set to `true` (either via command line or config file):
>
> 1.  During the initial setup, the values for `ocr_workaround` and `skip_scanned_detection` will be forced to `false` by `TranslationConfig`, regardless of whether you also set `--ocr-workaround` or `--skip-scanned-detection` flags.
> 2.  Then, during the scanned document detection phase (`DetectScannedFile` stage):
>     *   If the document is identified as heavily scanned (e.g., >80% scanned pages) AND `auto_enable_ocr_workaround` is `true` (i.e., `translation_config.auto_enable_ocr_workaround` is true), the system will then attempt to set both `ocr_workaround` to `true` and `skip_scanned_detection` to `true`.
>
> This means that `--auto-enable-ocr-workaround` effectively gives the system control to enable OCR processing for scanned documents, potentially overriding manual settings for `--ocr-workaround` and `--skip_scanned_detection` based on its detection results. If the document is *not* detected as heavily scanned, then the initial `false` values for `ocr_workaround` and `skip_scanned_detection` (forced by `--auto-enable-ocr-workaround` at the `TranslationConfig` initialization stage) will remain in effect unless changed by other logic.

---

## File: docs/ImplementationDetails/PDFParsing/PDFParsing.md

# PDF Parsing and Intermediate Layer Creation

> [!NOTE]
> This documentation may contain AI-generated content. While we strive for accuracy, there might be inaccuracies. Please report any issues via:
>
> - [GitHub Issues](https://github.com/funstory-ai/yadt/issues)
> - Community contribution (PRs welcome!)

## Background

The first step in the translation process is to parse the PDF document and create an intermediate layer (IL) representation. This step involves extracting text, styles, formulas, and layout information from the PDF while maintaining their relationships and properties.

## Goal

1. Extract text content while preserving character-level information
2. Maintain font and style information
3. Preserve document structure and layout
4. Handle special elements like XObjects and graphics
5. Create a structured intermediate representation for later processing

## Specific Implementation

The parsing process consists of several key components working together:

### Step 1: PDF Interpreter (PDFPageInterpreterEx)

1. Page content processing:
   - Parse PDF operators and their parameters
   - Handle graphics state operations
   - Process text and font operations
   - Manage XObject rendering

2. Graphics filtering:
   - Filter non-formula lines
   - Handle color space operations
   - Process stroke and fill operations

3. XObject handling:
   - Process form XObjects
   - Handle image XObjects
   - Maintain XObject hierarchy

### Step 2: PDF Converter (PDFConverterEx)

1. Character processing:
   - Extract character information
   - Maintain character positions
   - Preserve style attributes

2. Layout management:
   - Handle page boundaries
   - Process figure elements
   - Manage coordinate systems

3. Font handling:
   - Map font identifiers
   - Process font metadata
   - Handle CID fonts

### Step 3: Intermediate Layer Creator (ILCreater)

1. Document structure creation:
   - Build page hierarchy
   - Create character objects
   - Maintain font registry

2. Resource management:
   - Process font resources
   - Handle color spaces
   - Manage graphic states

3. XObject tracking:
   - Track XObject hierarchy
   - Maintain XObject states
   - Process form content

### Step 4: High-level Coordination

1. Process management:
   - Initialize resources
   - Coordinate component interactions
   - Handle progress tracking

2. Resource initialization:
   - Set up font management
   - Initialize graphics resources
   - Prepare document structure

3. Error handling:
   - Handle malformed content
   - Manage resource errors
   - Provide debug information

## Additional Features

1. Font management:
   - Support for CID fonts
   - Font metadata extraction
   - Font mapping capabilities

2. Graphics state tracking:
   - Color space management
   - Line style preservation
   - Transparency handling

3. Coordinate system handling:
   - Support for transformations
   - Boundary box calculations
   - Position normalization

4. Debug support:
   - Detailed logging
   - Intermediate file generation
   - Progress tracking

## Limitations

1. Complex PDF features:
   - Limited support for some PDF extensions
   - Simplified graphics model
   - Basic transparency support

2. Font handling:
   - Limited support for some font formats
   - Simplified font metrics
   - Basic font feature support

3. Performance considerations:
   - Memory usage for large documents
   - Processing time for complex layouts
   - Resource management overhead

## Configuration Options

The parsing process can be customized through `TranslationConfig`:

1. `debug`: Enable/disable debug mode and intermediate file generation
2. Font-related settings:
   - Font mapping configurations
   - CID font handling options
3. Layout processing options:
   - Page selection
   - Content filtering rules

---

## File: docs/ImplementationDetails/PDFCreation/PDFCreation.md

# PDF Creation

> [!NOTE]
> This documentation may contain AI-generated content. While we strive for accuracy, there might be inaccuracies. Please report any issues via:
>
> - [GitHub Issues](https://github.com/funstory-ai/yadt/issues)
> - Community contribution (PRs welcome!)

## Background

After translation and typesetting, we need to create the final PDF document that preserves all the formatting, styles, and layout of the original document while containing the translated text. The PDF creation process handles this final step.

## Goal

1. Create a new PDF document with translated content
2. Preserve all original formatting and styles
3. Support both monolingual and dual-language output
4. Maintain font consistency and character encoding
5. Optimize the output file size and performance

## Specific Implementation

The PDF creation process consists of several key steps:

### Step 1: Font Management

1. Font initialization:
   - Add required fonts to the document
   - Map font identifiers
   - Handle font encoding lengths

2. Font availability checking:
   - Check available fonts for each page
   - Handle XObject font requirements
   - Manage font resources

3. Font subsetting:
   - Optimize font usage
   - Reduce file size
   - Maintain character support

### Step 2: Content Rendering

1. Character processing:
   - Handle individual characters
   - Process character encodings
   - Manage character positioning

2. Graphics state handling:
   - Process color spaces
   - Handle transparency
   - Manage graphic state instructions

3. XObject management:
   - Process form XObjects
   - Handle drawing operations
   - Maintain XObject hierarchy

### Step 3: Document Assembly

1. Page construction:
   - Build page content
   - Process page resources
   - Handle page boundaries

2. Content stream creation:
   - Generate drawing operations
   - Handle text positioning
   - Manage content streams

3. Resource management:
   - Handle font resources
   - Manage XObject resources
   - Process graphic states

### Step 4: Output Generation

1. Monolingual output:
   - Create translated-only PDF
   - Optimize file size
   - Apply compression

2. Dual-language output:
   - Combine original and translated pages
   - Handle page ordering
   - Maintain document structure

3. File optimization:
   - Apply garbage collection
   - Enable compression
   - Optimize for linear reading

## Additional Features

1. Font handling:
   - Support for CID fonts
   - Font subsetting
   - Font resource management

2. Document optimization:
   - File size reduction
   - Performance optimization
   - Resource cleanup

3. Debug support:
   - Decompressed output
   - Debug information
   - Progress tracking

## Limitations

1. Font support:
   - Limited to available font formats
   - Font subsetting restrictions
   - Character encoding constraints

2. File size:
   - Dual-language output increases size
   - Font embedding impact
   - Resource duplication

3. Performance considerations:
   - Processing time for large documents
   - Memory usage during creation
   - Optimization overhead

## Configuration Options

The PDF creation process can be customized through `TranslationConfig`:

1. Output options:
   - `no_mono`: Disable monolingual output
   - `no_dual`: Disable dual-language output
   - Output file naming patterns

2. Optimization settings:
   - Compression options
   - Garbage collection
   - Font subsetting

3. Debug options:
   - Debug mode
   - Decompressed output
   - Progress tracking

---

## File: docs/ImplementationDetails/ParagraphFinding/ParagraphFinding.md

# Paragraph Finding

> [!NOTE]
> This documentation may contain AI-generated content. While we strive for accuracy, there might be inaccuracies. Please report any issues via:
>
> - [GitHub Issues](https://github.com/funstory-ai/yadt/issues)
> - Community contribution (PRs welcome!)

## Background

After PDF analysis, we need to identify paragraphs from individual characters. This is a crucial step before translation and typesetting, as it helps maintain the logical structure of the document.

## Goal

1. Group characters into meaningful paragraphs while preserving the document's logical structure
2. Handle special cases like table of contents, short lines, and multi-line paragraphs
3. Maintain layout information for later typesetting

## Specific Implementation

The paragraph finding process consists of four main steps:

### Step 1: Create Initial Paragraphs

1. Group characters into lines based on their spatial relationships
2. Create paragraphs based on layout information and XObject IDs
3. Characters that don't belong to text layouts are skipped

### Step 2: Process Paragraph Spacing

1. Remove completely empty lines
2. Handle trailing spaces within lines
3. Update paragraph boundary boxes and metadata

### Step 3: Calculate Line Width Statistics

1. Calculate the median width of all lines
2. This information is used for identifying potential paragraph breaks

### Step 4: Process Independent Paragraphs

1. Analyze paragraphs with multiple lines
2. Split paragraphs in two cases:
   - When encountering table of contents entries (identified by consecutive dots)
   - When finding lines significantly shorter than the median width (configurable via `short_line_split_factor`)

## Additional Features

1. Layout-aware processing:
   - Respects different layout types (plain text, title, figure caption, etc.)
   - Maintains layout priority order for overlapping regions

2. First line indent detection:
   - Automatically detects and marks paragraphs with first line indentation

3. Flexible character position detection:
   - Uses multiple position detection modes (middle, topleft, bottomright)
   - Special handling for characters with unreliable height information

## Limitations

1. The current implementation assumes left-to-right text direction

2. May not perfectly handle complex layouts with overlapping regions

3. Table of contents detection relies on consecutive dots pattern

4. Short line splitting might occasionally create incorrect paragraph breaks

## Configuration Options

The paragraph finding behavior can be customized through `TranslationConfig`:

1. `split_short_lines`: Enable/disable splitting paragraphs at short lines
2. `short_line_split_factor`: Threshold factor for short line detection (relative to median width)

---

## File: docs/ImplementationDetails/ILTranslator/ILTranslator.md

# Intermediate Layer Translator

> [!NOTE]
> This documentation may contain AI-generated content. While we strive for accuracy, there might be inaccuracies. Please report any issues via:
>
> - [GitHub Issues](https://github.com/funstory-ai/yadt/issues)
> - Community contribution (PRs welcome!)

## Background

After formula and style processing, we need to translate the document while preserving all formatting, formulas, and styles. The intermediate layer translator handles this complex task by using placeholders and style preservation techniques.

## Goal

1. Translate text while preserving document structure
2. Maintain formulas and special formatting
3. Handle rich text with different styles
4. Support concurrent translation for better performance

## Specific Implementation

The translation process consists of several key steps:

### Step 1: Translation Preparation

1. Process paragraphs:
   - Skip vertical text
   - Handle single-component paragraphs directly
   - Process multi-component paragraphs with placeholders

2. Create placeholders:
   - Formula placeholders for mathematical expressions
   - Rich text placeholders for styled text
   - Ensure placeholder uniqueness within each paragraph

### Step 2: Translation Input Creation

1. Analyze paragraph components:
   - Regular text components
   - Formula components
   - Styled text components

2. Handle special cases:
   - Skip pure formula paragraphs
   - Preserve original text when style matches base style
   - Handle font mapping cases

### Step 3: Translation Execution

1. Concurrent translation:
   - Use thread pool for parallel processing
   - Control QPS (Queries Per Second)
   - Track translation progress

2. Translation tracking:
   - Record original text
   - Record translated text
   - Save tracking information for debugging

### Step 4: Translation Output Processing

1. Parse translated text:
   - Extract text between placeholders
   - Restore formulas at placeholder positions
   - Restore rich text with original styles

2. Create new paragraph components:
   - Maintain style information
   - Preserve formula positioning
   - Handle empty text segments

## Additional Features

1. Style preservation:
   - Maintains original text styles
   - Handles font size variations
   - Preserves formatting attributes

2. Formula handling:
   - Preserves formula integrity
   - Maintains formula positioning
   - Supports complex mathematical expressions

3. Debug support:
   - Translation tracking
   - JSON output for debugging
   - Detailed logging

## Limitations

1. Vertical text is not supported

2. Complex nested styles might not be perfectly preserved

3. Placeholder conflicts could occur in rare cases

4. Translation quality depends on external translation engine

## Configuration Options

The translation process can be customized through `TranslationConfig`:

1. `qps`: Maximum queries per second for translation
2. `debug`: Enable/disable debug mode and tracking
3. Translation engine specific settings

---

## File: docs/ImplementationDetails/AsyncTranslate/AsyncTranslate.md

# Async Translation API

> [!NOTE]
> This documentation may contain AI-generated content. While we strive for accuracy, there might be inaccuracies. Please report any issues via:
>
> - [GitHub Issues](https://github.com/funstory-ai/yadt/issues)
> - Community contribution (PRs welcome!)

## Overview

The `yadt.high_level.async_translate` function provides an asynchronous interface for translating PDF files with real-time progress reporting. This function yields progress events that can be used to update progress bars or other UI elements.

## Usage

```python linenums="1"
async def translate_with_progress():
    config = TranslationConfig(
        input_file="example.pdf",
        translator=your_translator,
        # ... other configuration options
    )
    
    try:
        async for event in async_translate(config):
            if event["type"] == "progress_update":
                print(f"Progress: {event['overall_progress']}%")
            elif event["type"] == "finish":
                result = event["translate_result"]
                print(f"Translation completed: {result.original_pdf_path}")
            elif event["type"] == "error":
                print(f"Error occurred: {event['error']}")
                break
    except asyncio.CancelledError:
        print("Translation was cancelled")
    except KeyboardInterrupt:
        print("Translation was interrupted")
```

## Event Types

The function yields different types of events during the translation process:

### 1. Progress Start Event

Emitted when a translation stage begins:

```python
{
    "type": "progress_start",
    "stage": str,              # Name of the current stage
    "stage_progress": float,   # Always 0.0
    "stage_current": int,      # Current progress count (0)
    "stage_total": int         # Total items to process in this stage
}
```

### 2. Progress Update Event

Emitted periodically during translation (controlled by report_interval, default 0.1s):

```python
{
    "type": "progress_update",
    "stage": str,              # Name of the current stage
    "stage_progress": float,   # Progress percentage of current stage (0-100)
    "stage_current": int,      # Current items processed in this stage
    "stage_total": int,        # Total items to process in this stage
    "overall_progress": float  # Overall translation progress (0-100)
}
```

### 3. Progress End Event

Emitted when a stage completes:

```python
{
    "type": "progress_end",
    "stage": str,              # Name of the completed stage
    "stage_progress": float,   # Always 100.0
    "stage_current": int,      # Equal to stage_total
    "stage_total": int,        # Total items processed in this stage
    "overall_progress": float  # Overall translation progress (0-100)
}
```

### 4. Finish Event

Emitted when translation completes successfully:

```python
{
    "type": "finish",
    "translate_result": TranslateResult  # Contains paths to translated files and timing info
}
```

### 5. Error Event

Emitted if an error occurs during translation:

```python
{
    "type": "error",
    "error": str  # Error message
}
```

## Translation Stages

The translation process goes through the following stages in order:

1. ILCreater
2. LayoutParser
3. ParagraphFinder
4. StylesAndFormulas
5. ILTranslator
6. Typesetting
7. FontMapper
8. PDFCreater

Each stage will emit its own set of progress events.

## Cancellation

The translation process can be cancelled in several ways:

1. By raising a `CancelledError` (e.g., when using `asyncio.Task.cancel()`)
2. Through `KeyboardInterrupt` (e.g., when user presses Ctrl+C)
3. By calling `translation_config.cancel_translation()` method

Example of programmatic cancellation:

```python linenums="1"
async def translate_with_cancellation():
    config = TranslationConfig(
        input_file="example.pdf",
        translator=your_translator,
        # ... other configuration options
    )
    
    try:
        # Start translation in another task
        translation_task = asyncio.create_task(process_translation(config))
        
        # Simulate some condition that requires cancellation
        await asyncio.sleep(5)
        config.cancel_translation()  # This will trigger cancellation
        
        await translation_task  # Wait for the task to finish
    except asyncio.CancelledError:
        print("Translation was cancelled")

async def process_translation(config):
    async for event in async_translate(config):
        if event["type"] == "error":
            if isinstance(event["error"], asyncio.CancelledError):
                print("Translation was cancelled")
                break
            print(f"Error occurred: {event['error']}")
            break
        # ... handle other events ...
```

When cancelled:
- The function will log the cancellation reason
- All resources will be cleaned up properly
- Any ongoing translation tasks will be stopped
- A final error event with `CancelledError` will be emitted
- The function will exit gracefully

## Error Handling

Any errors during translation will be:
1. Logged with full traceback (if debug mode is enabled)
2. Reported through an error event
3. Cause the event stream to stop after the error event
4. Clean up resources properly before exiting

It's recommended to handle these events appropriately in your application to provide feedback to users. The example in the Usage section shows a basic error handling pattern.

---

## File: docs/ImplementationDetails/StylesAndFormulas/StylesAndFormulas.md

# Styles and Formulas Processing

> [!NOTE]
> This documentation may contain AI-generated content. While we strive for accuracy, there might be inaccuracies. Please report any issues via:
>
> - [GitHub Issues](https://github.com/funstory-ai/yadt/issues)
> - Community contribution (PRs welcome!)

## Background

After paragraph finding, we need to identify formulas and text styles within each paragraph. This step is crucial for maintaining mathematical expressions and text formatting during translation.

## Goal

1. Identify and preserve mathematical formulas
2. Detect and maintain consistent text styles
3. Handle special cases like subscripts and superscripts
4. Calculate proper offsets for formula positioning

## Specific Implementation

The processing consists of several main steps:

### Step 1: Formula Detection

1. Identify formula characters based on:
   - Formula-specific fonts
   - Special Unicode characters
   - Vertical text
   - Corner marks (subscripts/superscripts)

2. Group consecutive formula characters into formula units

### Step 2: Formula Processing

1. Process comma-containing formulas:
   - Split complex formulas at commas when appropriate
   - Preserve brackets and their contents
   - Convert simple number-only formulas to regular text

2. Merge overlapping formulas:
   - Handle cases where subscripts/superscripts are detected as separate formulas
   - Maintain proper character ordering

### Step 3: Style Analysis

1. Calculate base style for each paragraph:
   - Find common style attributes across all text
   - Handle font variations
   - Process graphic states

2. Group characters with identical styles:
   - Font properties
   - Size properties
   - Graphic state properties

### Step 4: Position Calculation

1. Calculate formula offsets:
   - Compute x-offset relative to surrounding text
   - Compute y-offset for proper vertical alignment
   - Handle line spacing variations

## Additional Features

1. Font mapping:
   - Maps different fonts to standard ones
   - Special handling for formula fonts

2. Style inheritance:
   - Maintains style hierarchy
   - Handles partial style overrides

3. Formula classification:
   - Distinguishes between translatable and non-translatable formulas
   - Special handling for numeric formulas with commas

## Limitations

1. Formula detection relies on font and character patterns

2. May not handle all types of mathematical notations

3. Complex subscript/superscript combinations might be misidentified

4. Limited support for vertical formulas

## Configuration Options

The formula and style processing can be customized through `TranslationConfig`:

1. `formular_font_pattern`: Regex pattern for identifying formula fonts
2. `formular_char_pattern`: Regex pattern for identifying formula characters

---

## File: docs/ImplementationDetails/Typesetting/Typesetting.md

# Typography

> [!NOTE]
> This documentation may contain AI-generated content. While we strive for accuracy, there might be inaccuracies. Please report any issues via:
>
> - [GitHub Issues](https://github.com/funstory-ai/yadt/issues)
> - Community contribution (PRs welcome!)

## Background

After translation, text needs to be typeset before placing into PDF.

Translated paragraphs can contain any combination of the following types:

1. PDF formulas

2. Single PDF original character

3. PDF original string with same style

4. Translated Unicode string with same style

Let's discuss different cases:

For the following 3 types, they can be directly transmitted transparently to new positions:

1. PDF formulas

2. Single PDF original character

3. PDF original string with same style

Only "translated Unicode string with same style" needs typesetting operation, as this step loses original layout information. However, since paragraphs may contain other components that need transparent transmission, their positions may also change and need to participate in typesetting.

## Goal

Try to fit all components within the original paragraph bounding box. If impossible, try to expand the bounding box in writing direction.

## Specific Implementation

First perform reflow judgment to determine if the paragraph needs reflow. If all elements can be transmitted transparently, no reflow is needed. Then, if reflow is needed, execute Algorithm 1:

1. Convert all elements to typesetting unit type, which records length and width information.

2. Start from top-left of original paragraph bounding box, place elements sequentially.

3. If current line cannot fit next element, wrap to next line.

4. Repeat 2-3 until all elements are placed or exceed original bounding box.

Algorithm 1 works normally when translated text is shorter than original. When translated text is longer, Algorithm 2 needs to be added:

1. Initialize element scaling factor as 1.0.

2. Initialize line spacing as 1.5.

3. Try typesetting using Algorithm 1.

4. If it cannot fit all elements:

   - First try to reduce line spacing by 0.1 step until reaching minimum line spacing (1.4)
   - If still cannot fit:
     - When scale > 0.6, reduce element scaling by 0.05
     - When scale <= 0.6, reduce element scaling by 0.1
     - Reset line spacing to 1.5
   - When scale becomes less than 0.7, adjust minimum line spacing to 1.1

5. Report error if element scaling is less than 0.1.

Algorithm 2 can fit translations of almost all languages in original position.

However, for special cases like "图 1" translated to "Figure 1", even with the above algorithms some text may still overflow. So Algorithm 3:

1. Before reducing scale, first try to expand the bounding box in writing direction.

2. Calculate paragraph's right whitespace by:

   - Using 90% of page crop box width as maximum limit
   - Checking for overlapping paragraphs on the right
   - Checking for overlapping figures on the right

3. Expand paragraph bounding box based on available whitespace.

4. If still cannot fit all elements, continue with scale reduction as in Algorithm 2.

## Additional Features

1. Mixed Chinese-English text handling:
   - Adds 0.5 character width spacing between Chinese and English text transitions
   - Excludes certain punctuation marks from this spacing rule
2. First line indent:

   - Adds 2 Chinese characters width indent for the first line when specified

3. Hanging punctuation:
   - Allows certain punctuation marks to extend beyond the right margin
   - Helps maintain better visual alignment

## Limitations

1. Currently, we use PDFPlumber for PDF analysis, this is only implemented for paragraphs, only handles left-to-right writing.

2. Cannot handle table of contents alignment by dots.

3. Poor performance, needs optimization.

4. No global page information consideration, inconsistent text sizes.

5. No advanced typography features, poor reading experience.

## Related Resources

[UTR #59: East Asian Spacing](https://www.unicode.org/reports/tr59/) specifies which characters need spacing between them.

---

## File: docs/ImplementationDetails/README.md

# Implementation Details

> [!NOTE]
> This documentation may contain AI-generated content. While we strive for accuracy, there might be inaccuracies. Please report any issues via:
>
> - [GitHub Issues](https://github.com/funstory-ai/yadt/issues)
> - Community contribution (PRs welcome!)

## Core Processing Flow

Main processing stages in order of actual execution and corresponding documentation:

1. [PDFParser.md](PDFParsing/PDFParsing.md): **PDF Parsing and Intermediate Layer Creation**

2. [LayoutParser](https://github.com/funstory-ai/yadt/blob/main/yadt/document_il/midend/layout_parser.py): **Layout OCR**

3. [ParagraphFinding.md](ParagraphFinding/ParagraphFinding.md): **Paragraph Recognition**

4. [StylesAndFormulas.md](StylesAndFormulas/StylesAndFormulas.md): **Style and Formula Processing**

5. [ILTranslator.md](ILTranslator/ILTranslator.md): **Intermediate Layer Translation**

6. [Typesetting.md](Typesetting/Typesetting.md): **Typesetting Processing**

7. [FontMapper](https://github.com/funstory-ai/yadt/blob/main/yadt/document_il/utils/fontmap.py): **Font Mapping**

8. [PDFCreation.md](PDFCreation/PDFCreation.md): **PDF Generation**

## API

1. [Async Translation API](AsyncTranslate/AsyncTranslate.md): **Async Translation API**

> [!TIP]
>
> Click on document links to view detailed implementation principles and configuration options