# Technical Documentation: allenai/olmocr > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/allenai/olmocr) ### Overview (`README.md`)
olmocr-2-full@2x

GitHub License GitHub release Tech Report v1 Tech Report v2 Demo Discord

A toolkit for converting PDFs and other image-based document formats into clean, readable, plain text format. Try the online demo: [https://olmocr.allenai.org/](https://olmocr.allenai.org/) Features: - Convert PDF, PNG, and JPEG based documents into clean Markdown - Support for equations, tables, handwriting, and complex formatting - Automatically removes headers and footers - Convert into text with a natural reading order, even in the presence of figures, multi-column layouts, and insets - Efficient, less than $200 USD per million pages converted - (Based on a 7B parameter VLM, so it requires a GPU) ### News - October 21, 2025 - v0.4.0 - [New model release](https://huggingface.co/allenai/olmOCR-2-7B-1025-FP8), boosts olmOCR-bench score by ~4 points using synthetic data and introduces RL training. - August 13, 2025 - v0.3.0 - [New model release](https://huggingface.co/allenai/olmOCR-7B-0825-FP8), fixes auto-rotation detection, and hallucinations on blank documents. - July 24, 2025 - v0.2.1 - [New model release](https://huggingface.co/allenai/olmOCR-7B-0725-FP8), scores 3 points higher on [olmOCR-Bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench), also runs significantly faster because it's default FP8, and needs much fewer retries per document. - July 23, 2025 - v0.2.0 - New cleaned up [trainer code](https://github.com/allenai/olmocr/tree/main/olmocr/train), makes it much simpler to train olmOCR models yourself. - June 17, 2025 - v0.1.75 - Switch from sglang to vllm based inference pipeline, updated docker image to CUDA 12.8. - May 23, 2025 - v0.1.70 - Official docker support and images are now available! [See Docker usage](#using-docker) - May 19, 2025 - v0.1.68 - [olmOCR-Bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench) launch, scoring 77.4. Launch includes 2 point performance boost in olmOCR pipeline due to bug fixes with prompts. - Mar 17, 2025 - v0.1.60 - Performance improvements due to better temperature selection in sampling. - Feb 25, 2025 - v0.1.58 - Initial public launch and demo. ### Benchmark [**olmOCR-Bench**](https://github.com/allenai/olmocr/tree/main/olmocr/bench): We also ship a comprehensive benchmark suite covering over 7,000 test cases across 1,400 documents to help measure performance of OCR systems.
ArXiv Old
scans
math		 Tables Old
scans		 Headers
&
footers		 Multi
column		 Long
tiny
text		 Base Overall
Mistral OCR API 77.2 67.5 60.6 29.3 93.6 71.3 77.1 99.4 72.0±1.1
Marker 1.10.1 83.8 66.8 72.9 33.5 86.6 80.0 85.7 99.3 76.1±1.1
MinerU 2.5.4* 76.6 54.6 84.9 33.7 96.6 78.2 83.5 93.7 75.2±1.1
DeepSeek-OCR 77.2 73.6 80.2 33.3 96.1 66.4 79.4 99.8 75.7±1.0
Nanonets-OCR2-3B 75.4 46.1 86.8 40.9 32.1 81.9 93.0 99.6 69.5±1.1
PaddleOCR-VL* 85.7 71.0 84.1 37.8 97.0 79.9 85.7 98.5 80.0±1.0
Infinity-Parser 7B* 84.4 83.8 85.0 47.9 88.7 84.2 86.4 99.8 82.5±?
Chandra OCR 0.1.0* 82.2 80.3 88.0 50.4 90.8 81.2 92.3 99.9 83.1±0.9
olmOCR v0.4.0 83.0 82.3 84.9 47.7 96.1 83.7 81.9 99.7 82.4±1.1
### Installation #### System Dependencies You will need to install poppler-utils and additional fonts for rendering PDF images. Install dependencies (Ubuntu/Debian): ```bash sudo apt-get update sudo apt-get install poppler-utils ttf-mscorefonts-installer msttcorefonts fonts-crosextra-caladea fonts-crosextra-carlito gsfonts lcdf-typetools ``` #### Python Installation Set up a conda environment and install olmocr. The requirements for running olmOCR are difficult to install in an existing python environment, so please do make a clean python environment to install into. ```bash conda create -n olmocr python=3.11 conda activate olmocr ``` Choose the installation option that matches your use case: **Option 1: Remote Inference (Lightweight)** If you plan to use a remote vLLM server with the `--server` flag, install the base package: ```bash pip install olmocr ``` This avoids installing heavy GPU dependencies like PyTorch (~2GB+). **Option 2: Local GPU Inference** Requirements: - Recent NVIDIA GPU (tested on RTX 4090, L40S, A100, H100) with at least 12 GB of GPU RAM - 30GB of free disk space For running inference with your own GPU: ```bash pip install olmocr[gpu] --extra-index-url https://download.pytorch.org/whl/cu128 # Recommended: Install flash infer for faster inference on GPU pip install https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.5%2Bcu128torch2.7-cp38-abi3-linux_x86_64.whl ``` **Option 3: Beaker Cluster Execution** For submitting jobs to Beaker clusters with the `--beaker` flag: ```bash pip install olmocr[beaker] ``` **Option 4: Benchmark Suite** For running the olmOCR benchmark suite: ```bash pip install olmocr[bench] ``` **Combined Installation** You can combine multiple options: ```bash # GPU + Beaker support pip install olmocr[gpu,beaker] --extra-index-url https://download.pytorch.org/whl/cu128 # GPU + Benchmark support pip install olmocr[gpu,bench] --extra-index-url https://download.pytorch.org/whl/cu128 ``` **Troubleshooting** If you run into errors about `too many open files`, update your ulimit: ```bash ulimit -n 65536 ``` ### Usage Examples For quick testing, try the [web demo](https://olmocr.allen.ai/). **Convert a Single PDF (Local GPU):** ```bash # Download a sample PDF curl -o olmocr-sample.pdf https://olmocr.allenai.org/papers/olmocr_3pg_sample.pdf # Convert it to markdown olmocr ./localworkspace --markdown --pdfs olmocr-sample.pdf ``` **Convert an Image file:** ```bash olmocr ./localworkspace --markdown --pdfs random_page.png ``` **Convert Multiple PDFs:** ```bash olmocr ./localworkspace --markdown --pdfs tests/gnarly_pdfs/*.pdf ``` **Use Remote Inference Server:** ```bash olmocr ./localworkspace --server http://remote-server:8000/v1 --model allenai/olmOCR-2-7B-1025-FP8 --markdown --pdfs *.pdf ``` With the `--markdown` flag, results will be stored as markdown files inside of `./localworkspace/markdown/`. > **Note:** You can also use `python -m olmocr.pipeline` instead of `olmocr` if you prefer. #### Viewing Results The `./localworkspace/` workspace folder will then have both [Dolma](https://github.com/allenai/dolma) and markdown files (if using `--markdown`). ```bash cat localworkspace/markdown/olmocr-sample.md ``` ``` olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models ... ``` ### Using an Inference Provider or External Server If you have a vLLM server already running elsewhere (or any inference platform implementing the OpenAI API), you can point olmOCR to use it instead of spawning a local instance. **Installation for Remote Inference:** ```bash # Lightweight installation - no GPU dependencies needed pip install olmocr ``` **Using an External Server:** ```bash # Use external vLLM server instead of local one olmocr ./localworkspace --server http://remote-server:8000/v1 --model allenai/olmOCR-2-7B-1025-FP8 --markdown --pdfs tests/gnarly_pdfs/*.pdf ``` The served model name in vLLM needs to match the value provided in `--model`. **Example vLLM Server Launch:** ```bash vllm serve allenai/olmOCR-2-7B-1025-FP8 --max-model-len 16384 ``` #### Verified External Providers We have tested `olmOCR-2-7B-1025-FP8` on these external model providers and confirmed that they work | | $/1M Input tokens | $/1M Output tokens | Example Command | |-----------------------------------------------------------------------------|-------------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Cirrascale](https://ai2endpoints.cirrascale.ai/models/overview) | $0.07 | $0.15 | `olmocr ./workspace --server https://ai2endpoints.cirrascale.ai/api --api_key sk-XXXXXXX --workers 1 --max_concurrent_requests 20 --model olmOCR-2-7B-1025 --pdfs tests/gnarly_pdfs/*.pdf` | | [DeepInfra](https://deepinfra.com/) | $0.09 | $0.19 | `olmocr ./workspace --server https://api.deepinfra.com/v1/openai --api_key DfXXXXXXX --workers 1 --max_concurrent_requests 20 --model allenai/olmOCR-2-7B-1025 --pdfs tests/gnarly_pdfs/*.pdf` | | [Parasail](https://www.saas.parasail.io/serverless?name=olmocr-7b-1025-fp8) | $0.10 | $0.20 | `olmocr ./workspace --server https://api.parasail.io/v1 --api_key psk-XXXXX --workers 1 --max_concurrent_requests 20 --model allenai/olmOCR-2-7B-1025 --pdfs tests/gnarly_pdfs/*.pdf` | Notes on arguments - `--server`: Defines the OpenAI-compatible endpoint: ex `https://api.deepinfra.com/v1/openai` - `--api_key`: Your API key, bassed in via Authorization Bearer HTTP header - `--max_concurrent_requests`: Max concurrent requests that will be in-flight to the inference provider at one time - `--workers`: Max number of page groups that will be processed at once. You may want to set this to `1` so that you finish one group of stuff before moving on. - `--pages_per_group`: You may want a smaller number of pages per group as many external provides have lower concurrent request limits - `--model`: The model identifier, ex. `allenai/olmOCR-2-7B-1025`, different providers have different names, and if you run locally, you can use `olmocr` - Other arguments work the same as with local inference ### Multi-node / Cluster Usage If you want to convert millions of PDFs using multiple nodes running in parallel, olmOCR supports reading PDFs from AWS S3 and coordinating work using an AWS S3 output bucket. **Start the first worker node:** ```bash olmocr s3://my_s3_bucket/pdfworkspaces/exampleworkspace --pdfs s3://my_s3_bucket/jakep/gnarly_pdfs/*.pdf ``` This sets up a simple work queue in your AWS bucket and starts converting PDFs. **On subsequent worker nodes:** ```bash olmocr s3://my_s3_bucket/pdfworkspaces/exampleworkspace ``` They will automatically start grabbing items from the same workspace queue. #### Using Beaker for Cluster Execution If you are at Ai2 and want to linearize millions of PDFs efficiently using [beaker](https://www.beaker.org), install with Beaker support: ```bash pip install olmocr[gpu,beaker] --extra-index-url https://download.pytorch.org/whl/cu128 ``` Then use the `--beaker` flag to prepare the workspace locally and launch N GPU workers in the cluster: ```bash olmocr s3://my_s3_bucket/pdfworkspaces/exampleworkspace --pdfs s3://my_s3_bucket/jakep/gnarly_pdfs/*.pdf --beaker --beaker_gpus 4 ``` ### Using Docker Pull the Docker image (large, includes the model, ~30GB): ```bash docker pull alleninstituteforai/olmocr:latest-with-model ``` For advanced users who want to manage their own model downloads, we also provide a base image without the model: ```bash docker pull alleninstituteforai/olmocr:latest ``` #### Quick Start - Process PDFs Process a single PDF in your current directory: ```bash docker run --gpus all \ -v $(pwd):/workspace \ alleninstituteforai/olmocr:latest-with-model \ -c "olmocr /workspace/output --markdown --pdfs /workspace/sample.pdf" ``` Process multiple PDFs: ```bash docker run --gpus all \ -v /path/to/pdfs:/input \ -v /path/to/output:/output \ alleninstituteforai/olmocr:latest-with-model \ -c "olmocr /output --markdown --pdfs /input/*.pdf" ``` #### Interactive Mode Run the container interactively for exploration and debugging: ```bash docker run -it --gpus all alleninstituteforai/olmocr:latest-with-model ``` > Visit our Docker repository on [Docker Hub](https://hub.docker.com/r/alleninstituteforai/olmocr) for more information. ### Full Documentation To see all available options: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Code overview There are some nice reusable pieces of the code that may be useful for your own projects: - A prompting strategy to get really good natural text parsing using ChatGPT 4o - [buildsilver.py](https://github.com/allenai/olmocr/blob/main/olmocr/data/buildsilver.py) - Basic filtering by language and SEO spam removal - [filter.py](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py) - SFT Finetuning code for Qwen2.5-VL - [train.py](https://github.com/allenai/olmocr/blob/main/olmocr/train/train.py) - GRPO RL Trainer - [grpo_train.py](https://github.com/allenai/olmocr/blob/main/olmocr/train/grpo_train.py) - Synthetic data generation - [mine_html_templates.py](https://github.com/allenai/olmocr/blob/main/olmocr/synth/mine_html_templates.py) - Processing millions of PDFs through a finetuned model using VLLM - [pipeline.py](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) - Viewing [Dolma docs](https://github.com/allenai/dolma) created from PDFs - [dolmaviewer.py](https://github.com/allenai/olmocr/blob/main/olmocr/viewer/dolmaviewer.py) ## Team **olmOCR** is developed and maintained by the AllenNLP team, backed by [the Allen Institute for Artificial Intelligence (AI2)](https://allenai.org/). AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering. To learn more about who specifically contributed to this codebase, see [our contributors](https://github.com/allenai/olmocr/graphs/contributors) page. ## License **olmOCR** is licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0). A full copy of the license can be found [on GitHub](https://github.com/allenai/olmocr/blob/main/LICENSE). ## Citing For olmOCR v1 and OlmOCR-bench: ```bibtex @misc{olmocrbench, title={{olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models}}, author={Jake Poznanski and Jon Borchardt and Jason Dunkelberger and Regan Huff and Daniel Lin and Aman Rangapur and Christopher Wilhelm and Kyle Lo and Luca Soldaini}, year={2025}, eprint={2502.18443}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2502.18443}, } ``` For olmOCR v2 Unit Testing Rewards with RL: ```bibtex @misc{olmocr2, title={olmOCR 2: Unit Test Rewards for Document OCR}, author={Jake Poznanski and Luca Soldaini and Kyle Lo}, year={2025}, eprint={2510.19817}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2510.19817}, } ``` --- ### Overview (`olmocr/bench/README.md`) # olmOCR-Bench Dataset Link: https://huggingface.co/datasets/allenai/olmOCR-bench We develop olmOCR-Bench in order to automatically and effectively evaluate document-level OCR of various tools. olmOCR-Bench works by testing various "facts" about document pages at the PDF-level. Our intention is that each "fact" is very simple, unambiguous, and machine-checkable, similar to a unit test. For example, once your document has been OCRed, we may check that a particular sentence appears exactly somewhere on the page. We stay away from soft metrics like edit distance comparisons, because they may assign lower scores for parses of the document that differ from the reference, but may in fact still be correct. For example, on a document containing multiple distinct articles: you want the text of each article to be grouped together, but the relative order of the two articles may not be critical. Also, some documents may have critical details, like switching x and y in an equation that can make all the difference in understanding, but would appear as just a single character edit in an edit-distance metric. olmOCR-bench operates on single page PDFs directly. We make this choice because PDFs do preserve some digital metadata and information which may be helpful to some OCR systems. Almost any other format can be converted to a PDF, but not the reverse, so we try to preserve these original documents where possible. We have run the benchmark against some contemporary OCR pipelines, but it is really easy to run it against your own OCR tools. Your tool just needs to support Markdown or plain text output.
## Results
ArXiv Old
scans
math		 Tables Old
scans		 Headers
&
footers		 Multi
column		 Long
tiny
text		 Base Overall
Mistral OCR API 77.2 67.5 60.6 29.3 93.6 71.3 77.1 99.4 72.0±1.1
Marker 1.10.1 83.8 66.8 72.9 33.5 86.6 80.0 85.7 99.3 76.1±1.1
MinerU 2.5.4* 76.6 54.6 84.9 33.7 96.6 78.2 83.5 93.7 75.2±1.1
DeepSeek-OCR 77.2 73.6 80.2 33.3 96.1 66.4 79.4 99.8 75.7±1.0
Nanonets-OCR2-3B 75.4 46.1 86.8 40.9 32.1 81.9 93.0 99.6 69.5±1.1
PaddleOCR-VL* 85.7 71.0 84.1 37.8 97.0 79.9 85.7 98.5 80.0±1.0
Infinity-Parser 7B* 84.4 83.8 85.0 47.9 88.7 84.2 86.4 99.8 82.5±?
Chandra OCR 0.1.0* 82.2 80.3 88.0 50.4 90.8 81.2 92.3 99.9 83.1±0.9
olmOCR (first release) 63.3 67.5 62.3 38.6 93.4 67.6 54.8 97.9 68.2±1.1
v0.1.60 + Dynamic temp scaling 71.4 73.1 65.6 40.5 93.2 76.6 64.9 96.7 72.8±1.2
v0.1.68 + Better prompting 76.3 76.0 70.2 43.2 94.1 77.5 71.9 96.8 75.8±1.0
v0.2.0 + New trainer, YAML, img resize, Qwen 2.5 VL 78.8 77.5 71.9 45.4 94.2 78.6 81.4 99.8 78.5±1.1
v0.3.0 + Handle blank pages 78.6 79.9 72.9 43.9 95.1 77.3 81.2 98.9 78.5±1.1
v0.4.0 + Synth data, RLVR, souping 83.0 82.3 84.9 47.7 96.1 83.7 81.9 99.7 82.4±1.1
Results are reproduced in-house, except those marked with *, which are reported by model authors. ## Sourcing Documents and Tests We define 7 distinct document types that we found olmOCR (or its earlier iterations) often struggled to process and defined custom acquisition strategies for each (described below). We removed documents that both contained PII and were not meant for public dissemination. We also decontaminate against documents that appear in olmOCR-Mix via URL level deduplication. To scale creation of test cases over these documents, we combined manual design and review with prompting GPT-4o. ### Document Types - **arXiv Math (AR)**: We downloaded a recent set of papers from the math subset of arXiv, selecting manuscripts with a single TeX source file and corresponding rendered PDF. To select a candidate LATEX expression from a page to use in a test, we (1) ran olmOCR to identify candidate pages with TeX, (2) match pages back to original TeX source, and (3) validate matched TeX rendering compatibility with KaTeX. We manually verify the final set of test cases to exclude instances where custom macros produce renderings that deviate from standard LATEX and to split multi-part equations into smaller test cases. - **Old Scans Math (OSM)**: We crawl old, public domain math textbooks from the Internet Archive, extracting random pages from these documents. We similarly use olmOCR to find candidate pages with formulas, but this time manually annotate each formula on the page to use as test cases. - **Tables (TA)**: We sampled more documents from the same internal crawled PDF repository used to create olmOCR-Mix and filtered to those which had tables using a simple prompt with Gemini-Flash-2.0. On pages with tables, we prompted Gemini-Flash-2.0 for the relationships between randomly chosen cells. We manually reviewed those tests for accuracy. - **Old Scans (OS)**: We sampled historical letters and typewritten documents with existing human transcriptions from the Library of Congress digital archives. We then wrote a small script to generate Natural Reading Order cases consisting of sentences that were naturally before or after one another in the original human transcriptions. We manually added test cases to cover some headers/footers which should have been excluded from any OCR version of these documents. All of the test cases then underwent a second pass of human review for accuracy. - **Headers Footers (HF)**: We sampled documents from the same internally crawled PDF repository as olmOCR-Mix. We used DocLayout-YOLO to identify page regions labeled as headers or footers using the abandon category. To extract the text from these header/footer regions, we visually mask out the rest of the document and prompt Gemini-Flash-2.0 for the content. These extracted snippets are added as test cases that should be absent in linearized output. We manually reviewed to remove mistakenly filtered text and to set conditions such as limiting the search area to the first N or last N characters. - **Multi Column (MC)**: We visually sample documents from our internal crawled PDF repository to find documents with multi-column layouts and multiple articles on one page. We use Claude-Sonnet-3.7 to render those pages to HTML, and from that HTML, we extract text segments before/after one another. We manually review each entry for accuracy. We purposely select simple text blocks from coherent regions of the document, and avoid including any math formulas, superscripts, or subscripts in these tests. - **Long Tiny Text (LTT)**: We crawled documents from the Internet Archive containing a large amount of dense, small print on a single page. Such documents include pages from a dictionary or pages of references from academic papers. We then generate test cases using Gemini-Flash-2.0 and verify them manually. ## Benchmark Principles As we created olmOCR-bench, we also kept a few general rules in mind: - We expect your OCR system to output a plain-text Unicode document in a reading order that would be considered natural. - Documents from the benchmark should fit on a standard A4 piece of paper and still be readable to a human. - Markdown syntax is allowed, but ignored. Ex. if we are looking for the word "enlightenment" to appear on a page, and your system outputs "**\*\*enlightenment\*\***" in Markdown bold, that still counts. - olmOCR-bench is not position sensitive, ex. we check that a sentence or math equation appears anywhere on a page. The exception to this is header/footer tests where we want to find simple page numbers appearing in the first or last few characters of a page. - Tables can be in either Markdown syntax, or as an html ``. - Math equations must render with [Katex](https://katex.org/) and be delimeted with $, $$, \\(, or \\[. - Math equations are not position sensitive either, so if we are checking for $ 3x^2 $ to appear on a page, then outputting $ \int_a^b{ 3x ^ 2dx} $ counts. - We normalize all Unicode to NFC before running the benchmark, so if your OCR model outputs é vs e + ◌́ then either way should not affect your benchmark score. - We normalize all the different variants of hyphens to the ascii -, all the variants of double quoets to ascii " and all variants of single quotes/apostrophes to ascii '. You should score the same on the benchmark if you output - vs — - All facts checked about documents are either pass/fail. We want it to be very clear if your OCR system fails a test, and if so, what output would make it pass. ## olmOCR-Bench Test classes - Text presence - This task makes sure that a given small piece of text (ex. 1-3 sentence level) is present within a parsed document. Soft/fuzzy matching is allowed, as well as specifying if the text must be in the first N or last N characters of the document. Case sensitive by default. - Text absense - This task makes sure that a given piece of next does NOT appear in the OCR'ed version of a document. We generally want our OCR systems to filter out content like headers/footers/page numbers from documents. The same fuzzy matching as in Text Presence tests is allowed. - Natural Reading Order - This task ensures that blocks of text which are present have a defined order relative to one another. For example, on a document that contains multiple news articles on one page, you'd want to see that the first sentence of the first article appears after the heading of that article. But, you may be okay with swapping the order of those two articles. - Table Accuracy - Both Markdown and HTML based tables are supported. These tests check that a cell with a given text exists somewhere in the table, and that its neighbors have certain properties. Ex. A cell exists on this page with text "4.5%" and above that is a cell with the text "2.4%". However, it's important to note that some tests depend on rowspan and colspan information being present in the table, which is only available with HTML based tables. This means that a model outputting only markdown tables cannot achieve a max score on this section. - Math Formula Accuracy - We render a given Latex style equation using Katex in a headless browser. And then see if it exists anywhere in the final OCRed document. Matching is performed on a relative symbol level, ex. in "\f\relax{x} = \int_{-\infty}^\infty x^2dx" we check that a ∫ appears to the left of a x, x appears to the left of dx, etc... ## Downloading and running the benchmark Currently the full benchmark data is located here: https://huggingface.co/datasets/allenai/olmOCR-bench To run a benchmark, first install the bench requirements ```bash conda create -n olmocr python=3.11 conda activate olmocr git clone https://github.com/allenai/olmocr.git cd olmocr # Install olmocr and the requirements needed to run the benchmark pip install -e .[bench] # Configure playwright headless browser to run the math rendering tests playwright install chromium # Now clone the benchmark data from hugging face, this includes the PDFs and JSON annotation data huggingface-cli download --repo-type dataset --resume-download allenai/olmOCR-bench --local-dir ./olmOCR-bench ``` Convert your documents ```bash # You will need to install the [gpu] subset of olmocr dependencies to run gpu inference # Then convert using using olmocr.bench.convert, see the olmocr/bench/runners directory for options pip install olmocr[gpu] --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/ python -m olmocr.bench.convert olmocr_pipeline --dir ./olmOCR-bench/bench_data # OR, you can use the pipeline to convert the benchmark PDFs and move them into the final format python -m olmocr.pipeline ./localworkspace --markdown --pdfs ./olmOCR-bench/bench_data/pdfs/**/*.pdf python olmocr/bench/scripts/workspace_to_bench.py localworkspace/ olmOCR-bench/bench_data/olmocr --bench-path ./olmOCR-bench/ ``` Now run the benchmark ```bash python -m olmocr.bench.benchmark --dir ./olmOCR-bench/bench_data ``` ## Previewing the benchmark questions We have an internal data annotation tool that can be used to review the questions in the benchmark, and make edits. image ```bash python -m olmocr.bench.review_app --port 5000 --debug ./olmOCR-bench/bench_data/multi_column.jsonl --force ``` --- ### Overview (`olmocr/train/README.md`) # olmOCR Training Guide This guide provides comprehensive instructions for training olmOCR models, including what you need to reproduce https://huggingface.co/allenai/olmOCR-2-7B-1025-FP8 on your own hardware. ## Environment setup The first step is to setup your python/conda environment, and set things up the same way as for running olmocr. Then, add in some extra training requirements: ```bash pip install .[train] pip install transformers==4.52.4 pip install flash-attn>=2.8.0.post2 --no-build-isolation ``` ### Dataset Format The training data should be organized as pairs of PDF files and their corresponding markdown annotations: **Important: Each PDF needs to be a single page only!** ``` data/ ├── document1.pdf ├── document1.md ├── document2.pdf ├── document2.md └── ... ``` Each markdown file should contain: 1. YAML front matter with metadata 2. The extracted text content Example markdown format: ```markdown --- primary_language: en is_rotation_valid: True rotation_correction: 0 is_table: False is_diagram: False --- Document text goes here... ``` The easiest way to grab a lot of files in this format is to use `prepare_olmocrmix.py` which will automatically download and prepare [olmOCR-mix-1025](https://huggingface.co/datasets/allenai/olmOCR-mix-1025) for your environment. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The second easiest way to get training data is to use olmOCR to convert a bunch of documents and then adjust the results to better suit you. Once you have an olmOCR workspace with some converted PDFs, you can use the `olmocr.data.prepare_workspace` script to extract those into the proper format. ```bash python -m olmocr.pipeline ./localworkspace --pdfs /home/username/pdfs/*.pdf python -m olmocr.data.prepare_workspace ./localworkspace ./localtrainingdata ``` `./localtrainingdata` will now have data in the proper format, each PDF split into a single page, etc, that you can use for fine tuning. ### Setup your config #### Reference configs for released models: - [olmOCR-7B-0725-FP8](https://huggingfae.co/allenai/olmOCR-7B-0725-FP8) was trained with [qwen25_vl_olmocrv2_2epoch.yaml](/olmcr/train/configs/v0.2.0/qwen25_vl_olmocrv2_2epoch.yaml) - [olmOCR-7B-0825-FP8](https://huggingface.co/allenai/olmOCR-7B-0825-FP8) was trained with [qwen25_vl_olmocrv3_rotation_1epoch.yaml](/olmocr/train/configs/v0.3.0/qwen25_vl_olmocrv3_rotation_1epoch.yaml) - [olmOCR-2-7B-1025-FP8](https://huggingface.co/allenai/olmOCR-2-7B-1025-FP8) was trained with [qwen25_vl_olmocrv4_rotation_1epoch_mix_1025_filtered.yaml](/olmocr/train/configs/v0.4.0/qwen25_vl_olmocrv4_rotation_1epoch_mix_1025_filtered.yaml) These are setup to train on a single B200 GPU, and training will take around 24-48 hours (~$300 if renting). A single epoch on our full model is ~270,000 page, so it's quite a big endeavour. You may need to adjust the paths in the dataset section of the config to match where you downloaded and extracted the training mix. And also set the output_dir to a location where you would like to save checkpoints. #### Finetuning Configs If you would like to finetune on a smaller amount of local data, with a LoRA adapter, try the following - Basic finetuning config: [qwen25_vl_olmocrv4_finetuning.yaml](/olmocr/train/configs/v0.4.0/qwen25_vl_olmocrv4_finetuning.yaml) You'll still need to adjust your paths accordingly. ### Launch training job ```bash python -m olmocr.train.train --config olmocr/train/configs/v0.4.0/qwen25_vl_olmocrv4_rotation_1epoch_mix_1025_filtered.yaml ``` ### Prepare Checkpoints and Quantize After training is done, you will need to call `prepare_checkpoint.py` to take the saved checkpoints and get them ready for use with VLLM. ```bash python -m olmocr.train.prepare_olmocr_checkpoint [source dir]/checkpoint-xxxx [destination] ``` If you just finetuned a LoRA adapter, you should run the `prepare_checkpoint.py` script as above, it will merge the weights into a full model and adjust the configs so you can easily run the model. #### FP8 Quantization The training process operates in BF16 weights natively, even if you are just training a LoRA adapter. We recommend doing an FP8 quantization step, whose performance is solidly in the error bars of the raw bfloat16 model, but uses less memory and inferences around 12% faster. ```bash python -m olmocr.train.compress_checkpoint --config olmocr/train/quantization_configs/qwen2_5vl_w8a8_fp8.yaml [destination] [destination-FP8] ``` ### GRPO Training [olmOCR-2-7B-1025-FP8](https://huggingface.co/allenai/olmOCR-2-7B-1025-FP8) adds an additional training step with GRPO RL based training occuring on a synthetic version of olmOCR-bench. [olmOCR-synthmix-1025](https://huggingface.co/datasets/allenai/olmOCR-synthmix-1025) was created by having Claude Sonnet take real PDF documents, then convert them into HTML templates. Those HTML templates were then rendered, and converted into synthetic olmOCR-bench style benchmarks. We then ran a GPRO based training process with a reward based on the benchmark score on this synthetic benchmark. You can use the following command to download the synthetic dataset: ```bash hf download allenai/olmOCR-synthmix-1025 --repo-type dataset --local-dir olmOCR-synthmix-1025 ``` This was generated using [mine_html_templates.py](olmocr/synth/mine_html_templates.py), and is in the same format as other olmOCR-bench test cases. This means you can measure performance against these cases directly in the same way as running any other olmOCR-bench test suite. The following scripts show how to start training, which is performed on an 8xH100 GPU node. One GPU is dedicated to running VLLM, while the other 7 are used to run training. At the moment, this code is quite specialized to our cluster here, but we hope to make it easier to run elsewhere in future releases. ```bash ./scripts/train/grpotrainer-beaker-multi-gpu-augusta.sh --num-gpus 8 --model_name s3://ai2-oe-data/jakep/olmocr/qwen2.5-vl-7b-olmocrv4_1epoch_promptv4_mix1025_more_rotation-8372 --train_bench_data_folder /data/jakep/grpo_data_mixes/olmocr-synthmix-1025-v2-rotate10p/bench_data --reward_bench 1.0 --reward_front_matter 1.0 --reward_eos 1.0 --beta 0.01 --name promptv4_mix1025_more_rotation_multigpu_v1_beta_01_lr2e-6_frontmatter1_0_eos_28gen_synthmix-1025_rotate10p_finalrun1 --seed 1 --gradient_accumulation_steps 28 --learning_rate 2e-6 --preemptible ./scripts/train/grpotrainer-beaker-multi-gpu-augusta.sh --num-gpus 8 --model_name s3://ai2-oe-data/jakep/olmocr/qwen2.5-vl-7b-olmocrv4_1epoch_promptv4_mix1025_more_rotation-8372 --train_bench_data_folder /data/jakep/grpo_data_mixes/olmocr-synthmix-1025-v2-rotate10p/bench_data --reward_bench 1.0 --reward_front_matter 1.0 --reward_eos 1.0 --beta 0.01 --name promptv4_mix1025_more_rotation_multigpu_v1_beta_01_lr2e-6_frontmatter1_0_eos_28gen_synthmix-1025_rotate10p_importanceseq_finalrun1 --seed 1 --importance_sampling_level sequence --gradient_accumulation_steps 28 --learning_rate 2e-6 --preemptible ``` 6 seeds were run, 3 with importance sampling level=sequence, and 3 with importance sampling level=token, and then merged into a final checkpoint. Souping can be done by passing more arguments to the `prepare_checkpoint` script. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Notes for AI2 If you are a collaborator of AI2, you can use the following scripts to run training and inference ```bash # Run training using Beaker scripts/train/newtrainer-beaker.sh --config [config file] # Prepare checkpoint from an interactive session with WEKA python -m olmocr.train.prepare_olmocr_checkpoint [source] [destination] # Compress the prepared model checkpoint to FP8 scripts/train/compress_model.sh [--calibration-pdfs PATTERN] # Run olmOCR bench scripts/run_benchmark.sh --model [destination] ``` --- ### Overview (`scripts/elo/README.md`) # elo rating Calculates elo rating of olmOCR vs other tools. ## Data The pairwise judgment data is stored in `ratings.csv` as win/loss counts: ``` MethodA,MethodB,A_wins,B_wins,A_rate(%),B_rate(%) marker,mineru,53,26,67.1,32.9 mineru,pdelf,22,55,28.6,71.4 gotocr_format,marker,26,45,36.6,63.4 marker,pdelf,31,49,38.8,61.3 gotocr_format,pdelf,29,41,41.4,58.6 gotocr_format,mineru,38,37,50.7,49.3 ``` *Note* `pdfelf` is olmOCR. ## Usage To calculate elo ratings, run the following command: ```bash python calculate_elo_ratings.py ratings.csv --num-bootstrap 5000 --num-elo-sims 100 --confidence-level 95 --seed 123 ``` It should print something like: ``` Bootstrapped Elo Ratings (95% CI): -------------------------------------------------- pdelf 1813.0 ± 84.9 [1605.9, 1930.0] mineru 1545.2 ± 99.7 [1336.7, 1714.1] marker 1429.1 ± 100.7 [1267.6, 1645.5] gotocr_format 1212.7 ± 82.0 [1097.3, 1408.3] Pairwise Significance Tests: -------------------------------------------------- gotocr_format vs marker Δ = -216.3 [-470.8, 135.0] p = 0.218 gotocr_format vs mineru Δ = -332.5 [-567.5, 19.3] p = 0.051 gotocr_format vs pdelf Δ = -600.3 [-826.1, -344.3] p = 0.000* marker vs mineru Δ = -116.1 [-365.4, 246.5] p = 0.430 marker vs pdelf Δ = -383.9 [-610.6, -10.9] p = 0.044* mineru vs pdelf Δ = -267.8 [-517.3, 104.0] p = 0.135 ``` which is also already saved in `results.txt`. To generate boxplots of elo ratings, run the following command: ```bash python draw_boxplots.py results.txt boxplots.png ``` which should save boxplots as `boxplots.png`. --- ### Source Index (`docs/source/index.md`) # **olmocr** ```{toctree} :maxdepth: 2 :hidden: :caption: Getting started installation overview ``` ```{toctree} :hidden: :caption: Development CHANGELOG CONTRIBUTING License GitHub Repository ``` ## Indices and tables ```{eval-rst} * :ref:`genindex` * :ref:`modindex` ``` --- ### Source Installation (`docs/source/installation.md`) Installation ============ **olmocr** supports Python >= 3.8. ## Installing with `pip` **olmocr** is available [on PyPI](https://pypi.org/project/olmocr/). Just run ```bash pip install olmocr ``` ## Installing from source To install **olmocr** from source, first clone [the repository](https://github.com/allenai/olmocr): ```bash git clone https://github.com/allenai/olmocr.git cd olmocr ``` Then run ```bash pip install -e . ``` --- --- METRICS --- - Files Extracted: 6 - Estimated Token Budget: ~11220 tokens - Canonical Reference: https://codewiki.google/github.com/allenai/olmocr