> [!WARNING]
> **This project is no longer maintained.** Security patches are not being applied. Consider using an alternative such as [LiteParse](https://github.com/run-llama/liteparse) if you need a local, Apache 2.0–licensed parsing solution.
- **Parsr**, is a minimal-footprint document (**image, pdf, docx, eml**) cleaning, parsing and extraction toolchain which generates readily available, organized and usable data in **JSON, Markdown (MD), CSV/Pandas DF** or **TXT** formats.
- It provides analysts, data scientists and developers with clean structured and label-enriched information set for ready-to-use applications ranging from data entry and document analysts automation, archival, and many others.
- Currently, Parsr can perform: document cleaning, _hierarchy regeneration_ (words, lines, paragraphs), detection of _headings, tables, lists, table of contents, page numbers, headers/footers, links_, and others. Check out [all the features](server/src/processing/README.md#1-current-processing-modules).
# Table of Contents
- [Table of Contents](#table-of-contents)
- [Getting Started](#getting-started)
- [Installation](#installation)
- [Usage](#usage)
- [Documentation](#documentation)
- [Contribute](#contribute)
- [Third Party Licenses](#third-party-licenses)
- [License](#license)
# Getting Started
## Installation
_-- The advanced installation guide is available [here](docs/installation.md) --_
The quickest way to install and run the Parsr API is through the [docker image](https://hub.docker.com/r/axarev/parsr):
```sh
docker pull axarev/parsr
```
If you also wish to install the GUI for sending documents and visualising results:
```sh
docker pull axarev/parsr-ui-localhost
```
Note: Parsr can also be installed bare-metal (not via Docker containers), the procedure for which is documented in the [installation guide](docs/installation.md).
## Usage
_-- The advanced usage guide is available [here](docs/usage.md) --_
To run the [API](docs/api-guide.md), issue:
```sh
docker run -p 3001:3001 axarev/parsr
```
which will launch it on [http://localhost:3001](http://localhost:3001).
Consult the documentation on the [usage of the API](docs/api-guide.md).
1. To access the **python** client to Parsr API, issue:
```sh
pip install parsr-client
```
To sample the **Jupyter Notebook**, using the python client, head over to the [jupyter demo](demo/parsr-jupyter-demo).
2) To use the GUI tool (the API needs to already be running), issue:
```sh
docker run -t -p 8080:80 axarev/parsr-ui-localhost:latest
```
Then, access it through [http://localhost:8080](http://localhost:8080).
Refer to the [Configuration documentation](docs/configuration.md) to interpret the configurable options in the GUI viewer.
The [API based usage](docs/usage.md#3-api) and the [command line usage](docs/usage.md#23-command-line-usage) are documented in the [advanced usage](docs/usage.md) guide.
# Documentation
All documentation files can be found [here](docs/README.md).
# Contribute
Please refer to the [contribution guidelines](CONTRIBUTING.md).
# Third Party Licenses
Third Party Libraries licenses for its [dependencies](docs/dependencies.md):
1. **QPDF**: Apache [http://qpdf.sourceforge.net](http://qpdf.sourceforge.net/)
2. **ImageMagick**: Apache 2.0 [https://imagemagick.org/script/license.php](https://imagemagick.org/script/license.php)
3. **Pdfminer.six**: MIT [https://github.com/pdfminer/pdfminer.six/blob/master/LICENSE](https://github.com/pdfminer/pdfminer.six/blob/master/LICENSE)
4. **PDF.js**: Apache 2.0 [https://github.com/mozilla/pdf.js](https://github.com/mozilla/pdf.js)
5. **Tesseract**: Apache 2.0 [https://github.com/tesseract-ocr/tesseract](https://github.com/tesseract-ocr/tesseract)
6. **Camelot**: MIT [https://github.com/camelot-dev/camelot](https://github.com/camelot-dev/camelot)
7. **MuPDF** (Optional dependency): AGPL [https://mupdf.com/license.html](https://mupdf.com/license.html)
8. **Pandoc** (Optional dependency): GPL [https://github.com/jgm/pandoc](https://github.com/jgm/pandoc)
# License
Copyright 2020 AXA Group Operations S.A.
Licensed under the [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0) license (see the [LICENSE](LICENSE) file).
---
### Overview (`docs/README.md`)
# Parsr Documentation
1. [Installation Guide](installation.md)
2. [Usage Guide](usage.md)
3. [Configuration Guide](configuration.md)
4. [API Guide](api-guide.md)
5. [JSON Output](json-output.md)
6. [ABBYY FineReader](abbyy-finereader.md)
6. [Architecture](architecture.md)
7. [Dependencies](dependencies.md)
8. [Docker](docker.md)
---
### Overview (`parsr-fix-pdf-font/README.md`)
# Parsr - Fix PDF Font
**Parsr-fix-pdf-font** is a utility designed specifically to remedy broken unicode maps for PDF fonts. Issues with broken unicode maps can arise due to various reasons, including incomplete or corrupt font embedding, or issues during the PDF creation process. Such problems can render text in a PDF file unreadable or undecipherable.
This tool leverages Tesseract.js, an optical character recognition engine, to recognize the broken glyphs present in the PDF. Once these glyphs are identified, **Parsr-fix-pdf-font** rebuilds the unicode map, ensuring that the PDF becomes readable and retains its original design and layout.
## Features
- OCR Powered Correction: Uses Tesseract.js to perform Optical Character Recognition on the broken glyphs, ensuring accurate text representation.
- Rebuilding Unicode Maps: After identifying the incorrect mappings, the tool regenerates the correct unicode map, preserving the original design of the PDF.
- Easy-to-Use Command Line Interface: Simplified command line usage for quick fixes.
## Requirements
nodejs >18
ImageMagick Convert
## Usage
Use the command line interface to run the Parsr tool:
```sh
cd parsr-fix-pdf-font
node parsr-fix-pdf-font.js --input ./test/testPage.pdf --output pdf.pdf --lang eng
```
Parameters:
- --input : Specifies the path to the source PDF file that needs to be fixed.
- --ouput : Designates the path where the fixed PDF will be saved. If the specified file already exists, it will be overwritten.
- --lang eng: Sets the language for the OCR process. By default, it's set to English (eng). Tesseract supports multiple languages, so ensure you choose the appropriate one for your document.
## Troubleshooting
If you encounter any issues:
Inspect PDF: Ensure that the PDF isn't password protected or encrypted. If it is, decrypt it before running the tool.
Language Mismatch: If the OCR isn't accurate, ensure you've chosen the correct language setting for the document.
## Limits
Tesseract OCR is not really good on single Glyph, but at least the text is readable / understandable for an LLM.
We do not reconstruct the XREF table yet. Using a tool like ```mutools clean ``` will allow you to fix them if needed.
## Contribution
Parsr is an open-source tool. Contributions in the form of bug reports, feature requests, or code are always welcome. Check our GitHub repository for more details.
---
### Overview (`clients/python-client/README.md`)
# Parsr Client
Provides a python interface to the Parsr tool via its API.
Parsr transforms PDF, documents and images into enriched, structured data.
Find out all about Parsr (including download) at [https://github.com/axa-group/Parsr](https://github.com/axa-group/Parsr).
## 1 Installation
```sh
pip install parsr-client
```
## 2 Usage
_Make sure that the Parsr Server is already running. Let us suppose that the address is `localhost:3001`_
### 2.1 Connect to the Parsr server
```python
from parsr_client import ParsrClient
parsr = ParsrClient('localhost:3001')
```
### 2.2 Send the document
```python
parsr.send_document(
file_path='README.pdf',
config_path='defaultConfig.json',
document_name='The Readme',
save_request_id=True)
```
### 2.4 Retrieve results
1. Get everything as a JSON:
```python
parsr.get_json()
```
2. As Markdown:
```python
parsr.get_markdown()
```
3. As text:
```python
parsr.get_text()
```
4. Get the first table on the first page:
```python
parsr.get_table(
page=1,
table=1,
)
```
5. Get all the versions of the document:
```python
parsr.get_revisions('The Readme')
```
6. Get pretty diffs between each successive pair of a document's revisions:
```python
parsr.compare_revisions('The Readme', pretty_html=True)
```
## 3 Interpreting the whole JSON output locally
The supplied `ParsrOutputInterpreter` class can be used to interpret the downloaded JSON output and generate higher level structures like the text body.
Here's an example to generate text body on the first page from the above example.
``` python
from parsr_client import ParsrOutputInterpreter
parsr_interpreter = ParsrOutputInterpreter(
parsr.get_json()
)
t = parsr_interpreter.get_text(
page_number=1
)
print(t)
```
---
### Overview (`demo/doc-versioning/README.md`)
# Doc Versioning
- Stores and displays document (PDF, Images, DOCx) revisions and diffs.
- Uses [Parsr](https://github.com/axa-group/Parsr) to perform the document data extraction.
- Based on Python 3
## Installation
Dependencies:
1. [Flask](https://flask.palletsprojects.com/en/1.1.x/installation/#install-flask)
2. [PyWebview](https://pywebview.flowrl.com/guide/installation.html#dependencies)
3. [Parsr-Client](https://pypi.org/project/parsr-client/) > v3.1.0
4. [markdown](https://python-markdown.github.io/install/)
## Usage
Once the dependencies have been set up, execute the following in the project folder:
``` sh
./run.py
```
---
### Overview (`demo/echo-module-py/README.md`)
# Demo Python Remote Module
This module is a demo on how you can create your own remote modules.
## Try it!
Run it with:
```
$ python3 echo-module.py
```
Then call it in your [configuration](../../docs/configuration.md):
```json
[
"remote",
{
"url": "http://localhost:8888"
}
]
```
---
### Overview (`demo/parsr-jupyter-demo/README.md`)
# Parsr - Jupyter Notebook
This folder includes a [jupyter notebook](./notebook.ipynb), which shows how a user can access the API using a python interface, showing the following functions:
1. Connecting to the Parsr API.
2. Sending a document (or a folder) to the Parsr API.
3. Retrieving the complete JSON output.
4. Retrieving other output formats such as Markdown, text, and tables (CSV and Pandas Dataframes).
5. Using the [parsr output interpreter class](../../clients/python-client/parsr_client/parsr_output_interpreter.py), which reconstructs usable elements from an already downloaded Parsr outputted JSON file.
## Installation
Install [pipenv](https://pipenv.readthedocs.io/en/latest/install/#installing-pipenv), then issue the following inside this folder (`demo/parsr-jupyter-demo`):
```sh
pipenv install
```
## Usage
To start the jupyter notebook, issue:
```sh
pipenv run jupyter lab notebook.ipynb
```
---
### Overview (`demo/vue-viewer/README.md`)
# vue-viewer
## Project setup
```
npm install
```
### Compiles and hot-reloads for development
```
npm run serve
```
### Compiles and minifies for production
```
npm run build
```
### Run your tests
```
npm run test
```
### Lints and fixes files
```
npm run lint
```
### Customize configuration
See [Configuration Reference](https://cli.vuejs.org/config/).
---
### Overview (`server/src/input/README.md`)
# Input Modules
The Input modules in Parsr perform the initial role of importing the raw data from the input files.
Each module performs on a particular type of input files, and generate different results.
Each module may or may not contain a set of configurable parameters, which (along with the usage documentation) can be consulted in the per-module documentation pages below.
Each module returns a valid `Document` object with an array of `Words` for each parsed `Page`.
## The Modules
1. [Pdfminer](pdfminer/README.md)
2. [PDF.js](pdf.js/README.md)
3. [Tesseract](tesseract/README.md)
4. [Google Vision](google-vision/README.md)
5. [Amazon Textract](amazon-textract/README.md)
6. [MS Cognitive Services](ms-cognitive-services/README.md)
7. [ABBYY](abbyy/README.md)
8. [JSON](json/README.md)
9. [MS Word](doc/README.md)
10. [Email](email/README.md)
## Supported input formats
Currently, the following file formats are available for Parsr:
Input format
Input modules
Pdfminer
pdf.js
ABBYY
Tesseract
JSON Extractor
Google Vision
Amazon Textract
MS Cognitive Services
.pdf
✓
✓
✓
✓
✗
✗
✗
✗
.docx
✓
✓
✓
✓
✗
✗
✗
✗
.eml
✓
✓
✓
✓
✗
✗
✗
✗
.tiff
✗
✗
✓
✓
✗
✓
✓
✓
.png
✗
✗
✓
✓
✗
✓
✓
✓
.jpeg
✗
✗
✓
✓
✗
✓
✓
✓
.json
✗
✗
✗
✗
✓
✗
✗
✗
.xml
✗
✗
✓
✗
✗
✗
✗
✗
This means that for processing a pdf file, 4 extractors can be chosen: pdfminer, pdf.js, ABBYY or Tesseract.
**Note:** *not all extractors share the same functionality or return the same information, so one should check for the best extractor given the use case.*
**Note:** *when using a json or xml file as input, extractor configuration will be ignored as there is currently only one extractor for each of this formats.*
---
### Overview (`server/src/output/README.md`)
# Output Modules
The output modules in Parsr perform a final role of generating the processed, enriched and structured output.
Each module generates a specific output file type, and represents a different granularity and type of data.
For more information, along with individual configuration options, consult the per-module documentation pages below:
## The Modules
1. [JSON](json/README.md)
2. [Simple JSON](simpleJson/README.md)
3. [Markdown](markdown/README.md)
4. [Text](text/README.md)
5. [CSV](csv/README.md)
6. [PDF](pdf/README.md)
---
### Overview (`server/src/processing/README.md`)
# Processing Modules
- [Processing Modules](#processing-modules)
- [1. Current Processing Modules](#1-current-processing-modules)
- [2. Create your own Processing Module](#2-create-your-own-processing-module)
- [2.1. Creating and Naming your Typescript Module](#21-creating-and-naming-your-typescript-module)
- [2.2. Add to Register](#22-add-to-register)
- [2.3. Add it to the Configuration](#23-add-it-to-the-configuration)
- [2.4. Run it!](#24-run-it)
The processing modules in Parsr perform a central role of cleaning and enriching the extracted raw output.
Each module performs a particular operation on a document representation, generates a new valid Document, and then passes it on to the next module for the next treatment.
Each module can contain a set of configurable parameters, which can be consulted in the per-module documentation pages below:
## 1. Current Processing Modules
1. [Drawing Detection](DrawingDetectionModule/README.md)
2. [Header and Footer Detection](HeaderFooterDetectionModule/README.md)
3. [Heading Detection](MLHeadingDetectionModule/README.md)
4. [Hierarchy Detection](HierarchyDetectionModule/README.md)
5. [Image Detection](ImageDetectionModule/README.md)
6. [Key-Value Pair Detection](KeyValueDetectionModule/README.md)
7. [Lines to Paragraph](LinesToParagraphModule/README.md)
8. [Link Detection](LinkDetectionModule/README.md)
9. [List Detection](ListDetectionModule/README.md)
10. [Number Correction](NumberCorrectionModule/README.md)
11. [Out of Page Removal](OutOfPageRemovalModule/README.md)
12. [Page Number Detection](PageNumberDetectionModule/README.md)
13. [Reading Order Detection](ReadingOrderDetectionModule/README.md)
14. [Redundancy Detection](RedundancyDetectionModule/README.md)
15. [Regex Matcher](RegexMatcherModule/README.md)
16. [Remote Module](RemoteModule/README.md)
17. [Separate Words](SeparateWordsModule/README.md)
18. [Table Detection](TableDetectionModule/README.md)
19. [Table of Contents Detection](TableOfContentsDetectionModule/README.md)
20. [Whitespace Removal](WhitespaceRemovalModule/README.md)
21. [Words To Line](WordsToLineModule/README.md)
## 2. Create your own Processing Module
Creating a custom module can be very useful to add some treatment on the document.
You have two ways to do it:
1. Use the [Remote Module](RemoteModule/README.md) that will send the JSON by HTTP and expect the modified JSON as an answer
2. Create a Typescript Module and add it to the pipeline
### 2.1. Creating and Naming your Typescript Module
The [template module folder](TemplateModule) shows how a module tree needs to be structured.
The folder name, the module's filename and the [class's name](https://github.com/axa-group/Parsr/blob/a92a254f7860bbfe51ec1f24171ef8d44c54ccac/server/src/processing/TemplateModule/TemplateModule.ts#L35) need to follow the [PascalCase naming convention](https://github.com/basarat/typescript-book/blob/master/docs/styleguide/styleguide.md#class).
You can copy the entire folder to help you having a boilerplate.
The template code also contains some handy comments to help you get started.
### 2.2. Add to Register
To add your newly created module to the register, simply open the [Cleaner file](../Cleaner.ts) `/server/src/Cleaner.ts` and add your module class to the `Cleaner.cleaningToolRegister` attribute.
### 2.3. Add it to the Configuration
If you want your module to run you need to enable it in your [configuration](../../../docs/configuration.md#3-Cleaner-Config).
Simply add a line in the `cleaner` array with the name of your module, and potential options.
### 2.4. Run it!
That's it! Your new awesome processing module should run and modify the document according to your needs!
---
### Overview (`server/src/input/abbyy/README.md`)
# ABBYY FineReader Server Input Module
The ABBYY FineReader is a high-precision OCR option provided to the users of the Parsr platform.
### Credentials
When ABBYY FineReader Server is chosen as the working OCR extraction solution, the following credentials need to be set on the host running Parsr:
1. `ABBYY_SERVER_URL` : The network address of the ABBYY FineReader Server.
2. `ABBYY_SERVER_VER` : The major version number of the ABBYY FineReader Server. For example: 14 for ABBYY FineReader Server 14.01.
3. `ABBYY_WORKFLOW` : The name of the server 's workflow to be called to process the file.
```json
"extractor": {
"pdf": "abbyy",
"ocr": "abbyy",
"credentials": {
"ABBYY_SERVER_URL": "...",
"ABBYY_SERVER_VER": "...",
"ABBYY_WORKFLOW": "..."
}
},
```
### Essential Server Configuration
On the side of the ABBYY FineReader Server, make sure the XML output is configured for the selected workflow:
1. Double click on the workflow to be used.
2. In the tab titled 'output', make sure the list of file formats exported contains the XML format if not, add it with the 'New' button.
3. Make sure the following settings are enabled on the XML format 's settings:
1. Character Attributes
2. Extended Character Attributes
3. Coordinates of the Original Image
4. Character Formatting
---
### Overview (`server/src/input/amazon-textract/README.md`)
# Amazon Textract Input Module
Extractor module for OCR'ing image files.
### Credentials
To use the Amazon Textract input module, you need a valid set of access keys from AWS with "_AmazonTextractFullAccess_"
You can generate one for free following [this guide](https://docs.aws.amazon.com/textract/latest/dg/getting-started.html).
Once you have the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, make sure to set those in the `"credentials"` key in the config file.
```json
"extractor": {
"pdf": "...",
"ocr": "amazon-textract",
"credentials": {
"AWS_ACCESS_KEY_ID": "...",
"AWS_SECRET_ACCESS_KEY": "..."
}
},
```
---
### Overview (`server/src/input/doc/README.md`)
# MS Word Input Module
Extractor module that parses .docx files to build documents.
This module will transform the .docx into a single PDF file with A4 pages, and then run the selected PDF extractor on this file.
### Credentials
This module requires no credentials to work.
---
### Overview (`server/src/input/email/README.md`)
# Email Input Module
Extractor module that parses .eml files (downloaded e-mails) to build documents.
This module will transform the .eml (and all its PDF and Image attachments) into a single PDF file with A4 pages, and then run the selected PDF extractor on this file.
### Credentials
This module requires no credentials to work.
---
### Overview (`server/src/input/google-vision/README.md`)
# Google Vision Input Module
Extractor module for OCR'ing image files.
### Credentials
To use the Google Vision input module, follow these steps:
1. [Create a new Google Cloud Platform App](https://console.cloud.google.com/projectcreate)
2. [Activate the Google Vision API](https://console.cloud.google.com/apis/library/vision.googleapis.com)
3. [Create new credentials](https://console.cloud.google.com/apis/credentials/wizard) and download the JSON file containing your credentials.
4. Copy the contents of the downloaded JSON file and paste everything as `"credentials"` value:
```json
"extractor": {
"pdf": "...",
"ocr": "google-vision",
"credentials": {
"type": "...",
"project_id": "...",
"private_key_id": "...",
"private_key": "...",
"client_email": "...",
"client_id": "...",
"auth_uri": "...",
"token_uri": "...",
"auth_provider_x509_cert_url": "...",
"client_x509_cert_url": "..."
}
},
```
5. Change your configuration to use `google-vision` for OCR.
_NB: This module only works with images yet. To make it work with PDF and TIFF, we'll need to use Batch File Annotation and Google Cloud Storage. [See the official guide](https://cloud.google.com/vision/docs/file-batch)._
---
### Overview (`server/src/input/json/README.md`)
# JSON Input Module
Extractor module that uses Parsr JSON output files as input to reconstruct a document.
### Credentials
This module requires no credentials to work.
---
### Overview (`server/src/input/ms-cognitive-services/README.md`)
# Microsoft Cognitive Services Input Module
Extractor module for OCR'ing image files.
### Credentials
To use the Microsoft Cognitive Services input module, you need a Microsoft Cognitive Services APIKEY.
You can generate one for free following [this link](https://portal.azure.com/#blade/HubsExtension/BrowseResourceBlade/resourceType/Microsoft.CognitiveServices%2Faccounts).
Once you have the key, make sure to set it in the `"credentials"` key in the config file.
```json
"extractor": {
"pdf": "...",
"ocr": "ms-cognitive-services",
"credentials": {
"OCP_APIM_SUBSCRIPTION_KEY": "...",
"OCP_APIM_ENDPOINT": ""
}
},
```
Optionally, you can change the region endpoint to the one that you want to use, via the `OCP_APIM_ENDPOINT` value. If omitted, the default value will be `https://westeurope.api.cognitive.microsoft.com/`.
A list with all available endpoint regions can be found [here](https://westus.dev.cognitive.microsoft.com/docs/services/5adf991815e1060e6355ad44/operations/2afb498089f74080d7ef85eb).
---
### Overview (`server/src/input/pdf.js/README.md`)
# PDF.js Input Module
Extractor module for PDF files.
This is the recommended module for very large documents as it's the fastest solution currently available.
### Credentials
This module requires no credentials to work.
---
### Overview (`server/src/input/pdfminer/README.md`)
# Pdfminer Input Module
Extractor module for PDF files.
### Credentials
This module requires no credentials to work.
---
### Overview (`server/src/input/tesseract/README.md`)
# Tesseract Input Module
Extractor module for OCR'ing image files.
### Credentials
This module requires no credentials to work.
---
### Overview (`server/src/output/csv/README.md`)
# CSV Output Module
CSV exporter will generate a csv file with the contents of each table on the document.
Example from [foo.pdf](../../../../samples/foo.pdf) sample file:
```
Cycle Name;KI (1/km);Distance (mi);Percent Fuel Savings;;;;
;;;;Improved Speed;Decreased Accel;Eliminate Stops;Decreased Idle
2012_2;3.30;1.3;5.9%;9.5%;29.2%;17.4%;
2145_1;0.68;11.2;2.4%;0.1%;9.5%;2.7%;
4234_1;0.59;58.7;8.5%;1.3%;8.5%;3.3%;
2032_2;0.17;57.8;21.7%;0.3%;2.7%;1.2%;
4171_1;0.07;173.9;58.1%;1.6%;2.1%;0.5%;
```
*As you can see, this will not take into account col/row spans.*
*This module will only output Tables from the document*
---
### Overview (`server/src/output/json/README.md`)
# JSON Output Module
This module exports a json file with information about the contents and structure of the document.
Example from [foo.pdf](../../../../samples/foo.pdf) sample file:
```
/* Detailed source-code truncated for AI context efficiency. */
```
By default the level of granularity of the export will be "word". You can configure it to "character" in the config file. For more information, refer to [Output Config/Granularity.](../../../../docs/configuration.md#42-Granularity).
For information about the complete structure of the json file, refer to the [JSON output format.](../../../../docs/json-output.md).
---
### Overview (`server/src/output/markdown/README.md`)
# Markdown Output Module
This module will generate a markdown file based on the contents of the input file.
Example from [foo.pdf](../../../../samples/foo.pdf) sample file:
```
## Table 2-1. Simulated fuel savings from isolated cycle improvements
| **Cycle Name** | **KI (1/km)** | **Distance (mi)** | **Percent Fuel Savings** |<|<|<|
|---|---|---|---|---|---|---|
|^|^|^| **Improved Speed** | **Decreased Accel** | **Eliminate Stops** | **Decreased Idle** |
| 2012\_2 | 3.30 | 1.3 | 5.9% | 9.5% | 29.2% | 17.4% |
| 2145\_1 | 0.68 | 11.2 | 2.4% | 0.1% | 9.5% | 2.7% |
| 4234\_1 | 0.59 | 58.7 | 8.5% | 1.3% | 8.5% | 3.3% |
| 2032\_2 | 0.17 | 57.8 | 21.7% | 0.3% | 2.7% | 1.2% |
| 4171\_1 | 0.07 | 173.9 | 58.1% | 1.6% | 2.1% | 0.5% |
Figure 2-1 extends the analysis from eliminating stops for the five example cycles and examines the additional benefit from avoiding slow-and-go driving below various speed thresholds.
```
*As it can be seen, Markdown exporter can differentiate between Paragraphs, Headings and Tables. It can also export images, tables of content and lists.*
---
### Overview (`server/src/output/pdf/README.md`)
# PDF Output Module
Generates a PDF document given the input file.
*This exporter relies on Markdown exporter to convert the document to Markdown and then uses Pandoc to convert it to a PDF. So as long as the markdown result is good, the PDF output will be good.*
---
### Overview (`server/src/output/simpleJson/README.md`)
# Simple json
This module exports a simpler json file with ordered information about the contents of the document. Here we do not export information about the positions / structure inside the document.
Example from [foo.pdf](../../../../samples/foo.pdf) sample file:
```json
[
{
"type": "heading",
"level": 2,
"content": "2 Quantifying Fuel-Saving Opportunities from Specific Driving Behavior Changes"
},
{
"type": "paragraph",
"content": "2.1 Savings from Improving Individual Driving Profiles"
},
{
"type": "heading",
"level": 5,
"content": "2.1.1 Drive Profile Subsample from Real-World Travel Survey"
},
{
"type": "paragraph",
"content": "The interim report (Gonder et al. 2010) included results from detailed analyses on five cycles selected from a large set of real-world global positioning system (GPS) travel data collected in 2006 as part of a study by the Texas Transportation Institute and the Texas Department of Transportation (Ojah and Pearson 2008). The cycles were selected to reflect a range of kinetic intensity (KI) values. (KI represents a ratio of characteristic acceleration to aerodynamic speed and has been shown to be a useful drive cycle classification parameter [O’Keefe et al. 2007].) To determine the maximum possible cycle improvement fuel savings, the real-world cycles were converted into equivalent “ideal” cycles using the following steps:"
}
]
```
For more information about the complete structure of the json file, refer to the [Simple JSON output format.](../../../../docs/simple-json-output.md)
*SimpleJson exporter can export Headings, Paragraphs, Lists, Tables and Tables of Content.*
---
### Overview (`server/src/output/text/README.md`)
# Text Output Module
This module exports a .txt file including only the text contents of the processed document.
Example from [foo.pdf](../../../../samples/foo.pdf) sample file:
```
2 Quantifying Fuel-Saving Opportunities from Specific Driving Behavior Changes
2.1 Savings from Improving Individual Driving Profiles
2.1.1 Drive Profile Subsample from Real-World Travel Survey
The interim report (Gonder et al. 2010) included results from detailed analyses on five cycles selected from a large set of real-world global positioning system (GPS) travel data collected in 2006 as part of a study by the Texas Transportation Institute and the Texas Department of Transportation (Ojah and Pearson 2008). The cycles were selected to reflect a range of kinetic intensity (KI) values. (KI represents a ratio of characteristic acceleration to aerodynamic speed and has been shown to be a useful drive cycle classification parameter [O’Keefe et al. 2007].) To determine the maximum possible cycle improvement fuel savings, the real-world cycles were converted into equivalent “ideal” cycles using the following steps:
```
*This does not differentiate between Headings and Paragraphs. This exporter can output Paragraphs, Lists, Tables and Tables of Content.*
---
### Overview (`server/src/processing/DrawingDetectionModule/README.md`)
# Drawing Detection Module
## Purpose
Groups SvgShapes into Drawings
## What it does
Converts all SvgShapes given by the extractor to a group of Drawings. Each Drawing is a set of Shapes that are visually together.
The module only modifies the document defined in the `drawingsFile` property, and **not the original document**. This is because of a decision we took to move Drawings information away from the input document, as it can heavily increase the file size.
## Parameters
None
## Dependencies
The module will look for a json file in the `drawingsFile` property of the document, containing the required information to build the Drawings. This file must be created in the extraction step of the process. If this file is not found or contains an invalid JSON format, the module execution will finish and the drawings document will not be modified.
## How it works
It sweeps each document page with a vertical and a horizontal control line, finding separations between groups.
## Accuracy
Very high.
The boundaries of each page and element however, need to be correctly specified by the extractor.
## Limitations
None
---
### Overview (`server/src/processing/HeaderFooterDetectionModule/README.md`)
# Header Footer Detection Module
## Purpose
Detect the header and footer areas in a document, and set the appropriate properties of all the elements contained within those areas.
## What it does
Sets the `isHeader` or `isFooter` property flags for each one of the elements detected as headers or footers respectively in the detected areas.
## Dependencies
[Lines To Paragraph Module](../LinesToParagraphModule/README.md)
## How it works
- It detects empty (or white) horizontal bands across the entire document (all pages), which do not contain an element.
- For the footer, the highest of these such bands, not exceeding the value L from the bottom is retained as the footer dividing line.
- For the header, the lowest of these bands is retained using the same algorithm, except that the distance is calculated form the top of the pages.
- The algorithm also detects page number mentions using regexp based matching inside the elements classified as headers or footers.
## Parameters
The following two parameters are available:
1. `maxMarginPercentage`: The percentage of the page up to which (both from the top and the bottom) the algorithm will search for header or footer classification.
2. `ignorePages`: The list of pages to be ignored in the header/footer search.
This typically includes book titles, table of contents, preface, and other pages which do not typically have the same header/footer layout as the rest of the document.
3. `similaritySizePercentage`: The percentage represent the difference of size between pages. This allow to apply the maxMarginPercentage option within groups of pages of similar size.
## Accuracy
Given a good `maxMarginPercentage`, very good.
The accuracy is directly proportional to the number of pages (samples) available to the algorithm.
## Limitations
- An omittance of non-useful pages (like the title page) in the `ignoredPages` list can produce a false negative.
---
### Overview (`server/src/processing/HierarchyDetectionModule/README.md`)
# Hierarchy Detection Module
## Purpose
Detects the Hierarchy within a document, respecting indentation of element blocks to create a hierarchical structure.
## What it does
Sets the 'childs' and 'parent' property of every element inside a hierarchical structure, like lists inside lists, paragraphs following headings, etc.
## Dependencies
[Lines To Paragraph Module](../LinesToParagraphModule/README.md)
## How it works
## Accuracy
## Limitations
---
### Overview (`server/src/processing/ImageDetectionModule/README.md`)
# Image Detection Module
## Purpose
The Image detection module detect images in a PDF document.
## What it does
It matches every image detected by the extractor to the correct image file and appends image URI to all image elements in the document.
## Dependencies
MuPDF: `mutool extract` is used to extract all image files from a PDF *when pdfminer is selected as pdf extractor*.
## Parameters
`ocrImages`: Allows to extract data from detected images using selected OCR. When `true`, all detected images will be replaced with data extracted by OCR.
`wordsImagesSource`: When `true`, all word extracted by OCR will have their propertie filled by the source of the original image.
## How it works
### with pdfminer:
1. It uses the name of every image (figure name) detected by PdfMiner and tries to match with a single image identifier extracted by 'dumppdf'.
```
//'Image17' will be used to match image XObject using dumppdf
```
2. `dumppdf` dumps the internal contents of a PDF file in pseudo-XML format, using this data the module tries to match each figure name with an image file name.
```
XObjectImage17 // '17' will be used to match a image file named 'img-0017' extracted by MuPDF
```
### with pdfjs:
For the case of pdfjs extractor, the library used to extract texts from the document can also natively extract every image. So a re-execution of pdfjs extractor is done, but asking only for image data.
## Accuracy
Almost **perfect**.
## Limitations
- Images will not be reconstructed if an image detected by extractor (pdfMiner) is not extracted by Muttol extract command then
- Some PDF can use more than one image file (one alpha image with one background image) to generate one visual image in PDF, in that case the module will detect both images.
- The module depends on the selected pdf extractor to have any way of extracting images from the document.
- For now, only 'pdfminer' and 'pdfjs' can extract images information. So if you use another pdf extractor, the module will skip its execution and return an unmodified document.
---
### Overview (`server/src/processing/KeyValueDetectionModule/README.md`)
# Key-Value Detection Module
## Purpose
Detect pairs of key-value throughout each page of a document, depending on a certain number of key-value patterns passed onto the pipeline in the configuration.
## What it does
Generates new `metadata` instances for each match of a key-value pair which validate a match above a certain threshold.
## Dependencies
[Words to Line Module](../WordsToLineModule/README.md)
## How it works
1. All the Line elements of a page are searched for matches with a set of predefined keys in the module parameters using a string matching algorithm based on the Sørensen–Dice coefficient.
Only the matches with a similarity score higher than a threshold are kept as candidates.
2. The resulting set of matched keys are kept as candidates, and a corresponding value is looked for in the vicinity of the found key (on the right of the separator character: ':' or ';' by default).
3. Each key with its corresponding value is saved as a KeyValueMetadata type and attached to the concerned elements.
## Parameters
The configuration of the module is where the key patterns are described.
Following is an example of the configuration of the key-value search module:
```json
[
"key-value-detection",
{
"thresholdRatio": 0.8,
"keyValueDividerChars": [":", ";"],
"keyPatterns": {
"Name": ["Name", "Fullname", "User"],
"Date of admission": ["ADMISSION DATE & TIME", "Adm Date/Time", "Reg/Admit Date"]
}
}
]
```
Here, the `keyValueDividerChars` describes the sets of characters that are used in the input document to separate the keys from values in a key-pair description.
`keyPatterns` describe objects describing the key names and patterns on which these key names need to be matched.
`threshold` describes the minimum similarity score for candidate substrings matches to pass.
## Example
Imagine a document containing the following text:
```
.....
Name: John
Surnames: Doe
Residence: Fake Street 123
.....
```
with config:
```json
{
"thresholdRatio": 0.8,
"keyValueDividerChars": [
":",
],
"keyPatterns": {
"First Name": [
"Name", "First Name"
],
"Surname": [
"Surname", "Surnames", "Last Name"
],
"Address": [
"Address", "Domicile", "Residence"
]
}
}
```
`document.metadata` property after execution of this module:
```json
[
{
"id": 1,
"elements": [
468,
476
],
"type": "key-value",
"data": {
"keyName": "First Name",
"keyElements": [
468
],
"valueElements": [
476
]
}
},
{
"id": 2,
"elements": [
487,
498
],
"type": "key-value",
"data": {
"keyName": "Surname",
"keyElements": [
487
],
"valueElements": [
498,
]
}
},
{
"id": 3,
"elements": [
399,
431
],
"type": "key-value",
"data": {
"keyName": "Address",
"keyElements": [
399
],
"valueElements": [
431
]
}
}
]
```
Here we can see that 3 matches have been found:
- **First Name**: Elements with IDs 468 and 476, corresponding to Words with contents "Name" and "John" respectively.
- **Surname**: Elements with IDs 487 and 498, corresponding to Words with contents "Surnames" and "Doe".
- **Address**: Elements with IDs 399 and 431, with contents "Residence" and "Fake Street 123"
Note that `keyElements` and `valueElements` properties are defined as an array, because more than one match can be found for that key-value pair.
## Accuracy
The accuracy is high.
There is a direct corelation with the quality of the readability of the input document, the OCR's output, as well as the threshold for acceptance of candidates.
## Limitations
The current implementation supposes that key value pairs are always arranged in a single line and in the form `key : value`.
This makes it applicable to only a few limited cases.
---
### Overview (`server/src/processing/LinesToParagraphModule/README.md`)
# Lines to Paragraph Module
## Purpose
Create paragraphs from a collection of lines.
## What it does
It creates new paragraph elements that contains arrays of line elements.
## Dependencies
- [Words to Lines](../WordsToLineModule/README.md)
- [Reading Order Module](../ReadingOrderDetectionModule/README.md)
## How it works
It simply takes every line one by one according to the reading order and stops and loops if the next line is on another paragraph.
## Parameters
1. `tolerance`: Ratio used when merging lines into paragraphs taking into account the line height and bottom distance to next line.
**TIP**: If you see two lines in same paragraph that should be split into two paragraphs decrease tolerance value, if you see two lines in different paragraphs that should be part of same paragraph increase tolerance value.
- Tolerance 0.1
- Tolerance 0.25
## Accuracy
Almost perfect
## Limitations
- It depends on the reading order detection quality
- To detect the space between paragraphs, it's currently using an heuristics and doesn't detect automatically according the the interline. So if a paragraph have a large interline spacing, the algorithm may fail and create one paragraph per line. That said, this rarely occurs according to our experience.
---
### Overview (`server/src/processing/LinkDetectionModule/README.md`)
# Link Detection Module
## Purpose
The Link detection module detects URLs, GoTo and mailto links inside PDF metadata and it's text contents.
## What it does
It adds the property `targetURL` to the matching Words.
## Dependencies
None.
## How it works
1. It uses pdfminer's `dumppdf` utility and `xml-stream` library to process document metadata as XML and find links with their bounding boxes and page number.
2. Also for each word on document it uses two RegExp to match URL's or emails as strings and also set their `targetURL`
## Accuracy
All correctly detected links from the extractor are well preserved, and the accuracy can thus be reported to be _pretty good_.
## Limitations
- Any 'Action' type link inside metadata is ignored for now. (Can't match their respective bounding boxes).
---
### Overview (`server/src/processing/ListDetectionModule/README.md`)
# List Detection Module
## Purpose
Detects unordered and ordered lists in the text from paragraphs, using bullet points and numberings as indications.
## What it does
Creates new List typed elements in the document based on positively detected paragraphs.
## Dependencies
None
---
### Overview (`server/src/processing/MLHeadingDetectionModule/README.md`)
# ML Heading Detection Module
## Purpose
Create headings from a collection of paragraphs.
## What it does
It creates new heading elements by detecting headings lines from all lines contained in paragraphs.
## Dependencies
- [Lines to paragraph Module](../LinesToParagraphModule/README.md)
## How it works
It uses a series of Machine Learning methods to determine if a block of text is a Heading, based on properties like font styles, text color, number of words, etc.
Also, for all the detected Headings it calculates its level property based on font size, font weight, color, etc.
## Accuracy
Accuracy depends on the quality of the trained models.
Better trained models can be easily imported into this module by following the instruction in the [Training Module Guide](./train_model/README.md).
---
### Overview (`server/src/processing/NumberCorrectionModule/README.md`)
# Number Correction Module
## Purpose
Perform error detection and correction on financial numbers.
## What it does
It iterates on every lines and try to apply a correction on numbers such that:
- `13S` becomes `135`
- `o.oo` becomes `0.00`
- `1,802,86` becomes `1,802.86`
- `443 65` becomes `443.65`
This module has been made to perform error correction on scanned invoices.
## Dependencies
None
## How it works
It generates edits and use a scoring system to select the best candidate.
It tries to match a regex that can be change via the module's options and can also take a whitelist of elements that should be accepted as is.
## Accuracy
Good
## Limitations
- Can have some false positive and correct things that are not errors on numbers.
---
### Overview (`server/src/processing/OutOfPageRemovalModule/README.md`)
# Out of Page Removal Module
## Purpose
Removes all the elements of the page whose bounding boxes are not physically inside the page.
## What it does
It filters out only the elements which are inside the bounding boxes described in the page's `box` property.
## Dependencies
None
## How it works
It is a simple filter that checks if all the contents of `page.elements` for each page are inside the limits of the page described by `page.box`.
## Accuracy
Very high.
The boundaries of each page and element however, need to be correctly specified by the extractor.
## Limitations
None
---
### Overview (`server/src/processing/PageNumberDetectionModule/README.md`)
# Page Number Detection Module
## Purpose
Detects the page number for each page in a document.
## What it does
Searches for "number" paragraphs outside the page margins, and sets the 'isPageNumber' property to each matched paragraph in the document.
## Dependencies
- [Header Footer Detection](../HeaderFooterDetectionModule/README.md)
## Limitations
This module relies on *accurate page margins* set by the Header Footer Detection Module.
---
### Overview (`server/src/processing/ReadingOrderDetectionModule/README.md`)
# Reading Order Module
## Purpose
Detect the reading order of the document.
## What it does
It adds the `order` property on every element of the page. Elements can then be sorted according to this property.
## Dependencies
None
## How it works
It's based on a XY-cut approach with some optimization.
First, the algorithm will try to find possible vertical cuts in the page between elements. Then, it will perform cuts and try to find possible horizontal cuts in the left part, then the right part. For horizontal cuts, the algorithm will re-assemble blocks if there's some common vertical cuts. This improvement has been made to avoid splitting two columns of text of every line.
## Accuracy
Good
## Limitations
- It sometimes fails if bounding boxes are too far from each others.
- It doesn't work for right to left languages, but can be readapted easily in the code by inverting some functions and sorts.
- It doesn't work when there's a column are in an L shape.
---
### Overview (`server/src/processing/RedundancyDetectionModule/README.md`)
# Redundancy Detection Module
## Purpose
Detects and remove duplicate textual elements on each page of the document.
## What it does
Removes all the elements which exist as duplicates in the document.
If an element **A** has the same content and bounding box as another element **B**, then one of them will be removed.
## Dependencies
None
## How it works
1. Creates groups of text based on location, by checking between each pairs of elements if they are aligned and overlap vertically.
2. For each group, check if the elements are really duplicates by checking the concurrency of their content and bounding boxes.
3. If yes, we keep only one of the elements in the group and remove the others from the collection.
## Accuracy
Good
## Limitations
If the contents and location of the two duplicates are not identical, the removal might not take place.
The accuracy in that case will depend on the delta.
---
### Overview (`server/src/processing/RegexMatcherModule/README.md`)
# Regex Matcher Module
## Purpose
Matches the text against a certain number of Regular Expressions, configurable in the configuration.
## What it does
Generates new metadata info with RegexMetadata object, containing info about the match result and label for the word.
## Example
with config:
```json
{
"isCaseSensitive": true,
"isGlobal": true,
"queries": [
{
"label": "Number",
"regex": "[0-9]+"
},
{
"label": "unit",
"regex": "second|seconds|mph|kilometers|km"
}
]
}
```
`document.metadata` property after execution of this module:
```json
[
{
"id": 1,
"elements": [
561
],
"type": "regex",
"data": {
"name": "Number",
"regex": "[0-9]+",
"fullMatch": "2008",
"groups": []
}
},
{
"id": 2,
"elements": [
684
],
"type": "regex",
"data": {
"name": "unit",
"regex": "second|seconds|mph|kilometers|km",
"fullMatch": "mph",
"groups": []
}
}
]
```
*Here we can see that 2 words had been matched, with element id 561 and 684. Information about the regexp is also included.*
## Dependencies
None
## Accuracy
Results accuracy depends on the regular expressions sent in the configuration.
---
### Overview (`server/src/processing/RemoteModule/README.md`)
# Remote Module
## Purpose
This module is a bit different than others, because it doesn't change the document by itself.
It exports the document as [JSON](../../../../docs/json-output.md), call an API with it and expect a modified JSON back.
## How to use it
First of all, you need to have a small web server that will handle the API call.
You can use our [example of a remove module in python](../../../../demo/echo-module-py/README.md) as a start.
Your server needs to handle a HTTP `POST` request on the given URL, respond with the modified JSON.
## Dependencies
None
---
### Overview (`server/src/processing/SeparateWordsModule/README.md`)
# Separate Words Module Module
## Purpose
Separates words that are incorrectly joined by a punctuation mark.
## How it works
For every word in the document that is not in the dictionary, checks if it contains punctuation marks like . : , ; ! ?, etc and tries to find words that are separated by this marks.
## Dependencies
None
## Limitations
The module uses the words array of [an-array-of-english-words](https://www.npmjs.com/package/an-array-of-english-words) NPM package. So it can only work with documents written in english.
---
### Overview (`server/src/processing/TableDetection2Module/README.md`)
# Table Detection Module
## Purpose
Table extraction from the PDF file.
## What it does
Given a pdf it generates for each table detected a `table` element containing an array of `tableRows`.
## Dependencies
- Third-party python library [Tabula](https://pypi.org/project/tabula-py/)
- Third-party python library [Camelot](https://camelot-py.readthedocs.io/en/master/index.html)
- [Java JDK 8](https://www.oracle.com/java/technologies/javase-downloads.html)
---
### Overview (`server/src/processing/TableDetectionModule/README.md`)
# Table Detection Module
## Purpose
Table extraction from the PDF file.
## What it does
Given a pdf it generates for each table detected a `table` element containing an array of `tableRows`.
## Dependencies
- Third-party python library [Camelot](https://camelot-py.readthedocs.io/en/master/index.html)
## How it works
Given a table detected by Camelot this module fills `table` contents using and array of `tableRows` that are filled using an array of `tableCells` where each `tableCell` is filled by searching words in the document that are contained in cell bounds.
## Parameters
Following is an example of the configuration of the table-detection module:
```json
[
"table-detection",
{
"checkDrawings": true,
"runConfig": [
{
"pages": [1, 2, 3], // or [] for all pages
"flavor": "lattice",
"table_areas": []
}
]
}
]
```
- checkDrawings: Boolean value to specify if the module should look for table candidates using the previously detected Drawings. Default: true,
- runConfig: Array of different configurations for the doc
- pages: List of numbers representing pages.
- flavor: The parsing method to use ('lattice' or 'stream'). Lattice is used by default. For more information on each parsing method, [check this information](https://camelot-py.readthedocs.io/en/master/user/how-it-works.html).
- table_areas: Array of coordinates forming a box to indicate the position of a table with no outlines or visible borders.
Each table_area is a string of the form "x1,y1,x2,y2" where (x1, y1) -> top-left and (x2, y2) -> bottom-right in PDF coordinate space. In PDF coordinate space, the bottom-left corner of the page is the origin, with coordinates (0, 0).
**This parameter is optional and only used with 'stream' flavor.**
For more information on table_areas param, feel free to check [the camelot documentation](https://camelot-py.readthedocs.io/en/master/user/advanced.html#specify-table-areas).
## Accuracy
The accuracy is high.
## Limitations
- Only works with text-based PDFs and not scanned documents.
---
### Overview (`server/src/processing/TableOfContentsDetectionModule/README.md`)
# Table of Contents Detection Module
## Purpose
Detects and extracts tables of content from the PDF file.
## What it does
Given a pdf it generates a `TableOfContent` element containing an array of `TableOfContentItem`'s with info for each item.
## How it works
Searches for keywords and specific paragraph formats and then extracts the info from each detected paragraph with Regular Expressions
## Parameters
Following is an example of the configuration of the table-of-contents-detection module:
```json
[
"table-of-contents-detection",
{
"pageKeywords": [
"pag",
"pagina",
"page,
]
}
]
```
- pageKeywords: Optional. Array of "page" string to prepend and search for TOC items with format "page X - Section A".
Defaults to "pag".
## Accuracy
The accuracy is high on one-column documents.
---
### Overview (`server/src/processing/TemplateModule/README.md`)
A template module showing how to make an easily integrable template into the Parsr pipeline.
---
### Overview (`server/src/processing/WhitespaceRemovalModule/README.md`)
# Whitespace Removal Module
## Purpose
Removes textual elements containing only whitespace as content.
## What it does
Removes (filters out) all the textual elements not containing any content at all.
## Dependencies
None
## Parameters
`minWidth`: The minimum width to see if an element is at least a certain size for it to be taken into consideration as a candidate for removal
## How it works
All whitespace textual elements are checked to see if their width is less than `minWidth`, then checked if they are overlapping with other text elements (a very common case), and then deleted.
## Accuracy
Good. The module treats the most common cases, but its completeness is based on observation. New edge-cases might appear and will be interesting to treat in the future.
---
### Overview (`server/src/processing/WordsToLineModule/README.md`)
# Words to Line Module
## Purpose
Create lines from a bunch of words, according to the reading order.
## What it does
It creates new line elements that contains arrays of word elements.
## Dependencies
[Reading Order Module](../ReadingOrderDetectionModule/README.md)
## How it works
It simply takes every word one by one according to the reading order and stops and loops if the next word is on another line.
## Accuracy
Almost perfect
## Limitations
- It depends on the reading order detection quality
- It sometimes fails if lines are close to each others and there's an exponent
---
### Overview (`server/src/processing/WordsToLineNewModule/README.md`)
# Words to Line New Module
## Purpose
Create lines from a bunch of words inside pages.
## What it does
It creates new line elements that contains arrays of word elements.
## Dependencies
No dependencie.
## How it works
It takes every word of the page one by one, then it is splitted into line depending of vertical alignement, finaly each line can be splitted when a space between two words is big enough according to the average space and common space between words.
## Accuracy
Almost perfect.
## Limitations
If on specific case it is not working, you can tune up by incrementing / decrementing value of average and common space detected between words.
---
### Overview (`server/src/processing/MlHeadingDetectionModule/train_model/README.md`)
## Required dependencies
`python3 -m pip install spacy`
`python3 -m pip install markdown`
`python3 -m pip install imblearn`
`python3 -m pip install sklearn_porter`
`python3 -m pip install scikit-learn==0.22.2`
`python3 -m spacy download en_core_web_sm`
`pip3 install beautifulsoup4`
## Building the Dataset (this step is optional)
To build the dataset used to train the model run `python3 build_dataset.py
