# Technical Documentation: deanmalmgren/textract

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/deanmalmgren/textract)

### Changelog (`docs/changelog.rst`)

Change Log
==========

This project uses `semantic versioning <http://semver.org/>`_ to
track version numbers, where backwards incompatible changes
(highlighted in **bold**) bump the major version of the package.

NEXT RELEASE
-------------------

.. THANKS FOR CONTRIBUTING; ADD YOUR UNRELEASED CHANGES HERE!

2.1.0
-------------------

* Publish a wheel alongside the sdist on PyPI (`#560`_ by `@KyleKing`_)
* **Drop support for Python 3.9, now requiring 3.10+** (`#585`_ by `@KyleKing`_)
* Publish a ``-full`` Docker image to GHCR with all system dependencies
  preinstalled except LibreOffice, which most users don't need; includes
  pocketsphinx for offline audio transcription (`#600`_ by `@KyleKing`_)
* Minor nitpick to remove unused line (`#559`_ by `@KyleKing`_)
* Utilize openpyxl for xlsx after xlrd dropped support (`#572`_ by `@KyleKing`_)
* Prevent opening a cmd window with ``Popen()`` on Windows (`#574`_ by
  `@KyleKing`_)
* Replace unmaintained ``antiword`` with LibreOffice for ``.doc`` extraction
  (`#582`_ by `@KyleKing`_)
* Add ``.ods`` spreadsheet support (`#583`_ by `@KyleKing`_)
* Report which file extension triggered a missing module or missing
  executable error (`#575`_ by `@KyleKing`_)
* Standardize PDF text extraction to UTF-8 to avoid mangled non-Latin text
  (`#596`_ by `@KyleKing`_)
* Honor ``input_encoding`` in the csv, json, eml, and html parsers and add
  the ``--input-encoding`` CLI flag; introduces ``DecodedParser`` base class
  for parsers that operate on decoded text. A valid but incorrect
  ``input_encoding`` now raises a friendly ``InvalidInputEncoding`` error
  instead of a raw ``UnicodeDecodeError`` (`#573`_ by `@KyleKing`_)
* Beta: accept in-memory and streamed input via ``process_bytes``,
  ``process_stream``, and CLI ``-`` stdin, normalized through a new
  ``Source`` abstraction. csv reads lazily with an explicit
  ``input_encoding`` instead of buffering the whole document; docx opens
  in memory via the new ``BytesParser`` base class. **The core now invokes
  ``parser.process_source(source, ...)`` instead of ``parser.process(filename,
  ...)``, so third-party parsers that override ``process`` must migrate to
  ``process_source`` (or ``extract``/``extract_from_text``/
  ``extract_from_bytes``, which are still honored)** (`#588`_ by `@KyleKing`_)

2.0.0
-------------------

* Major modernization of textract absorbing the `textract-py3` fork (`#543`_ by `@KyleKing`_)

  * Implement GitHub Actions CI/CD and improve test coverage
  * Fix minor inconsistencies on Windows
  * Resolve dependency specification errors by migrating to uv
  * Update some minimum dependency versions and sets minimum Python to 3.9
  * Migrate to PyPi Trusted Publisher for releases

* Multiple Fixes:

  * Return null on empty stream (`#422`_ by `@TheElementalOfDestruction`_)
  * Enable encoding detection for the txt parser (`#456`_ by `@LoicGrobol`_)
  * Catch ShellError from pdf2txt.py (`#495`_ by `@dhrim`_)
  * Support python3.12 (`#502`_ by `@branchv`_)

* Documentation Updates

  * Add colons to issue template for consistency (`#520`_ by `@mcp292`_)
  * Fix a few typos (`#430`_ by `@timgates42`_)

1.6.5
-------------------

* switched epub parsing to MIT license compatible package (`#411`_ by
  `@jhale1805`_)

1.6.4
-------------------

* several bug fixes, including:

  * fixing dependency declarations (`#162`_ by `@lillypad`_)

1.6.1
-------------------

* several bug fixes, including:

  * fixing the readthedocs build (`#150`_)

1.6.0
-------------------

* Let the user provide file extension as an argument when the file name has no
  extension (`#148`_ by `@motazsaad`_)

* Added ability to parse audio with ``pocketsphinx`` (`#122`_ by `@barrust`_)

* Added ability to parse ``.psv`` and ``.tsv`` files (`#141`_)

* several bug fixes, including:

  * checking for the importability of a parser rather than the presense of the
    file (`#136`_ by `@AusIV`_)

  * manage versions with `bumpversion <https://pypi.python.org/pypi/bumpversion>`_
    (`#146`_)

  * properly reporting on missing external dependencies (`#139`_ by `@AusIV`_)

  * pin `chardet` to version 2.1.1 to avoid decode errors (`#107`_)

  * avoid unicode decode error with html parser (`#147`_ by `@suned`_)

  * enabling autocomplete and improving error handling (`#149`_)

1.5.0
-----

* Added python 3 support, including pdfminer (`#104`_ by `@sirex`_ via `#126`_)

* Python 3 support for ``pdfminer`` using ``pdfminer.six`` (`#116`_ by
  `@jaraco`_ via `#126`_)

* fixed security vulnerability by properly using subprocess.call (`#114`_ by
  `@pierre-ernst`_)

* updating to ``tesseract`` 3.03 (`#127`_)

* adding a ``.tif`` synonym for ``.tiff`` files (`#113`_ by `@onionradish`_)

* improved ``.docx`` support using ``docx2txt`` (`#100`_ by `@ankushshah89`_)

* several bug fixes, including:

  * including all requirements for ``Pillow`` (`#119`_ by `@akoumjian`_)

1.4.0
-----

* added layout preservation option for pdftotext pdf extractor (`#93`_ by
  `@ankushshah89`_)

* added simple support for extensionless filenames, treating them as plain
  ``.txt`` files (`#85`_)

* several bug fixes, including:

  * now extracting the text in tables from docx files at the end of the text
    extraction (`#92`_ by `@jsmith-mploir`_)

  * faster testing framework by only rebuilding test data when needed (`#90`_)

  * fixed ``.html`` and ``.epub`` parsers to deal with beautifulsoup4
    upgrades

  * using official ``msg-extractor`` now that it has a native ``setup.py``

  * updated tests for ``.html``, ``.ogg``, ``.wav``, and ``.mp3`` file types to
    be consistent with more recent versions of the underlying packages.

1.3.0
-----

* support for ``.rtf`` files (`#84`_)

* support for ``.msg`` files (`#87`_ and `#17`_ by `@anthonygarvan`_)

1.2.0
-----

* support for ``.tiff`` files (`#81`_)

* added support for other languages for tesseract (`#76`_ by `@anderser`_)

* added ``--option/-O`` flag to pass arbitrary arguments for things like
  languages into textract

* several bug fixes, including:

  * fix bug with doing OCR on multi-page pdfs and removing temporary directory
    (`#82`_ by `@pudo`_)

  * correctly accounting for whitespace in ``.odt`` documents (`#79`_
    by `@evfredericksen`_)

  * standardizing testing environment to be compatible with different versions
    of third-party command line tools (`#78`_)

1.1.0
-----

* support for ``.wav``, ``.mp3``, and ``.ogg`` files (`#56`_ and
  `#62`_ by `@arvindch`_)

* support for ``.csv`` files (`#64`_)

* support for scanned ``.pdf`` files with tesseract (`#66`_ by
  `@pudo`_)

* support for ``.htm`` files (`#69`_)

* several bug fixes, including:

  * ``.odt`` parser now correctly extracts text in order (`#61`_ by
    `@levivm`_)

  * fixed Docker development environment compatability with the
    Vagrant VM environment (`#73`_ by `@ShawnMilo`_)

* several internal improvements, including:

  * improvements in the python documentation (`#70`_)

  * improved html output with reduced whitespace around inline
    elements in output text (`#58`_ by `@eiotec`_)

1.0.0
-----

* **standardized encoding of output with** ``-e/--encoding`` **option**
  (`#39`_)

* support for ``.xls`` and ``.xlsx`` files (`#42`_ and `#55`_ by `@levivm`_)

* support for ``.epub`` files (`#40`_ by `@kokxx`_)

* several bug fixes, including:

  * removing tesseract version info from output of image parsers
    (`#48`_)

  * problems with spaces in filenames (`#53`_)

  * concurrancy problems with tesseract (`#44`_ by `@ShawnMilo`_,
    `#41`_ by `@christomitov`_)

* several internal improvements, including:

  * switching to using class-based parsers to abstract away the common
    functionality between different parser classes (`#39`_)

  * switching to using a python-based test suite and added
    standardized text tests to make sure output is consistent across
    file types (`#49`_)

  * including support for Docker-based testing (`#46`_ by `@ShawnMilo`_)

0.5.1
-----

* several bug fixes, including:

  * documentation fixes

  * shell commands hanging on large files (`#33`_)

0.5.0
-----

* support for ``.json`` files (`#13`_ by `@anthonygarvan`_)

* support for ``.odt`` files (`#29`_ by `@christomitov`_)

* support for ``.ps`` files (`#25`_)

* support for ``.gif``, ``.jpg``, ``.jpeg``, and ``.png`` files
  (`#30`_ by `@christomitov`_)

* several bug fixes, including:

  * improved fallback handling in ``.pdf`` parser if the ``pdftotext``
    command line utility isn't installed (`#26`_)

  * improved documentation for installation instructions on non-Ubuntu
    operating systems (`#21`_, `#26`_)

* several internal improvements, including:

  * cleaned up implementation of extension parsers to avoid magic

0.4.0
-----

* support for ``.html`` files (`#7`_)

* support for ``.eml`` files (`#4`_)

* automated the documentation for the python package using
  sphinx-apidoc in docs/Makefile (`#9`_)

0.3.0
-----

* support for ``.txt`` files, haha (`#8`_)

* fixed installation bug with not properly including requirements
  files in the manifest

0.2.0
-----

* support for ``.doc`` files (`#2`_)

* support for ``.pdf`` files (`#3`_)

* several bug fixes, including:

  * fixing tab complete bug no file paths (`#6`_)

  * fixing tests to make sure the work properly on travis-ci

0.1.0
-----

* Initial release, support for ``.docx`` and ``.pptx``

.. list of contributors that are linked to above. putting links here
.. to make the text above relatively clean

.. _@akoumjian: https://github.com/akoumjian
.. _@anthonygarvan: https://github.com/anthonygarvan
.. _@anderser: https://github.com/anderser
.. _@ankushshah89: https://github.com/ankushshah89
.. _@arvindch: https://github.com/arvindch
.. _@barrust: https://github.com/barrust
.. _@branchv: https://github.com/branchv
.. _@AusIV: https://github.com/AusIV
.. _@christomitov: https://github.com/christomitov
.. _@dhrim: https://github.com/dhrim
.. _@eiotec: https://github.com/eiotec
.. _@evfredericksen: https://github.com/evfredericksen
.. _@jaraco: https://github.com/jaraco
.. _@jhale1805: https://github.com/jhale1805
.. _@jsmith-mploir: https://github.com/jsmith-mploir
.. _@kokxx: https://github.com/Kokxx
.. _@KyleKing: https://github.com/KyleKing
.. _@levivm: https://github.com/levivm
.. _@lillypad: https://github.com/lillypad
.. _@LoicGrobol: https://github.com/LoicGrobol
.. _@mcp292: https://github.com/mcp292
.. _@motazsaad: https://github.com/motazsaad
.. _@onionradish: https://github.com/onionradish
.. _@pierre-ernst: https://github.com/pierre-ernst
.. _@pudo: https://github.com/pudo
.. _@ShawnMilo: https://github.com/ShawnMilo
.. _@sirex: https://github.com/sirex
.. _@suned: https://github.com/suned
.. _@TheElementalOfDestruction: https://github.com/TheElementalOfDestruction
.. _@timgates42: https://github.com/timgates42

.. list of issues that have been resolved. putting links here to make
.. the text above relatively clean

.. _#2: https://github.com/deanmalmgren/textract/issues/2
.. _#3: https://github.com/deanmalmgren/textract/issues/3
.. _#4: https://github.com/deanmalmgren/textract/issues/4
.. _#6: https://github.com/deanmalmgren/textract/issues/6
.. _#7: https://github.com/deanmalmgren/textract/issues/7
.. _#8: https://github.com/deanmalmgren/textract/issues/8
.. _#9: https://github.com/deanmalmgren/textract/issues/9
.. _#13: https://github.com/deanmalmgren/textract/issues/13
.. _#17: https://github.com/deanmalmgren/textract/issues/17
.. _#21: https://github.com/deanmalmgren/textract/issues/21
.. _#25: https://github.com/deanmalmgren/textract/issues/25
.. _#26: https://github.com/deanmalmgren/textract/issues/26
.. _#29: https://github.com/deanmalmgren/textract/issues/29
.. _#30: https://github.com/deanmalmgren/textract/issues/30
.. _#33: https://github.com/deanmalmgren/textract/issues/33
.. _#39: https://github.com/deanmalmgren/textract/issues/39
.. _#40: https://github.com/deanmalmgren/textract/issues/40
.. _#41: https://github.com/deanmalmgren/textract/issues/41
.. _#42: https://github.com/deanmalmgren/textract/issues/42
.. _#44: https://github.com/deanmalmgren/textract/issues/44
.. _#46: https://github.com/deanmalmgren/textract/issues/46
.. _#48: https://github.com/deanmalmgren/textract/issues/48
.. _#49: https://github.com/deanmalmgren/textract/issues/49
.. _#53: https://github.com/deanmalmgren/textract/issues/53
.. _#55: https://github.com/deanmalmgren/textract/issues/55
.. _#56: https://github.com/deanmalmgren/textract/issues/56
.. _#58: https://github.com/deanmalmgren/textract/issues/58
.. _#61: https://github.com/deanmalmgren/textract/issues/61
.. _#62: https://github.com/deanmalmgren/textract/issues/62
.. _#64: https://github.com/deanmalmgren/textract/issues/64
.. _#66: https://github.com/deanmalmgren/textract/issues/66
.. _#69: https://github.com/deanmalmgren/textract/issues/69
.. _#70: https://github.com/deanmalmgren/textract/issues/70
.. _#73: https://github.com/deanmalmgren/textract/issues/73
.. _#76: https://github.com/deanmalmgren/textract/issues/76
.. _#78: https://github.com/deanmalmgren/textract/issues/78
.. _#79: https://github.com/deanmalmgren/textract/issues/79
.. _#81: https://github.com/deanmalmgren/textract/issues/81
.. _#82: https://github.com/deanmalmgren/textract/issues/82
.. _#84: https://github.com/deanmalmgren/textract/issues/84
.. _#85: https://github.com/deanmalmgren/textract/issues/85
.. _#87: https://github.com/deanmalmgren/textract/issues/87
.. _#90: https://github.com/deanmalmgren/textract/issues/90
.. _#92: https://github.com/deanmalmgren/textract/issues/92
.. _#93: https://github.com/deanmalmgren/textract/issues/93
.. _#100: https://github.com/deanmalmgren/textract/issues/100
.. _#104: https://github.com/deanmalmgren/textract/issues/104
.. _#107: https://github.com/deanmalmgren/textract/issues/107
.. _#113: https://github.com/deanmalmgren/textract/issues/113
.. _#114: https://github.com/deanmalmgren/textract/issues/114
.. _#116: https://github.com/deanmalmgren/textract/issues/116
.. _#119: https://github.com/deanmalmgren/textract/issues/119
.. _#122: https://github.com/deanmalmgren/textract/issues/122
.. _#126: https://github.com/deanmalmgren/textract/issues/126
.. _#127: https://github.com/deanmalmgren/textract/issues/127
.. _#136: https://github.com/deanmalmgren/textract/issues/136
.. _#139: https://github.com/deanmalmgren/textract/issues/139
.. _#141: https://github.com/deanmalmgren/textract/issues/141
.. _#146: https://github.com/deanmalmgren/textract/issues/146
.. _#147: https://github.com/deanmalmgren/textract/issues/147
.. _#148: https://github.com/deanmalmgren/textract/issues/148
.. _#149: https://github.com/deanmalmgren/textract/issues/149
.. _#150: https://github.com/deanmalmgren/textract/issues/150
.. _#162: https://github.com/deanmalmgren/textract/issues/162
.. _#411: https://github.com/deanmalmgren/textract/issues/411
.. _#422: https://github.com/deanmalmgren/textract/pull/422
.. _#430: https://github.com/deanmalmgren/textract/pull/430
.. _#456: https://github.com/deanmalmgren/textract/pull/456
.. _#495: https://github.com/deanmalmgren/textract/pull/495
.. _#502: https://github.com/deanmalmgren/textract/pull/502
.. _#520: https://github.com/deanmalmgren/textract/pull/520
.. _#543: https://github.com/deanmalmgren/textract/pull/543
.. _#559: https://github.com/deanmalmgren/textract/pull/559
.. _#560: https://github.com/deanmalmgren/textract/pull/560
.. _#572: https://github.com/deanmalmgren/textract/pull/572
.. _#573: https://github.com/deanmalmgren/textract/pull/573
.. _#574: https://github.com/deanmalmgren/textract/pull/574
.. _#575: https://github.com/deanmalmgren/textract/pull/575
.. _#582: https://github.com/deanmalmgren/textract/pull/582
.. _#583: https://github.com/deanmalmgren/textract/pull/583
.. _#585: https://github.com/deanmalmgren/textract/pull/585
.. _#588: https://github.com/deanmalmgren/textract/pull/588
.. _#596: https://github.com/deanmalmgren/textract/pull/596
.. _#600: https://github.com/deanmalmgren/textract/pull/600

---

### Command Line Interface (`docs/command_line_interface.rst`)

.. _command-line-interface:

Command line interface
======================

textract
--------

.. argparse::
   :module: textract.cli
   :func: get_parser
   :prog: textract

.. note::

    To make the command line interface as usable as possible,
    autocompletion of available options with textract is enabled by
    @kislyuk's amazing `argcomplete
    <https://github.com/kislyuk/argcomplete>`_ package.  Follow
    instructions to `enable global autocomplete
    <https://github.com/kislyuk/argcomplete#activating-global-completion>`_
    and you should be all set.

.. note::

    **Beta:** pass ``-`` as the filename to read the document from stdin
    instead of a file on disk, so textract works as a pipe::

        curl https://example.com/report.pdf | textract --extension pdf -

    Since there's no filename to sniff the extension from, ``--extension``
    is required. For csv, adding ``--input-encoding`` also lets textract
    read and decode the input lazily instead of buffering the whole
    document, so a large piped csv doesn't need to be fully loaded before
    processing starts::

        cat huge.csv | textract --extension csv --input-encoding utf_8 -

    This currently only holds for csv; other formats still buffer the
    document. See :ref:`python-package` for the equivalent library
    functions (``process_bytes``, ``process_stream``) and their current
    limitations.

---

### Contributing (`docs/contributing.rst`)

.. _contributing:

Contributing
============

The overarching goal of this project is to make it as easy as possible
to extract raw text from any document for the purposes of most natural
language processing tasks. In practice, this means that this project
should preferentially provide tools that correctly produce output that
has words in the correct order but that whitespace between words,
formatting, etc is totally irrelevant. As the various parsers mature,
I fully expect the output to become more readable to support
additional use cases, like `extracting text to appear in web pages
<https://github.com/deanmalmgren/textract/pull/58#issuecomment-53697943>`_.

Importantly, this project is committed to being as agnostic about how
the content is extracted as it is about the means in which the text is
analyzed downstream. This means that ``textract`` should support
multiple modes of extracting text from any document and provide
reasonably good defaults (defaulting to tools that tend to produce the
correct word sequence).

Another important aspect of this project is that we want to have
extremely good documentation. If you notice a type-o, error, confusing
statement etc, please fix it!


.. _contributing-quick-start:

Quick start
-----------

1. `Fork <https://github.com/deanmalmgren/textract/fork>`_ and clone the
   project:

   .. code-block:: bash

        git clone https://github.com/YOUR-USERNAME/textract.git

2. Install dependencies and run tests:

   .. code-block:: bash

        make sync
        uv run pytest

3. Install system dependencies for your platform. See the
   :ref:`installation <installation>` guide for platform-specific
   instructions (macOS, Ubuntu/Debian, Windows, FreeBSD).

4. Contribute! There are several `open issues
   <https://github.com/deanmalmgren/textract/issues>`_ that provide
   good places to dig in and send pull requests; your help is greatly
   appreciated!

   Current build status: |Build Status|


.. |Build Status| image:: https://travis-ci.org/deanmalmgren/textract.png
   :target: https://travis-ci.org/deanmalmgren/textract


Contribution workflow
---------------------

Any and all contributions are welcome and appreciated. To make it easy
to keep things organized, this project uses the
`general guidelines <https://guides.github.com/introduction/flow/>`_
for the fork-branch-pull request model for github. Briefly, this means:

1. Make sure your fork's ``master`` branch is up to date:

   .. code-block:: bash

        git remote add deanmalmgren https://github.com/deanmalmgren/textract.git
        git checkout master
        git pull deanmalmgren/master

2. Start a feature branch with a descriptive name about what you're
   trying to accomplish:

   .. code-block:: bash

        git checkout -b csv-support

3. Make commits to this feature branch (``csv-support``, in this case)
   in a way that other people can understand with good commit messages
   to explain the changes you've made:

   .. code-block:: bash

        code textract/parsers/csv_parser.py
        git add textract/parsers/csv_parser.py
        git commit -m 'feat: added csv_parser'

4. Open a PR on GitHub with your changes, explain the motivation for
   the PR, any manual testing, and link any relevant issues:

   .. code-block:: bash

        git push origin csv-support
        chrome http://github.com/deanmalmgren/textract/compare


Common contributions: support for new file type
------------------------------------------------

This project has really taken off, much more so than I would have
thought (thanks everybody!). To help new contributors, I thought I'd
jot down some notes for one of the more common contributions---how to
add support for hitherto unsupported file type ``.abc123``:

* write a ``Parser`` class in ``textract/parsers/abc123_parser.py`` that
  inherits from ``textract.parsers.utils.BaseParser`` or
  ``textract.parsers.utils.ShellParser`` and implements the
  ``extract(self, filename, **kwargs)`` method.

* add a test file in ``tests/abc123/raw_text.abc123`` and generate the
  expected output. For most file types that use pure Python libraries,
  run textract on it:

  .. code-block:: shell

     textract tests/abc123/raw_text.abc123 > tests/abc123/raw_text.txt

  For file types that require external tools (PDF OCR, image OCR, PostScript),
  add the file to ``tests/Makefile`` and run:

  .. code-block:: shell

     cd tests && make

  Then add the basic test suite by creating
  a file called ``tests/test_abc123.py`` with content that looks
  something like this:

  .. code-block:: python

     # tests/test_abc123.py
     import unittest

     import base


     class Abc123TestCase(unittest.TestCase, base.BaseParserTestCase):
         extension = 'abc123'

  now you should be able to run tests on your parser with ``uv run pytest
  tests/test_abc123.py`` or the tests for every parser with ``uv run pytest``.

* if your package relies on any external sources, be sure to add them
  in ``pyproject.toml`` (for python packages) or document system
  dependencies and update the installation documentation accordingly in
  ``docs/installation.rst``.

* add documentation about the awesome new file format this is being
  supported in ``docs/index.rst``

* finally, make sure the entire test suite passes by running
  ``uv run pytest`` and fix any lingering problems.


Style guidelines
----------------

As a general rule of thumb, the goal of this package is to be as
readable as possible to make it easy for novices and experts alike to
contribute to the source code in meaningful ways. Pull requests that
favor cleverness or optimization over readability are less likely to be
incorporated.

To make this notion of "readability" more concrete, here are a few
stylistic guidelines that are inspired by other projects and we
generally recommend:

-  write functions and methods that can `fit on a screen or two of a
   standard
   terminal <https://www.kernel.org/doc/Documentation/CodingStyle>`_
   --- no more than approximately 40 lines.

-  unless it makes code less readable, adhere to `PEP
   8 <http://legacy.python.org/dev/peps/pep-0008/>`_ style
   recommendations --- use an appropriate amount of whitespace.

-  `code comments should be about *what* is being done, not *how* it is
   being done <https://www.kernel.org/doc/Documentation/CodingStyle>`_
   --- that should be self-evident from the code itself.


Development Setup
-----------------

Install dependencies::

    make sync

This runs ``uv sync`` and, on macOS, re-signs any compiled C extensions that
may have an invalid code signature due to Python 3.14 build tooling. See
`apple codesigning and scikit-build-core <https://github.com/scikit-build/scikit-build-core/issues>`_
for upstream context.

Install system dependencies (macOS, using `Homebrew <https://brew.sh/>`_)::

    brew install libreoffice tesseract ghostscript poppler sox unrtf

Install system dependencies (Ubuntu/Debian)::

    apt-get install libreoffice-writer tesseract-ocr ghostscript poppler-utils sox libsox-fmt-mp3 unrtf

Install system dependencies (Windows, using `Chocolatey <https://chocolatey.org/install>`_)::

    choco install tesseract ghostscript sox.portable poppler libreoffice-fresh -y

.. note::

   The canonical list of system dependencies is in `.github/actions/setup/action.yml
   <https://github.com/deanmalmgren/textract/blob/master/.github/actions/setup/action.yml>`_.


Optional: pre-commit checks with hk
------------------------------------

This isn't required to contribute, but you can install `hk <https://hk.jdx.dev/>`_ to run the
same ``ruff``, ``ruff format``, and ``pyright`` checks (defined in ``hk.pkl``) automatically as
git hooks, instead of running ``make lint``/``make fix`` by hand.

hk is installed via `mise <https://mise.jdx.dev/>`_:

.. code-block:: bash

    curl https://mise.run | sh
    mise use -g hk
    hk install --mise

``hk install --mise`` wires up the git hooks and tells hk to use mise to fetch any additional
tools its steps need. Once installed, hk runs automatically on ``git commit``/``git push``; run
it manually with ``hk check --all`` or ``hk fix --all``.


Releasing
---------

Versioning follows `semantic versioning <http://semver.org/>`_. The version is declared in two
places: ``pyproject.toml`` (``version``) and ``textract/__init__.py`` (``VERSION``). Use the
helper script to update both atomically:

.. code-block:: bash

    python scripts/bump_version.py 1.7.0

**Update the changelog** in ``docs/changelog.rst`` — add a new section above the previous release
with a short summary of notable changes. See existing entries for the format.

**Tag and push:**

.. code-block:: bash

    git add pyproject.toml textract/__init__.py docs/changelog.rst
    git commit -m "chore: release 1.7.0"
    git tag v1.7.0
    git push --follow-tags

CI runs tests, creates a GitHub Release with auto-generated notes from merged PRs, and publishes to PyPI.

---

### Index (`docs/index.rst`)

.. textract documentation master file, created by
   sphinx-quickstart on Fri Jul  4 11:09:09 2014.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

textract
================================

As undesirable as it might be, more often than not there is extremely
useful information embedded in Word documents, PowerPoint
presentations, PDFs, etc---so-called "dark data"---that would be
valuable for further textual analysis and visualization. While
:ref:`several packages <supporting>` exist for extracting content from
each of these formats on their own, this package provides a single
interface for extracting content from any type of file, without any
irrelevant markup.

This package provides two primary facilities for doing this, the
:ref:`command line interface <command-line-interface>`

.. code-block:: bash

    textract path/to/file.extension

or the :ref:`python package <python-package>`

.. code-block:: python

    # some python file
    import textract
    text = textract.process("path/to/file.extension")

.. _supporting:

Currently supporting
--------------------

textract supports a growing list of file types for text extraction. If
you don't see your favorite file type here, please recommend other
file types by either mentioning them on the `issue tracker
<https://github.com/deanmalmgren/textract/issues>`_ or by
:ref:`contributing a pull request <contributing>`.


* ``.csv`` via python builtins

* ``.tsv`` and ``.tab`` via python builtins

* ``.doc`` via `LibreOffice`_ (optional; see installation docs)

* ``.docx`` via `python-docx2txt`_

* ``.eml`` via python builtins

* ``.epub`` via `ebooklib`_

* ``.gif`` via `tesseract-ocr`_

* ``.jpg`` and ``.jpeg`` via `tesseract-ocr`_

* ``.json`` via python builtins

* ``.html`` and ``.htm`` via `beautifulsoup4`_

* ``.mp3`` via `sox`_, `SpeechRecognition`_, and `pocketsphinx`_

* ``.msg`` via `msg-extractor`_

* ``.odt`` via python builtins

* ``.ods`` via `odfpy <https://pypi.python.org/pypi/odfpy>`_

* ``.ogg`` via `sox`_, `SpeechRecognition`_, and `pocketsphinx`_

* ``.pdf`` via `pdftotext`_ (default) or `pdfminer.six`_

* ``.png`` via `tesseract-ocr`_

* ``.pptx`` via `python-pptx`_

* ``.ps`` via `ps2ascii`_

* ``.rtf`` via `unrtf`_

* ``.tiff`` and ``.tif`` via `tesseract-ocr`_

* ``.txt`` via python builtins

* ``.wav`` via `SpeechRecognition`_ and `pocketsphinx`_

* ``.xlsx`` via `xlrd <https://pypi.python.org/pypi/xlrd>`_

* ``.xls`` via `xlrd <https://pypi.python.org/pypi/xlrd>`_

.. this is a list of all the packages that textract uses for extraction
.. _LibreOffice: https://www.libreoffice.org/
.. _beautifulsoup4: http://beautiful-soup-4.readthedocs.org/en/latest/
.. _ebooklib: https://github.com/aerkalov/ebooklib
.. _msg-extractor: https://github.com/mattgwwalker/msg-extractor
.. _pdfminer.six: https://github.com/goulu/pdfminer
.. _pdftotext: http://poppler.freedesktop.org/
.. _pocketsphinx: https://github.com/cmusphinx/pocketsphinx/
.. _ps2ascii: https://www.ghostscript.com/doc/current/Use.htm
.. _python-docx2txt: https://github.com/ankushshah89/python-docx2txt
.. _python-pptx: https://python-pptx.readthedocs.org/en/latest/
.. _SpeechRecognition: https://pypi.python.org/pypi/SpeechRecognition/
.. _sox: http://sox.sourceforge.net/
.. _tesseract-ocr: https://code.google.com/p/tesseract-ocr/
.. _unrtf: http://www.gnu.org/software/unrtf/

.. _related-projects:

Related projects
----------------

Of course, textract isn't the first project with the aim to provide a
simple interface for extracting text from any document. But this is,
to the best of my knowledge, the only project that is written in
python (a language commonly chosen by the natural language processing
community) and is :ref:`method agnostic about how content is extracted
<contributing>`. I'm sure that there are other similar projects out
there, but here is a small sample of similar projects:

* `Apache Tika <http://tika.apache.org/>`_ has `very similar, if not
  identical, aims as textract
  <https://github.com/deanmalmgren/textract/issues/12>`_ and has
  impressive coverage of a wide range of file formats. It is written
  in java.

* `textract (node.js) <https://github.com/dbashford/textract>`_ has
  similar aims as this textract package (including an identical name!
  great minds...). It is written in node.js.

* `pandoc <http://johnmacfarlane.net/pandoc/>`_ is intended to be a
  document conversion tool (a much more difficult task!), but it does have
  `the ability to convert to plain text
  <http://johnmacfarlane.net/pandoc/demos.html>`_. It is written in
  Haskell.


Contents:

.. toctree::
   :maxdepth: 2

   command_line_interface
   python_package
   installation
   platform_differences
   contributing
   changelog


Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

---

### Installation (`docs/installation.rst`)

.. _installation:

Installation
============

One of the main goals of textract is to make it as easy as possible to
start using textract (meaning that installation should be as quick and
painless as possible). This package is built on top of several python
packages and other source libraries. Python packages are installed
automatically when using ``pip`` or ``uv``. The source libraries are a
separate matter though and largely depend on your operating system.

Modern Python tooling options:

.. code-block:: bash

    # One-off execution
    uvx textract path/to/file.pdf

    # Install as tool
    uv tool install textract

Ubuntu / Debian
---------------

First install system packages using apt-get, then install textract from PyPI:

.. code-block:: bash

    apt-get install python-dev libxml2-dev libxslt1-dev unrtf poppler-utils ghostscript tesseract-ocr \
    flac ffmpeg lame libmad0 libsox-fmt-mp3 sox libjpeg-dev swig libpulse-dev
    pip install textract
    # or with uv
    uv pip install textract

.. note::

    It may also be necessary to install ``zlib1g-dev`` on Docker
    instances of Ubuntu. See `issue #19
    <https://github.com/deanmalmgren/textract/pull/19>`_ for details

macOS
-----

These steps rely on you having `Homebrew <https://brew.sh/>`_ installed.
First install system packages, then install textract from PyPI:

.. code-block:: bash

    brew install ghostscript poppler sox tesseract unrtf swig
    pip install textract
    # or with uv
    uv pip install textract

.. note::

    ``ghostscript`` provides ``ps2ascii`` for ``.ps`` file extraction.

.. note::

    Depending on how you have python configured on your system with
    homebrew, you may also need to install the python
    development header files for textract to properly install.

.. note::

    **pocketsphinx and Python 3.14+ on macOS**: No prebuilt wheel exists yet
    for Python 3.14 on macOS ARM64. When installed from source,
    scikit-build-core's post-build steps invalidate the code signature, causing
    macOS to kill the process with ``SIGKILL (Code Signature Invalid)`` when the
    extension loads. If you see a ``RuntimeError`` mentioning an invalid code
    signature when using ``method="sphinx"``, re-sign the extension:

    .. code-block:: bash

        python -c "
        import subprocess
        from pathlib import Path
        import importlib.util
        spec = importlib.util.find_spec('pocketsphinx')
        for loc in (spec.submodule_search_locations or []):
            for so in Path(loc).glob('_pocketsphinx.cpython-*-darwin.so'):
                subprocess.run(['codesign', '-s', '-', '-f', str(so)], check=True)
                print(f'Re-signed: {so}')
        "

    This is a known upstream issue; once a Python 3.14 macOS wheel is published
    to PyPI the workaround will no longer be needed.

Windows
-------

Install `Chocolatey <https://chocolatey.org/install>`_ then install system packages:

.. code-block:: powershell

    choco install tesseract ghostscript sox.portable poppler -y
    pip install textract
    # or with uv
    uv pip install textract

.. note::

    Two parsers are **not supported on Windows**:

    - ``.mp3`` / ``.ogg``: `sox.portable <https://community.chocolatey.org/packages/sox.portable>`_
      does not include ``libmad``. SoX dynamically loads ``libmad.dll`` for MP3
      decoding but does not ship it due to patent restrictions — see the
      `SoX mailing list discussion <https://sourceforge.net/p/sox/mailman/message/27169341/>`_.
      Use Linux or macOS where ``libsox-fmt-mp3``/``mad`` are available.

    - ``.rtf``: `unrtf <https://www.gnu.org/software/unrtf/>`_ is a GNU project
      with no Windows port and no Chocolatey package. RTF extraction is only
      available on Linux (``apt install unrtf``) and macOS (``brew install unrtf``).

FreeBSD
-------

First install system packages using pkg, then install textract from PyPI:

.. code-block:: bash

    pkg install lang/python38 devel/py-pip textproc/libxml2 textproc/libxslt textproc/unrtf \
    graphics/poppler print/pstotext graphics/tesseract audio/flac multimedia/ffmpeg audio/lame audio/sox \
    graphics/jpeg-turbo
    pip install textract
    # or with uv
    uv pip install textract

RHEL / Rocky Linux / AlmaLinux / CentOS Stream
-----------------------------------------------

textract isn't tested in CI on any Red Hat family distro (GitHub Actions has no
hosted RHEL runner), but ``scripts/test_redhat_family_install.sh`` in this
repository probes it against real images with Docker. On Rocky Linux 8/9,
AlmaLinux 9 and CentOS Stream 9, install the system packages plus a newer
Python from AppStream/EPEL (the system ``python3`` is 3.9 on EL9 and 3.6 on
EL8, both below textract's ``>=3.10`` floor), then build your virtualenv with
that newer Python:

.. code-block:: bash

    # EL9 (Rocky 9, AlmaLinux 9, CentOS Stream 9)
    dnf install -y epel-release
    dnf config-manager --set-enabled crb   # powertools on EL8
    dnf install -y python3.13 python3.13-devel python3.13-pip gcc \
        libxml2-devel libxslt-devel libjpeg-turbo-devel swig \
        tesseract sox ghostscript poppler-utils libreoffice-writer
    python3.13 -m venv .venv && source .venv/bin/activate
    pip install textract

.. note::

    On Rocky Linux 8, use ``python3.11`` instead of ``python3.12``:
    AppStream's ``python3.12`` package links against a newer ``libexpat``
    than EL8 ships, and ``pip`` fails at import time with
    ``undefined symbol: XML_SetBillionLaughsAttackProtectionMaximumAmplification``.
    ``python3.11`` doesn't hit this.

.. note::

    ``.rtf`` is **not supported**: ``unrtf`` isn't packaged for EL8 or EL9,
    in EPEL or elsewhere. Pre-convert with LibreOffice instead, the same
    workaround used for legacy ``.doc`` files
    (see :ref:`converting-legacy-doc-files`):

    .. code-block:: bash

        soffice --headless --convert-to txt --outdir out/ *.rtf

Every other extension, including ``.doc`` via LibreOffice and ``.mp3``/``.wav``
via ``--method sphinx``, is confirmed working on all four distros above.

Red Hat's own UBI9 container images (``registry.access.redhat.com/ubi9/ubi``)
are **not sufficient** on their own: without a paid RHEL subscription
registered via ``subscription-manager``, UBI9's mirrored repos don't carry
``tesseract``, ``libreoffice-writer``, ``swig`` or a Python newer than the
system 3.9, even with EPEL enabled. Use Rocky Linux, AlmaLinux or CentOS
Stream instead, or register the subscription to unlock RHEL's full AppStream
content.

.. note::

    These results are from ``linux/arm64`` images (Docker on Apple Silicon);
    package availability has not been separately verified on ``linux/amd64``.

Docker
------

A container image is published to `GHCR
<https://github.com/deanmalmgren/textract/pkgs/container/textract>`_ with every
system dependency (tesseract, ghostscript, poppler, sox, pocketsphinx)
preinstalled, if you want to try out all of textract's capabilities without
local installation and/or want to run textract in a hosted environment.

.. code-block:: bash

    docker run --rm -v "$PWD:/data" ghcr.io/deanmalmgren/textract:latest-full /data/path/to/file.pdf

Any CLI option works the same way, for example forcing OCR on a scanned PDF:

.. code-block:: bash

    docker run --rm -v "$PWD:/data" ghcr.io/deanmalmgren/textract:latest-full --method tesseract /data/scan.pdf

Audio transcription defaults to Google's network speech API; the image also
includes pocketsphinx, so ``--method sphinx`` works offline:

.. code-block:: bash

    docker run --rm -v "$PWD:/data" ghcr.io/deanmalmgren/textract:latest-full --method sphinx /data/audio.wav

The image doesn't include LibreOffice by default; see
:ref:`converting-legacy-doc-files` for how to add it if you need ``.doc``
support.

.. _converting-legacy-doc-files:

Converting legacy ``.doc`` files
--------------------------------

Legacy ``.doc`` (Word 97-2003 binary) files are extracted with LibreOffice,
which is an *optional* runtime dependency: textract only shells out to the
``soffice`` executable when you actually process a ``.doc`` file, and every
other format works without it. None of the install instructions above include
it, since most textract users never touch legacy ``.doc`` files.

If you do need it, install it yourself:

.. code-block:: bash

    # Ubuntu / Debian
    apt-get install libreoffice-writer

    # macOS (via Homebrew)
    brew install --cask xquartz libreoffice

    # Windows (via Chocolatey)
    choco install libreoffice-fresh -y

    # FreeBSD
    pkg install editors/libreoffice

    # Docker: add to the apt-get install line in the Dockerfile and rebuild
    apt-get install libreoffice-writer dbus-x11

Once ``soffice`` is on your ``PATH``, textract picks it up automatically the
next time it processes a ``.doc`` file.

If you would rather not install LibreOffice alongside textract at all (for
example in a slim container), you can pre-convert your ``.doc`` files to
``.docx`` once, elsewhere, and hand the results to textract, whose ``.docx``
parser is pure Python:

.. code-block:: bash

    soffice --headless --convert-to docx --outdir out/ *.doc

Then run ``textract out/whatever.docx`` as usual. This keeps the heavier
converter out of your extraction pipeline while still supporting legacy
documents.

.. _ocr-troubleshooting:

Troubleshooting OCR (images and scanned PDFs)
---------------------------------------------

Text extraction from images (``.jpg``, ``.png``, ``.gif``) and scanned PDFs
relies on `tesseract-ocr <https://tesseract-ocr.github.io/>`_. If textract
returns empty output, whitespace, or placeholder characters instead of text,
tesseract is usually either not installed or not on your ``PATH``.

Verify that tesseract is installed and reachable:

.. code-block:: bash

    tesseract --version      # prints the installed version
    which tesseract          # Linux / macOS: prints the resolved path
    where tesseract          # Windows (PowerShell / cmd): prints the resolved path

If those commands fail, install tesseract with your platform's package manager
(see the sections above) and re-open a shell so the updated ``PATH`` takes
effect.

Confirm tesseract can read your file on its own, independent of textract:

.. code-block:: bash

    tesseract scanned_page.png stdout

If that prints the expected text but ``textract scanned_page.png`` does not,
the problem is in how textract invokes tesseract rather than tesseract itself,
and is worth reporting as a bug.

.. note::

    Scanned PDFs need the ``tesseract`` method explicitly, because the default
    ``pdftotext`` path only extracts an existing text layer (which a scan does
    not have):

    .. code-block:: bash

        textract --method tesseract scanned.pdf

    This also requires ``pdftoppm`` (part of poppler) to rasterize each page
    before OCR.

Common failure modes:

- If textract fails with "The command ``tesseract`` failed because the
  executable is not installed", install ``tesseract-ocr`` and confirm
  ``tesseract --version`` works in the same shell you run textract from.
- Empty output or image placeholders mean tesseract isn't running. Test it
  directly with ``tesseract <file> stdout`` and check your ``PATH``.
- Wrong or missing characters for non-English text mean the language pack is
  missing. Install it for your language (for example ``tesseract-ocr-ces`` for
  Czech on Debian/Ubuntu) so tesseract can recognize the script.

Reference: CI System Dependencies
----------------------------------

The canonical list of system dependencies is maintained in the GitHub Actions
workflow at `.github/actions/setup/action.yml <https://github.com/deanmalmgren/textract/blob/main/.github/actions/setup/action.yml>`_.
This is what CI uses and is kept up-to-date with each platform's requirements.

Don't see your operating system installation instructions here?
---------------------------------------------------------------

My apologies! Installing system packages is a bit of a drag and its
hard to anticipate all of the different environments that need to be
accommodated (wouldn't it be awesome if there were a system-agnostic
package manager or, better yet, if python could install these system
dependencies for you?!?!). If you're operating system doesn't have
documentation about how to install the textract dependencies, please
:ref:`contribute a pull request <contributing>` with:

1. A new section in here with the appropriate details about how to
   install things. In particular, please give instructions for how to
   install the following libraries before running ``pip install
   textract``:

    - `libxml2 2.6.21 or later <http://xmlsoft.org/downloads.html>`_
      is required by the ``.docx`` parser which uses `lxml
      <http://lxml.de/installation.html#requirements>`_ via
      python-docx.

    - `libxslt 1.1.15 or later
      <http://xmlsoft.org/XSLT/downloads.html>`_ is required by the
      ``.docx`` parser which users `lxml
      <http://lxml.de/installation.html#requirements>`_ via
      python-docx.

    - python header files are required for building lxml.

    - `LibreOffice <https://www.libreoffice.org/>`_ (the ``soffice``
      executable) is *optionally* required by the ``.doc`` parser. It is only
      invoked when you extract a legacy ``.doc`` file; if it isn't installed,
      textract raises an error explaining how to install it or pre-convert the
      file. This replaces ``antiword``, which is no longer maintained or
      packaged. See :ref:`converting-legacy-doc-files` for a batch
      pre-conversion alternative.

    - `pdftotext <https://poppler.freedesktop.org/>`_ (part of poppler) is
      *optionally* required by the ``.pdf`` parser (there is a pure python
      fallback that works if pdftotext isn't installed).

    - `ps2ascii <https://www.ghostscript.com/>`_ (part of ghostscript)
      is required by the ``.ps`` parser.

    - `tesseract-ocr <https://tesseract-ocr.github.io/tessdoc/Installation.html>`_
      is required by the ``.jpg``, ``.png`` and ``.gif`` parser.

    - `sox <https://sox.sourceforge.net/>`_
      is required by the ``.mp3`` and ``.ogg`` parser.
      You need to install ffmpeg, lame, libmad0 and libsox-fmt-mp3,
      before building sox, for these filetypes to work.

2. Add a requirements file to the `requirements directory
   <https://github.com/deanmalmgren/textract/tree/master/requirements>`_
   of the project with the lower-cased name of your operating system
   (e.g. ``requirements/windows``) so we can try to keep these things
   up to date in the future.

---

### Platform Differences (`docs/platform_differences.rst`)

Known Platform Differences
===========================

``textract`` shells out to third-party tools (Poppler, Ghostscript, sox,
tesseract) to extract text from binary formats. Those tools are installed
via different package managers per OS in CI (apt on Linux, brew on macOS,
choco on Windows), and their builds don't always produce byte-identical
output for the same input file: line-wrap width, whitespace padding, and
even punctuation glyphs (curly vs straight quotes) can vary by version.

The table below is generated from a single registry
(``tests/platform_limitations.py``) that the test suite's ``xfail``/
``skip`` markers also read from, so this list and the actual test
behavior can't silently drift apart
(enforced by ``tests/test_platform_limitations.py``).

.. list-table::
   :header-rows: 1
   :widths: 20 15 45 20

   * - Format
     - Platform
     - Known difference
     - Covered by
   * - MP3
     - Windows
     - sox.portable on Windows lacks libmad for MP3 decoding
     - ``tests/test_mp3.py::Mp3TestCase``
   * - PDF
     - Windows
     - PDF content may differ on Windows
     - ``tests/test_pdf.py::PdfTestCase``,
       ``tests/test_source_input.py::SourceInputTestCase::test_cli_stdin``
   * - PDF (tesseract OCR)
     - Linux (CI)
     - Tesseract OCR output varies by version; Linux CI has different output
     - ``tests/test_pdf.py::PdfTestCase::test_method_python/test_method_cli``
   * - PostScript (.ps)
     - Windows
     - PS text layout may differ between gswin64c txtwrite and ps2ascii
     - ``tests/test_ps.py::PsTestCase``

If your use case depends on byte-exact output across platforms, pin the
extraction to a single OS, or normalize whitespace/punctuation on your
side the way ``tests/base.py``'s ``dewrap()`` helper does for the test
suite itself.

---

### Python Package (`docs/python_package.rst`)

.. _python-package:

Python package
==============

This package is organized to make it as easy as possible to add new
extensions and support the continued growth and coverage of
textract. For almost all applications, you will just have to do
something like this::

    import textract
    text = textract.process('path/to/file.extension')

to obtain text from a document. You can also pass keyword arguments to
``textract.process``, for example, to use a particular method for
parsing a pdf like this::

    import textract
    text = textract.process('path/to/a.pdf', method='pdfminer')

or to specify a particular output encoding::

    import textract
    text = textract.process('path/to/file.extension', encoding='ascii')

By default, input encodings are inferred using `chardet
<https://github.com/chardet/chardet>`_. You can override this and specify
the input file's encoding explicitly, which is useful when chardet's
guess is wrong or its confidence is too low (e.g. for short files)::

    import textract
    text = textract.process('path/to/file.csv', input_encoding='cp1251')

The same option is available on the command line as ``--input-encoding``.
If ``input_encoding`` doesn't match the file's actual encoding, a
:class:`textract.exceptions.InvalidInputEncoding` error is raised
rather than silently producing corrupted text.

When the file name has no extension, you specify the file's extension as an argument
to ``textract.process`` like this::

    import textract
    text = textract.process('path/to/file', extension='docx')

.. _beta-in-memory-input:

Beta: in-memory and streamed input
-----------------------------------

If you already have the document's bytes in memory (e.g. an HTTP response
body) rather than a path on disk, ``process_bytes`` skips writing your own
temp file::

    import requests
    import textract

    response = requests.get('https://example.com/report.pdf')
    text = textract.process_bytes(response.content, extension='pdf')

For a readable binary stream (a socket, an open file object, ``sys.stdin.
buffer``), ``process_stream`` works the same way::

    import textract

    with open('path/to/file.docx', 'rb') as stream:
        text = textract.process_stream(stream, extension='docx')

Both require ``extension`` explicitly, since there's no filename to detect
it from; omitting it raises :class:`textract.exceptions.ExtensionRequired`.
Both also emit a ``FutureWarning`` since this API is still beta and may
change.

For csv specifically, passing ``input_encoding`` lets textract read and
decode the input lazily line-by-line instead of buffering the whole
document in memory first (this is also what powers the CLI's ``-i``/
``--input-encoding`` flag with ``-`` stdin, see
:ref:`command-line-interface`)::

    import textract

    with open('path/to/huge.csv', 'rb') as stream:
        text = textract.process_stream(stream, extension='csv', input_encoding='utf_8')

Other formats don't have a streaming implementation yet and still buffer
the whole input regardless of ``input_encoding``. See the "Next steps"
note in ``tests/test_source_input.py`` for what's left.

.. _additional-options:

Additional options
------------------

Some parsers also enable additional options which can be passed in as keyword
arguments to the ``textract.process`` function. Here is a quick table of
available options that are available to the different types of parsers:

======  =========  ===========================================================
parser  option     description
======  =========  ===========================================================
gif     language   Specify `the language`_ for OCR-ing text with tesseract
jpg     language   Specify `the language`_ for OCR-ing text with tesseract
pdf     language   For use when ``method='tesseract'``, specify `the language`_
pdf     layout     With ``method='pdftotext'`` (default), preserve the layout
png     language   Specify `the language`_ for OCR-ing text with tesseract
tiff    language   Specify `the language`_ for OCR-ing text with tesseract
======  =========  ===========================================================

As an example of using these additional options, you can extract text from a
Norwegian PDF using Tesseract OCR like this::

    text = textract.process(
        'path/to/norwegian.pdf',
        method='tesseract',
        language='nor',
    )


A look under the hood
---------------------

When ``textract.process('path/to/file.extension')`` is called,
``textract.process`` looks for a module called
``textract.parsers.extension_parser`` that also contains a ``Parser``.


.. autofunction:: textract.parsers.process

Importantly, the ``textract.parsers.extension_parser.Parser`` class
must inherit from ``textract.parsers.utils.BaseParser``.

.. autoclass:: textract.parsers.utils.BaseParser
    :members:
    :undoc-members:
    :show-inheritance:

Whatever the caller passed in (a filename, ``bytes``, or a stream) is
normalized into a single ``textract.parsers.utils.Source`` before reaching
a parser, so parsers don't have to own file I/O themselves. A parser
declares which form it needs by which of ``BaseParser``'s three
input-kind subclasses it inherits from.

.. autoclass:: textract.parsers.utils.Source
    :members:
    :undoc-members:

Many of the parsers rely on command line utilities to do some of the
parsing. For convenience, the ``textract.parsers.utils.ShellParser``
class (a kind of ``PathParser``, see below) includes some convenience
methods for streamlining access to the command line.

.. autoclass:: textract.parsers.utils.PathParser
    :members:
    :undoc-members:
    :show-inheritance:

.. autoclass:: textract.parsers.utils.ShellParser
    :members:
    :undoc-members:
    :show-inheritance:

Parsers with a native Python implementation that consume decoded text
rather than a filename (csv, eml, html, json) can instead inherit from
``textract.parsers.utils.DecodedParser``, which reads and decodes the
file (honoring ``input_encoding``) before handing the text to
:meth:`.DecodedParser.extract_from_text`.

.. autoclass:: textract.parsers.utils.DecodedParser
    :members:
    :undoc-members:
    :show-inheritance:

**Beta:** parsers whose underlying library can consume raw bytes or a
file-like object directly (e.g. docx/xlsx/pptx/epub opening a zip) can
instead inherit from ``textract.parsers.utils.BytesParser``, which never
needs a temp file.

.. autoclass:: textract.parsers.utils.BytesParser
    :members:
    :undoc-members:
    :show-inheritance:


A few specific examples
-----------------------

There are quite a few parsers included with ``textract``. Rather than
elaborating all of them, here are a few that demonstrate how parsers
work.

.. autoclass:: textract.parsers.epub_parser.Parser
    :members:
    :undoc-members:
    :show-inheritance:

.. autoclass:: textract.parsers.doc_parser.Parser
    :members:
    :undoc-members:
    :show-inheritance:


.. _the language: https://code.google.com/p/tesseract-ocr/downloads/list

---

### Tests Doc Raw Text (`tests/doc/raw_text.txt`)

I love word documents. They are lovely. They make me so happy I could smile. And that is why I wrote this package.

Sample text is hard. That is where http://hipsum.co comes in handy.

Semiotics church-key VHS, Truffaut cliche actually vegan. Cray Austin pop-up disrupt letterpress, kitsch fixie Cosby sweater cliche craft beer PBR&B. Gentrify cornhole Tonx McSweeney's, Shoreditch keffiyeh ethnic Marfa 90's kogi American Apparel. Shabby chic distillery church-key locavore beard, food truck chillwave sartorial deep v flannel authentic Tumblr narwhal kogi organic. Cred vegan jean shorts Banksy forage Neutra dreamcatcher, hashtag Bushwick polaroid pork belly flannel keytar Portland post-ironic. Cred hoodie vegan, food truck leggings Austin pour-over banjo trust fund before they sold out cray Intelligentsia plaid typewriter. Williamsburg XOXO plaid Carles Austin tofu.
Carles Tonx keffiyeh, leggings 90's lo-fi kogi viral semiotics Brooklyn biodiesel tousled bespoke kitsch. Vinyl Tonx art party Thundercats retro, viral asymmetrical artisan bicycle rights bitters master cleanse Kickstarter YOLO. Seitan street art semiotics twee skateboard, PBR&B VHS hashtag meh. Thundercats semiotics shabby chic forage single-origin coffee retro, 3 wolf moon iPhone mumblecore 90's trust fund Intelligentsia. Beard gluten-free seitan, VHS sartorial pork belly gastropub meh whatever authentic synth. Beard single-origin coffee irony fixie, before they sold out Pitchfork kitsch readymade. Helvetica butcher wayfarers, lomo artisan hashtag Brooklyn four loko fanny pack 90's mustache 8-bit.
Meh jean shorts selfies, crucifix selvage Helvetica Carles PBR Vice Banksy roof party master cleanse ugh PBR&B. Lo-fi freegan salvia photo booth, Wes Anderson skateboard Odd Future. Etsy art party Bushwick keffiyeh. Pork belly 3 wolf moon butcher mustache. YOLO raw denim lo-fi, hoodie gentrify Schlitz 8-bit sriracha Shoreditch retro brunch. Williamsburg farm-to-table beard, mlkshk Banksy fap kogi Etsy art party squid semiotics. XOXO church-key Pitchfork mlkshk irony tote bag.
Farm-to-table brunch tattooed hoodie keytar, literally selvage authentic trust fund deep v Thundercats Kickstarter narwhal locavore. Swag disrupt chambray, leggings shabby chic gastropub YOLO plaid hoodie Williamsburg Godard mixtape. Retro Godard keytar biodiesel, freegan paleo Etsy you probably haven't heard of them Pitchfork Schlitz readymade small batch cred. Pug trust fund paleo, 90's fixie typewriter next level banjo. Banksy occupy authentic master cleanse Bushwick fingerstache selfies, direct trade craft beer cliche +1 cray. Locavore four loko biodiesel Neutra chia mlkshk. Fanny pack YOLO Portland, mlkshk PBR&B single-origin coffee drinking vinegar 8-bit flannel gentrify stumptown pop-up.
Oh. You need a little dummy text for your mockup? How quaint.
I bet you are still using Bootstrap too

---

--- METRICS ---
- Files Extracted: 8
- Estimated Token Budget: ~14518 tokens
- Canonical Reference: https://codewiki.google/github.com/deanmalmgren/textract
