## File: README.md [](https://explosion.ai) # spaCy: Industrial-strength NLP spaCy is a library for **advanced Natural Language Processing** in Python and Cython. It's built on the very latest research, and was designed from day one to be used in real products. spaCy comes with [pretrained pipelines](https://spacy.io/models) and currently supports tokenization and training for **70+ languages**. It features state-of-the-art speed and **neural network models** for tagging, parsing, **named entity recognition**, **text classification** and more, multi-task learning with pretrained **transformers** like BERT, as well as a production-ready [**training system**](https://spacy.io/usage/training) and easy model packaging, deployment and workflow management. spaCy is commercial open-source software, released under the [MIT license](https://github.com/explosion/spaCy/blob/master/LICENSE). ๐Ÿ’ซ **Version 3.8 out now!** [Check out the release notes here.](https://github.com/explosion/spaCy/releases) [](https://github.com/explosion/spaCy/actions/workflows/tests.yml) [](https://github.com/explosion/spaCy/releases) [](https://pypi.org/project/spacy/) [](https://anaconda.org/conda-forge/spacy) [](https://github.com/explosion/wheelwright/releases) [](https://github.com/ambv/black) [](https://pypi.org/project/spacy/) [](https://anaconda.org/conda-forge/spacy) ## ๐Ÿ“– Documentation | Documentation | | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | โญ๏ธ **[spaCy 101]** | New to spaCy? Here's everything you need to know! | | ๐Ÿ“š **[Usage Guides]** | How to use spaCy and its features. | | ๐Ÿš€ **[New in v3.0]** | New features, backwards incompatibilities and migration guide. | | ๐Ÿช **[Project Templates]** | End-to-end workflows you can clone, modify and run. | | ๐ŸŽ› **[API Reference]** | The detailed reference for spaCy's API. | | โฉ **[GPU Processing]** | Use spaCy with CUDA-compatible GPU processing. | | ๐Ÿ“ฆ **[Models]** | Download trained pipelines for spaCy. | | ๐Ÿฆ™ **[Large Language Models]** | Integrate LLMs into spaCy pipelines. | | ๐ŸŒŒ **[Universe]** | Plugins, extensions, demos and books from the spaCy ecosystem. | | โš™๏ธ **[spaCy VS Code Extension]** | Additional tooling and features for working with spaCy's config files. | | ๐Ÿ‘ฉโ€๐Ÿซ **[Online Course]** | Learn spaCy in this free and interactive online course. | | ๐Ÿ“ฐ **[Blog]** | Read about current spaCy and Prodigy development, releases, talks and more from Explosion. | | ๐Ÿ“บ **[Videos]** | Our YouTube channel with video tutorials, talks and more. | | ๐Ÿ”ด **[Live Stream]** | Join Matt as he works on spaCy and chat about NLP. | | ๐Ÿ›  **[Changelog]** | Changes and version history. | | ๐Ÿ’ **[Contribute]** | How to contribute to the spaCy project and code base. | | ๐Ÿ‘• **[Swag]** | Support us and our work with unique, custom-designed swag! | | [](https://explosion.ai/tailored-solutions) | Custom NLP consulting, implementation and strategic advice by spaCyโ€™s core development team. Streamlined, production-ready, predictable and maintainable. Send us an email or take our 5-minute questionnaire, and well'be in touch! **[Learn more →](https://explosion.ai/tailored-solutions)** | [spacy 101]: https://spacy.io/usage/spacy-101 [new in v3.0]: https://spacy.io/usage/v3 [usage guides]: https://spacy.io/usage/ [api reference]: https://spacy.io/api/ [gpu processing]: https://spacy.io/usage#gpu [models]: https://spacy.io/models [large language models]: https://spacy.io/usage/large-language-models [universe]: https://spacy.io/universe [spacy vs code extension]: https://github.com/explosion/spacy-vscode [videos]: https://www.youtube.com/c/ExplosionAI [live stream]: https://www.youtube.com/playlist?list=PLBmcuObd5An5_iAxNYLJa_xWmNzsYce8c [online course]: https://course.spacy.io [blog]: https://explosion.ai [project templates]: https://github.com/explosion/projects [changelog]: https://spacy.io/usage#changelog [contribute]: https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md [swag]: https://explosion.ai/merch ## ๐Ÿ’ฌ Where to ask questions The spaCy project is maintained by the [spaCy team](https://explosion.ai/about). Please understand that we won't be able to provide individual support via email. We also believe that help is much more valuable if it's shared publicly, so that more people can benefit from it. | Type | Platforms | | ------------------------------- | --------------------------------------- | | ๐Ÿšจ **Bug Reports** | [GitHub Issue Tracker] | | ๐ŸŽ **Feature Requests & Ideas** | [GitHub Discussions] ยท [Live Stream] | | ๐Ÿ‘ฉโ€๐Ÿ’ป **Usage Questions** | [GitHub Discussions] ยท [Stack Overflow] | | ๐Ÿ—ฏ **General Discussion** | [GitHub Discussions] ยท [Live Stream] | [github issue tracker]: https://github.com/explosion/spaCy/issues [github discussions]: https://github.com/explosion/spaCy/discussions [stack overflow]: https://stackoverflow.com/questions/tagged/spacy [live stream]: https://www.youtube.com/playlist?list=PLBmcuObd5An5_iAxNYLJa_xWmNzsYce8c ## Features - Support for **70+ languages** - **Trained pipelines** for different languages and tasks - Multi-task learning with pretrained **transformers** like BERT - Support for pretrained **word vectors** and embeddings - State-of-the-art speed - Production-ready **training system** - Linguistically-motivated **tokenization** - Components for named **entity recognition**, part-of-speech-tagging, dependency parsing, sentence segmentation, **text classification**, lemmatization, morphological analysis, entity linking and more - Easily extensible with **custom components** and attributes - Support for custom models in **PyTorch**, **TensorFlow** and other frameworks - Built in **visualizers** for syntax and NER - Easy **model packaging**, deployment and workflow management - Robust, rigorously evaluated accuracy ๐Ÿ“– **For more details, see the [facts, figures and benchmarks](https://spacy.io/usage/facts-figures).** ## โณ Install spaCy For detailed installation instructions, see the [documentation](https://spacy.io/usage). - **Operating system**: macOS / OS X ยท Linux ยท Windows (Cygwin, MinGW, Visual Studio) - **Python version**: Python >=3.7, <3.13 (only 64 bit) - **Package managers**: [pip] ยท [conda] (via `conda-forge`) [pip]: https://pypi.org/project/spacy/ [conda]: https://anaconda.org/conda-forge/spacy ### pip Using pip, spaCy releases are available as source packages and binary wheels. Before you install spaCy and its dependencies, make sure that your `pip`, `setuptools` and `wheel` are up to date. ```bash pip install -U pip setuptools wheel pip install spacy ``` To install additional data tables for lemmatization and normalization you can run `pip install spacy[lookups]` or install [`spacy-lookups-data`](https://github.com/explosion/spacy-lookups-data) separately. The lookups package is needed to create blank models with lemmatization data, and to lemmatize in languages that don't yet come with pretrained models and aren't powered by third-party libraries. When using pip it is generally recommended to install packages in a virtual environment to avoid modifying system state: ```bash python -m venv .env source .env/bin/activate pip install -U pip setuptools wheel pip install spacy ``` ### conda You can also install spaCy from `conda` via the `conda-forge` channel. For the feedstock including the build recipe and configuration, check out [this repository](https://github.com/conda-forge/spacy-feedstock). ```bash conda install -c conda-forge spacy ``` ### Updating spaCy Some updates to spaCy may require downloading new statistical models. If you're running spaCy v2.0 or higher, you can use the `validate` command to check if your installed models are compatible and if not, print details on how to update them: ```bash pip install -U spacy python -m spacy validate ``` If you've trained your own models, keep in mind that your training and runtime inputs must match. After updating spaCy, we recommend **retraining your models** with the new version. ๐Ÿ“– **For details on upgrading from spaCy 2.x to spaCy 3.x, see the [migration guide](https://spacy.io/usage/v3#migrating).** ## ๐Ÿ“ฆ Download model packages Trained pipelines for spaCy can be installed as **Python packages**. This means that they're a component of your application, just like any other module. Models can be installed using spaCy's [`download`](https://spacy.io/api/cli#download) command, or manually by pointing pip to a path or URL. | Documentation | | | -------------------------- | ---------------------------------------------------------------- | | **[Available Pipelines]** | Detailed pipeline descriptions, accuracy figures and benchmarks. | | **[Models Documentation]** | Detailed usage and installation instructions. | | **[Training]** | How to train your own pipelines on your data. | [available pipelines]: https://spacy.io/models [models documentation]: https://spacy.io/usage/models [training]: https://spacy.io/usage/training ```bash # Download best-matching version of specific model for your spaCy installation python -m spacy download en_core_web_sm # pip install .tar.gz archive or .whl from path or URL pip install /Users/you/en_core_web_sm-3.0.0.tar.gz pip install /Users/you/en_core_web_sm-3.0.0-py3-none-any.whl pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0.tar.gz ``` ### Loading and using models To load a model, use [`spacy.load()`](https://spacy.io/api/top-level#spacy.load) with the model name or a path to the model data directory. ```python import spacy nlp = spacy.load("en_core_web_sm") doc = nlp("This is a sentence.") ``` You can also `import` a model directly via its full name and then call its `load()` method with no arguments. ```python import spacy import en_core_web_sm nlp = en_core_web_sm.load() doc = nlp("This is a sentence.") ``` ๐Ÿ“– **For more info and examples, check out the [models documentation](https://spacy.io/docs/usage/models).** ## โš’ Compile from source The other way to install spaCy is to clone its [GitHub repository](https://github.com/explosion/spaCy) and build it from source. That is the common way if you want to make changes to the code base. You'll need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, [pip](https://pip.pypa.io/en/latest/installing/), [virtualenv](https://virtualenv.pypa.io/en/latest/) and [git](https://git-scm.com) installed. The compiler part is the trickiest. How to do that depends on your system. | Platform | | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Ubuntu** | Install system-level dependencies via `apt-get`: `sudo apt-get install build-essential python-dev git` . | | **Mac** | Install a recent version of [XCode](https://developer.apple.com/xcode/), including the so-called "Command Line Tools". macOS and OS X ship with Python and git preinstalled. | | **Windows** | Install a version of the [Visual C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) or [Visual Studio Express](https://visualstudio.microsoft.com/vs/express/) that matches the version that was used to compile your Python interpreter. | For more details and instructions, see the documentation on [compiling spaCy from source](https://spacy.io/usage#source) and the [quickstart widget](https://spacy.io/usage#section-quickstart) to get the right commands for your platform and Python version. ```bash git clone https://github.com/explosion/spaCy cd spaCy python -m venv .env source .env/bin/activate # make sure you are using the latest pip python -m pip install -U pip setuptools wheel pip install -r requirements.txt pip install --no-build-isolation --editable . ``` To install with extras: ```bash pip install --no-build-isolation --editable .[lookups,cuda102] ``` ## ๐Ÿšฆ Run tests spaCy comes with an [extensive test suite](spacy/tests). In order to run the tests, you'll usually want to clone the repository and build spaCy from source. This will also install the required development dependencies and test utilities defined in the [`requirements.txt`](requirements.txt). Alternatively, you can run `pytest` on the tests from within the installed `spacy` package. Don't forget to also install the test utilities via spaCy's [`requirements.txt`](requirements.txt): ```bash pip install -r requirements.txt python -m pytest --pyargs spacy ``` --- ## File: examples/training/README.md [](https://explosion.ai) # spaCy examples See [examples/README.md](../README.md) --- ## File: examples/README.md [](https://explosion.ai) # spaCy examples For spaCy v3 we've converted many of the [v2 example scripts](https://github.com/explosion/spaCy/tree/v2.3.x/examples/) into end-to-end [spacy projects](https://spacy.io/usage/projects) workflows. The workflows include all the steps to go from data to packaged spaCy models. ## ๐Ÿช Pipeline component demos The simplest demos for training a single pipeline component are in the [`pipelines`](https://github.com/explosion/projects/blob/v3/pipelines) category including: - [`pipelines/ner_demo`](https://github.com/explosion/projects/blob/v3/pipelines/ner_demo): Train a named entity recognizer - [`pipelines/textcat_demo`](https://github.com/explosion/projects/blob/v3/pipelines/textcat_demo): Train a text classifier - [`pipelines/parser_intent_demo`](https://github.com/explosion/projects/blob/v3/pipelines/parser_intent_demo): Train a dependency parser for custom semantics ## ๐Ÿช Tutorials The [`tutorials`](https://github.com/explosion/projects/blob/v3/tutorials) category includes examples that work through specific NLP use cases end-to-end: - [`tutorials/textcat_goemotions`](https://github.com/explosion/projects/blob/v3/tutorials/textcat_goemotions): Train a text classifier to categorize emotions in Reddit posts - [`tutorials/nel_emerson`](https://github.com/explosion/projects/blob/v3/tutorials/nel_emerson): Use an entity linker to disambiguate mentions of the same name Check out the [projects documentation](https://spacy.io/usage/projects) and browse through the [available projects](https://github.com/explosion/projects/)! ## ๐Ÿš€ Get started with a demo project The [`pipelines/ner_demo`](https://github.com/explosion/projects/blob/v3/pipelines/ner_demo) project converts the spaCy v2 [`train_ner.py`](https://github.com/explosion/spaCy/blob/v2.3.x/examples/training/train_ner.py) demo script into a spaCy v3 project. 1. Clone the project: ```bash python -m spacy project clone pipelines/ner_demo ``` 2. Install requirements and download any data assets: ```bash cd ner_demo python -m pip install -r requirements.txt python -m spacy project assets ``` 3. Run the default workflow to convert, train and evaluate: ```bash python -m spacy project run all ``` Sample output: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` 4. Package the model: ```bash python -m spacy project run package ``` 5. Visualize the model's output with [Streamlit](https://streamlit.io): ```bash python -m spacy project run visualize-model ``` --- ## File: extra/example_data/textcat_example_data/README.md ## Examples of textcat training data spacy JSON training files were generated from JSONL with: ``` python textcatjsonl_to_trainjson.py -m en file.jsonl . ``` `cooking.json` is an example with mutually-exclusive classes with two labels: * `baking` * `not_baking` `jigsaw-toxic-comment.json` is an example with multiple labels per instance: * `insult` * `obscene` * `severe_toxic` * `toxic` ### Data Sources * `cooking.jsonl`: https://cooking.stackexchange.com. The meta IDs link to the original question as `https://cooking.stackexchange.com/questions/ID`, e.g., `https://cooking.stackexchange.com/questions/2` for the first instance. * `jigsaw-toxic-comment.jsonl`: [Jigsaw Toxic Comments Classification Challenge](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge) ### Data Licenses * `cooking.jsonl`: CC BY-SA 4.0 ([`CC_BY-SA-4.0.txt`](CC_BY-SA-4.0.txt)) * `jigsaw-toxic-comment.jsonl`: * text: CC BY-SA 3.0 ([`CC_BY-SA-3.0.txt`](CC_BY-SA-3.0.txt)) * annotation: CC0 ([`CC0.txt`](CC0.txt)) --- ## File: extra/example_data/ner_example_data/README.md ## Examples of NER/IOB data that can be converted with `spacy convert` To convert an IOB file to `.spacy` ([`DocBin`](https://spacy.io/api/docbin)) for spaCy v3: ```bash python -m spacy convert -c iob -s -n 10 -b en_core_web_sm file.iob . ``` See all the `spacy convert` options: https://spacy.io/api/cli#convert --- The spaCy v2 JSON training files were generated using **spaCy v2** with: ```bash python -m spacy convert -c iob -s -n 10 -b en file.iob ``` To convert an existing JSON training file to `.spacy` for spaCy v3, convert with **spaCy v3**: ```bash python -m spacy convert file.json . ``` --- ## File: extra/DEVELOPER_DOCS/README.md [](https://explosion.ai) # Developer Documentation This directory includes additional documentation and explanations of spaCy's internals. It's mostly intended for the spaCy core development team and contributors interested in the more complex parts of the library. The documents generally focus on more abstract implementation details and how specific methods and algorithms work, and they assume knowledge of what's already available in the [usage documentation](https://spacy.io/usage) and [API reference](https://spacy.io/api). If you're looking to contribute to spaCy, make sure to check out the documentation and [contributing guide](https://github.com/explosion/spaCy/blob/master/CONTRIBUTING.md) first. --- ## File: website/docs/models/index.mdx --- title: Trained Models & Pipelines teaser: Downloadable trained pipelines and weights for spaCy menu: - ['Quickstart', 'quickstart'] - ['Conventions', 'conventions'] - ['Pipeline Design', 'design'] --- {/* TODO: include interactive demo */} ### Quickstart {hidden="true"} > #### ๐Ÿ“– Installation and usage > > For more details on how to use trained pipelines with spaCy, see the > [usage guide](/usage/models). ## Package naming conventions {id="conventions"} In general, spaCy expects all pipeline packages to follow the naming convention of `[lang]_[name]`. For spaCy's pipelines, we also chose to divide the name into three components: 1. **Type:** Capabilities (e.g. `core` for general-purpose pipeline with tagging, parsing, lemmatization and named entity recognition, or `dep` for only tagging, parsing and lemmatization). 2. **Genre:** Type of text the pipeline is trained on, e.g. `web` or `news`. 3. **Size:** Package size indicator, `sm`, `md`, `lg` or `trf`. `sm` and `trf` pipelines have no static word vectors. For pipelines with default vectors, `md` has a reduced word vector table with 20k unique vectors for ~500k words and `lg` has a large word vector table with ~500k entries. For pipelines with floret vectors, `md` vector tables have 50k entries and `lg` vector tables have 200k entries. For example, [`en_core_web_sm`](/models/en#en_core_web_sm) is a small English pipeline trained on written web text (blogs, news, comments), that includes vocabulary, syntax and entities. ### Package versioning {id="model-versioning"} Additionally, the pipeline package versioning reflects both the compatibility with spaCy, as well as the model version. A package version `a.b.c` translates to: - `a`: **spaCy major version**. For example, `2` for spaCy v2.x. - `b`: **spaCy minor version**. For example, `3` for spaCy v2.3.x. - `c`: **Model version**. Different model config: e.g. from being trained on different data, with different parameters, for different numbers of iterations, with different vectors, etc. For a detailed compatibility overview, see the [`compatibility.json`](https://github.com/explosion/spacy-models/tree/master/compatibility.json). This is also the source of spaCy's internal compatibility check, performed when you run the [`download`](/api/cli#download) command. ## Trained pipeline design {id="design"} The spaCy v3 trained pipelines are designed to be efficient and configurable. For example, multiple components can share a common "token-to-vector" model and it's easy to swap out or disable the lemmatizer. The pipelines are designed to be efficient in terms of speed and size and work well when the pipeline is run in full. When modifying a trained pipeline, it's important to understand how the components **depend on** each other. Unlike spaCy v2, where the `tagger`, `parser` and `ner` components were all independent, some v3 components depend on earlier components in the pipeline. As a result, disabling or reordering components can affect the annotation quality or lead to warnings and errors. Main changes from spaCy v2 models: - The [`Tok2Vec`](/api/tok2vec) component may be a separate, shared component. A component like a tagger or parser can [listen](/api/architectures#Tok2VecListener) to an earlier `tok2vec` or `transformer` rather than having its own separate tok2vec layer. - Rule-based exceptions move from individual components to the `attribute_ruler`. Lemma and POS exceptions move from the tokenizer exceptions to the attribute ruler and the tag map and morph rules move from the tagger to the attribute ruler. - The lemmatizer tables and processing move from the vocab and tagger to a separate `lemmatizer` component. ### CNN/CPU pipeline design {id="design-cnn"} In the `sm`/`md`/`lg` models: - The `tagger`, `morphologizer` and `parser` components listen to the `tok2vec` component. If the lemmatizer is trainable (v3.3+), `lemmatizer` also listens to `tok2vec`. - The `attribute_ruler` maps `token.tag` to `token.pos` if there is no `morphologizer`. The `attribute_ruler` additionally makes sure whitespace is tagged consistently and copies `token.pos` to `token.tag` if there is no tagger. For English, the attribute ruler can improve its mapping from `token.tag` to `token.pos` if dependency parses from a `parser` are present, but the parser is not required. - The `lemmatizer` component for many languages requires `token.pos` annotation from either `tagger`+`attribute_ruler` or `morphologizer`. - The `ner` component is independent with its own internal tok2vec layer. #### CNN/CPU pipelines with floret vectors The Croatian, Finnish, Korean, Slovenian, Swedish and Ukrainian `md` and `lg` pipelines use [floret vectors](/usage/v3-2#vectors) instead of default vectors. If you're running a trained pipeline on texts and working with [`Doc`](/api/doc) objects, you shouldn't notice any difference with floret vectors. With floret vectors no tokens are out-of-vocabulary, so [`Token.is_oov`](/api/token#attributes) will return `False` for all tokens. If you access vectors directly for similarity comparisons, there are a few differences because floret vectors don't include a fixed word list like the vector keys for default vectors. - If your workflow iterates over the vector keys, you need to use an external word list instead: ```diff - lexemes = [nlp.vocab[orth] for orth in nlp.vocab.vectors] + lexemes = [nlp.vocab[word] for word in external_word_list] ``` - [`Vectors.most_similar`](/api/vectors#most_similar) is not supported because there's no fixed list of vectors to compare your vectors to. ### Transformer pipeline design {id="design-trf"} In the transformer (`trf`) pipelines, the `tagger`, `parser` and `ner` (if present) all listen to the `transformer` component. The `attribute_ruler` and `lemmatizer` have the same configuration as in the CNN models. For spaCy v3.0-v3.6, `trf` pipelines use [`spacy-transformers`](https://github.com/explosion/spacy-transformers) and the transformer output in `doc._.trf_data` is a [`TransformerData`](/api/transformer#transformerdata) object. For spaCy v3.7+, `trf` pipelines use [`spacy-curated-transformers`](https://github.com/explosion/spacy-curated-transformers) and `doc._.trf_data` is a [`DocTransformerOutput`](/api/curatedtransformer#doctransformeroutput) object. ### Modifying the default pipeline {id="design-modify"} For faster processing, you may only want to run a subset of the components in a trained pipeline. The `disable` and `exclude` arguments to [`spacy.load`](/api/top-level#spacy.load) let you control which components are loaded and run. Disabled components are loaded in the background so it's possible to reenable them in the same pipeline in the future with [`nlp.enable_pipe`](/api/language/#enable_pipe). To skip loading a component completely, use `exclude` instead of `disable`. #### Disable part-of-speech tagging and lemmatization To disable part-of-speech tagging and lemmatization, disable the `tagger`, `morphologizer`, `attribute_ruler` and `lemmatizer` components. ```python # Note: English doesn't include a morphologizer nlp = spacy.load("en_core_web_sm", disable=["tagger", "attribute_ruler", "lemmatizer"]) nlp = spacy.load("en_core_web_trf", disable=["tagger", "attribute_ruler", "lemmatizer"]) ``` The lemmatizer depends on `tagger`+`attribute_ruler` or `morphologizer` for a number of languages. If you disable any of these components, you'll see lemmatizer warnings unless the lemmatizer is also disabled. **v3.3**: Catalan, English, French, Russian and Spanish **v3.0-v3.2**: Catalan, Dutch, English, French, Greek, Italian, Macedonian, Norwegian, Polish, Russian and Spanish #### Use senter rather than parser for fast sentence segmentation If you need fast sentence segmentation without dependency parses, disable the `parser` use the `senter` component instead: ```python nlp = spacy.load("en_core_web_sm") nlp.disable_pipe("parser") nlp.enable_pipe("senter") ``` The `senter` component is ~10× faster than the parser and more accurate than the rule-based `sentencizer`. #### Switch from trainable lemmatizer to default lemmatizer Since v3.3, a number of pipelines use a trainable lemmatizer. You can check whether the lemmatizer is trainable: ```python nlp = spacy.load("de_core_web_sm") assert nlp.get_pipe("lemmatizer").is_trainable ``` If you'd like to switch to a non-trainable lemmatizer that's similar to v3.2 or earlier, you can replace the trainable lemmatizer with the default non-trainable lemmatizer: ```python # Requirements: pip install spacy-lookups-data nlp = spacy.load("de_core_web_sm") # Remove existing lemmatizer nlp.remove_pipe("lemmatizer") # Add non-trainable lemmatizer from language defaults # and load lemmatizer tables from spacy-lookups-data nlp.add_pipe("lemmatizer").initialize() ``` #### Switch from rule-based to lookup lemmatization For the Dutch, English, French, Greek, Macedonian, Norwegian and Spanish pipelines, you can swap out a trainable or rule-based lemmatizer for a lookup lemmatizer: ```python # Requirements: pip install spacy-lookups-data nlp = spacy.load("en_core_web_sm") nlp.remove_pipe("lemmatizer") nlp.add_pipe("lemmatizer", config={"mode": "lookup"}).initialize() ``` #### Disable everything except NER For the non-transformer models, the `ner` component is independent, so you can disable everything else: ```python nlp = spacy.load("en_core_web_sm", disable=["tok2vec", "tagger", "parser", "attribute_ruler", "lemmatizer"]) ``` In the transformer models, `ner` listens to the `transformer` component, so you can disable all components related tagging, parsing, and lemmatization. ```python nlp = spacy.load("en_core_web_trf", disable=["tagger", "parser", "attribute_ruler", "lemmatizer"]) ``` #### Move NER to the end of the pipeline As of v3.1, the NER component is at the end of the pipeline by default. For access to `POS` and `LEMMA` features in an `entity_ruler`, move `ner` to the end of the pipeline after `attribute_ruler` and `lemmatizer`: ```python # load without NER nlp = spacy.load("en_core_web_sm", exclude=["ner"]) # source NER from the same pipeline package as the last component nlp.add_pipe("ner", source=spacy.load("en_core_web_sm")) # insert the entity ruler nlp.add_pipe("entity_ruler", before="ner") ``` --- ## File: website/docs/api/architectures.mdx --- title: Model Architectures teaser: Pre-defined model architectures included with the core library source: spacy/ml/models menu: - ['Tok2Vec', 'tok2vec-arch'] - ['Transformers', 'transformers'] - ['Pretraining', 'pretrain'] - ['Parser & NER', 'parser'] - ['Tagging', 'tagger'] - ['Text Classification', 'textcat'] - ['Span Classification', 'spancat'] - ['Entity Linking', 'entitylinker'] - ['Coreference', 'coref-architectures'] --- A **model architecture** is a function that wires up a [`Model`](https://thinc.ai/docs/api-model) instance, which you can then use in a pipeline component or as a layer of a larger network. This page documents spaCy's built-in architectures that are used for different NLP tasks. All trainable [built-in components](/api#architecture-pipeline) expect a `model` argument defined in the config and document their the default architecture. Custom architectures can be registered using the [`@spacy.registry.architectures`](/api/top-level#registry) decorator and used as part of the [training config](/usage/training#custom-functions). Also see the usage documentation on [layers and model architectures](/usage/layers-architectures). ## Tok2Vec architectures {id="tok2vec-arch",source="spacy/ml/models/tok2vec.py"} ### spacy.Tok2Vec.v2 {id="Tok2Vec"} > #### Example config > > ```ini > [model] > @architectures = "spacy.Tok2Vec.v2" > > [model.embed] > @architectures = "spacy.CharacterEmbed.v2" > # ... > > [model.encode] > @architectures = "spacy.MaxoutWindowEncoder.v2" > # ... > ``` Construct a tok2vec model out of two subnetworks: one for embedding and one for encoding. See the ["Embed, Encode, Attend, Predict"](https://explosion.ai/blog/deep-learning-formula-nlp) blog post for background. | Name | Description | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `embed` | Embed tokens into context-independent word vector representations. For example, [CharacterEmbed](/api/architectures#CharacterEmbed) or [MultiHashEmbed](/api/architectures#MultiHashEmbed). ~~Model[List[Doc], List[Floats2d]]~~ | | `encode` | Encode context into the embeddings, using an architecture such as a CNN, BiLSTM or transformer. For example, [MaxoutWindowEncoder](/api/architectures#MaxoutWindowEncoder). ~~Model[List[Floats2d], List[Floats2d]]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | ### spacy.HashEmbedCNN.v2 {id="HashEmbedCNN"} > #### Example Config > > ```ini > [model] > @architectures = "spacy.HashEmbedCNN.v2" > pretrained_vectors = null > width = 96 > depth = 4 > embed_size = 2000 > window_size = 1 > maxout_pieces = 3 > subword_features = true > ``` Build spaCy's "standard" tok2vec layer. This layer is defined by a [MultiHashEmbed](/api/architectures#MultiHashEmbed) embedding layer that uses subword features, and a [MaxoutWindowEncoder](/api/architectures#MaxoutWindowEncoder) encoding layer consisting of a CNN and a layer-normalized maxout activation function. | Name | Description | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `width` | The width of the input and output. These are required to be the same, so that residual connections can be used. Recommended values are `96`, `128` or `300`. ~~int~~ | | `depth` | The number of convolutional layers to use. Recommended values are between `2` and `8`. ~~int~~ | | `embed_size` | The number of rows in the hash embedding tables. This can be surprisingly small, due to the use of the hash embeddings. Recommended values are between `2000` and `10000`. ~~int~~ | | `window_size` | The number of tokens on either side to concatenate during the convolutions. The receptive field of the CNN will be `depth * window_size * 2 + 1`, so a 4-layer network with a window size of `2` will be sensitive to 17 words at a time. Recommended value is `1`. ~~int~~ | | `maxout_pieces` | The number of pieces to use in the maxout non-linearity. If `1`, the [`Mish`](https://thinc.ai/docs/api-layers#mish) non-linearity is used instead. Recommended values are `1`-`3`. ~~int~~ | | `subword_features` | Whether to also embed subword features, specifically the prefix, suffix and word shape. This is recommended for alphabetic languages like English, but not if single-character tokens are used for a language such as Chinese. ~~bool~~ | | `pretrained_vectors` | Whether to also use static vectors. ~~bool~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | ### spacy.Tok2VecListener.v1 {id="Tok2VecListener"} > #### Example config > > ```ini > [components.tok2vec] > factory = "tok2vec" > > [components.tok2vec.model] > @architectures = "spacy.HashEmbedCNN.v2" > width = 342 > > [components.tagger] > factory = "tagger" > > [components.tagger.model] > @architectures = "spacy.Tagger.v2" > > [components.tagger.model.tok2vec] > @architectures = "spacy.Tok2VecListener.v1" > width = ${components.tok2vec.model.width} > ``` A listener is used as a sublayer within a component such as a [`DependencyParser`](/api/dependencyparser), [`EntityRecognizer`](/api/entityrecognizer)or [`TextCategorizer`](/api/textcategorizer). Usually you'll have multiple listeners connecting to a single upstream [`Tok2Vec`](/api/tok2vec) component that's earlier in the pipeline. The listener layers act as **proxies**, passing the predictions from the `Tok2Vec` component into downstream components, and communicating gradients back upstream. Instead of defining its own `Tok2Vec` instance, a model architecture like [Tagger](/api/architectures#tagger) can define a listener as its `tok2vec` argument that connects to the shared `tok2vec` component in the pipeline. Listeners work by caching the `Tok2Vec` output for a given batch of `Doc`s. This means that in order for a component to work with the listener, the batch of `Doc`s passed to the listener must be the same as the batch of `Doc`s passed to the `Tok2Vec`. As a result, any manipulation of the `Doc`s which would affect `Tok2Vec` output, such as to create special contexts or remove `Doc`s for which no prediction can be made, must happen inside the model, **after** the call to the `Tok2Vec` component. | Name | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `width` | The width of the vectors produced by the "upstream" [`Tok2Vec`](/api/tok2vec) component. ~~int~~ | | `upstream` | A string to identify the "upstream" `Tok2Vec` component to communicate with. By default, the upstream name is the wildcard string `"*"`, but you could also specify the name of the `Tok2Vec` component. You'll almost never have multiple upstream `Tok2Vec` components, so the wildcard string will almost always be fine. ~~str~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | ### spacy.MultiHashEmbed.v2 {id="MultiHashEmbed"} > #### Example config > > ```ini > [model] > @architectures = "spacy.MultiHashEmbed.v2" > width = 64 > attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"] > rows = [2000, 1000, 1000, 1000] > include_static_vectors = true > ``` Construct an embedding layer that separately embeds a number of lexical attributes using hash embedding, concatenates the results, and passes it through a feed-forward subnetwork to build a mixed representation. The features used can be configured with the `attrs` argument. The suggested attributes are `NORM`, `PREFIX`, `SUFFIX` and `SHAPE`. This lets the model take into account some subword information, without construction a fully character-based representation. If pretrained vectors are available, they can be included in the representation as well, with the vectors table kept static (i.e. it's not updated). | Name | Description | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `width` | The output width. Also used as the width of the embedding tables. Recommended values are between `64` and `300`. If static vectors are included, a learned linear layer is used to map the vectors to the specified width before concatenating it with the other embedding outputs. A single maxout layer is then used to reduce the concatenated vectors to the final width. ~~int~~ | | `attrs` | The token attributes to embed. A separate embedding table will be constructed for each attribute. ~~List[Union[int, str]]~~ | | `rows` | The number of rows for each embedding tables. Can be low, due to the hashing trick. Recommended values are between `1000` and `10000`. The layer needs surprisingly few rows, due to its use of the hashing trick. Generally between 2000 and 10000 rows is sufficient, even for very large vocabularies. A number of rows must be specified for each table, so the `rows` list must be of the same length as the `attrs` parameter. ~~List[int]~~ | | `include_static_vectors` | Whether to also use static word vectors. Requires a vectors table to be loaded in the [`Doc`](/api/doc) objects' vocab. ~~bool~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | ### spacy.CharacterEmbed.v2 {id="CharacterEmbed"} > #### Example config > > ```ini > [model] > @architectures = "spacy.CharacterEmbed.v2" > width = 128 > rows = 7000 > nM = 64 > nC = 8 > ``` Construct an embedded representation based on character embeddings, using a feed-forward network. A fixed number of UTF-8 byte characters are used for each word, taken from the beginning and end of the word equally. Padding is used in the center for words that are too short. For instance, let's say `nC=4`, and the word is "jumping". The characters used will be `"jung"` (two from the start, two from the end). If we had `nC=8`, the characters would be `"jumpping"`: 4 from the start, 4 from the end. This ensures that the final character is always in the last position, instead of being in an arbitrary position depending on the word length. The characters are embedded in a embedding table with a given number of rows, and the vectors concatenated. A hash-embedded vector of the `NORM` of the word is also concatenated on, and the result is then passed through a feed-forward network to construct a single vector to represent the information. | Name | Description | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `width` | The width of the output vector and the `NORM` hash embedding. ~~int~~ | | `rows` | The number of rows in the `NORM` hash embedding table. ~~int~~ | | `nM` | The dimensionality of the character embeddings. Recommended values are between `16` and `64`. ~~int~~ | | `nC` | The number of UTF-8 bytes to embed per word. Recommended values are between `3` and `8`, although it may depend on the length of words in the language. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | ### spacy.MaxoutWindowEncoder.v2 {id="MaxoutWindowEncoder"} > #### Example config > > ```ini > [model] > @architectures = "spacy.MaxoutWindowEncoder.v2" > width = 128 > window_size = 1 > maxout_pieces = 3 > depth = 4 > ``` Encode context using convolutions with maxout activation, layer normalization and residual connections. | Name | Description | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `width` | The input and output width. These are required to be the same, to allow residual connections. This value will be determined by the width of the inputs. Recommended values are between `64` and `300`. ~~int~~ | | `window_size` | The number of words to concatenate around each token to construct the convolution. Recommended value is `1`. ~~int~~ | | `maxout_pieces` | The number of maxout pieces to use. Recommended values are `2` or `3`. ~~int~~ | | `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ | ### spacy.MishWindowEncoder.v2 {id="MishWindowEncoder"} > #### Example config > > ```ini > [model] > @architectures = "spacy.MishWindowEncoder.v2" > width = 64 > window_size = 1 > depth = 4 > ``` Encode context using convolutions with [`Mish`](https://thinc.ai/docs/api-layers#mish) activation, layer normalization and residual connections. | Name | Description | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `width` | The input and output width. These are required to be the same, to allow residual connections. This value will be determined by the width of the inputs. Recommended values are between `64` and `300`. ~~int~~ | | `window_size` | The number of words to concatenate around each token to construct the convolution. Recommended value is `1`. ~~int~~ | | `depth` | The number of convolutional layers. Recommended value is `4`. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ | ### spacy.TorchBiLSTMEncoder.v1 {id="TorchBiLSTMEncoder"} > #### Example config > > ```ini > [model] > @architectures = "spacy.TorchBiLSTMEncoder.v1" > width = 64 > depth = 2 > dropout = 0.0 > ``` Encode context using bidirectional LSTM layers. Requires [PyTorch](https://pytorch.org). | Name | Description | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `width` | The input and output width. These are required to be the same, to allow residual connections. This value will be determined by the width of the inputs. Recommended values are between `64` and `300`. ~~int~~ | | `depth` | The number of recurrent layers, for instance `depth=2` results in stacking two LSTMs together. ~~int~~ | | `dropout` | Creates a Dropout layer on the outputs of each LSTM layer except the last layer. Set to 0.0 to disable this functionality. ~~float~~ | | **CREATES** | The model using the architecture. ~~Model[List[Floats2d], List[Floats2d]]~~ | ### spacy.StaticVectors.v2 {id="StaticVectors"} > #### Example config > > ```ini > [model] > @architectures = "spacy.StaticVectors.v2" > nO = null > nM = null > dropout = 0.2 > key_attr = "ORTH" > > [model.init_W] > @initializers = "glorot_uniform_init.v1" > ``` Embed [`Doc`](/api/doc) objects with their vocab's vectors table, applying a learned linear projection to control the dimensionality. Unknown tokens are mapped to a zero vector. See the documentation on [static vectors](/usage/embeddings-transformers#static-vectors) for details. | Name | Description | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `nO` | The output width of the layer, after the linear projection. ~~Optional[int]~~ | | `nM` | The width of the static vectors. ~~Optional[int]~~ | | `dropout` | Optional dropout rate. If set, it's applied per dimension over the whole batch. Defaults to `None`. ~~Optional[float]~~ | | `init_W` | The [initialization function](https://thinc.ai/docs/api-initializers). Defaults to [`glorot_uniform_init`](https://thinc.ai/docs/api-initializers#glorot_uniform_init). ~~Callable[[Ops, Tuple[int, ...]]], FloatsXd]~~ | | `key_attr` | This setting is ignored in spaCy v3.6+. To set a custom key attribute for vectors, configure it through [`Vectors`](/api/vectors) or [`spacy init vectors`](/api/cli#init-vectors). Defaults to `"ORTH"`. ~~str~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Ragged]~~ | ### spacy.FeatureExtractor.v1 {id="FeatureExtractor"} > #### Example config > > ```ini > [model] > @architectures = "spacy.FeatureExtractor.v1" > columns = ["NORM", "PREFIX", "SUFFIX", "SHAPE", "ORTH"] > ``` Extract arrays of input features from [`Doc`](/api/doc) objects. Expects a list of feature names to extract, which should refer to token attributes. | Name | Description | | ----------- | ------------------------------------------------------------------------ | | `columns` | The token attributes to extract. ~~List[Union[int, str]]~~ | | **CREATES** | The created feature extraction layer. ~~Model[List[Doc], List[Ints2d]]~~ | ## Transformer architectures {id="transformers",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/architectures.py"} The following architectures are provided by the package [`spacy-transformers`](https://github.com/explosion/spacy-transformers). See the [usage documentation](/usage/embeddings-transformers#transformers) for how to integrate the architectures into your training config. Note that in order to use these architectures in your config, you need to install the [`spacy-transformers`](https://github.com/explosion/spacy-transformers). See the [installation docs](/usage/embeddings-transformers#transformers-installation) for details and system requirements. ### spacy-transformers.TransformerModel.v3 {id="TransformerModel"} > #### Example Config > > ```ini > [model] > @architectures = "spacy-transformers.TransformerModel.v3" > name = "roberta-base" > tokenizer_config = {"use_fast": true} > transformer_config = {} > mixed_precision = true > grad_scaler_config = {"init_scale": 32768} > > [model.get_spans] > @span_getters = "spacy-transformers.strided_spans.v1" > window = 128 > stride = 96 > ``` Load and wrap a transformer model from the [HuggingFace `transformers`](https://huggingface.co/transformers) library. You can use any transformer that has pretrained weights and a PyTorch implementation. The `name` variable is passed through to the underlying library, so it can be either a string or a path. If it's a string, the pretrained weights will be downloaded via the transformers library if they are not already available locally. In order to support longer documents, the [TransformerModel](/api/architectures#TransformerModel) layer allows you to pass in a `get_spans` function that will divide up the [`Doc`](/api/doc) objects before passing them through the transformer. Your spans are allowed to overlap or exclude tokens. This layer is usually used directly by the [`Transformer`](/api/transformer) component, which allows you to share the transformer weights across your pipeline. For a layer that's configured for use in other components, see [Tok2VecTransformer](/api/architectures#Tok2VecTransformer). | Name | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Any model name that can be loaded by [`transformers.AutoModel`](https://huggingface.co/transformers/model_doc/auto.html#transformers.AutoModel). ~~str~~ | | `get_spans` | Function that takes a batch of [`Doc`](/api/doc) object and returns lists of [`Span`](/api) objects to process by the transformer. [See here](/api/transformer#span_getters) for built-in options and examples. ~~Callable[[List[Doc]], List[Span]]~~ | | `tokenizer_config` | Tokenizer settings passed to [`transformers.AutoTokenizer`](https://huggingface.co/transformers/model_doc/auto.html#transformers.AutoTokenizer). ~~Dict[str, Any]~~ | | `transformer_config` | Transformer settings passed to [`transformers.AutoConfig`](https://huggingface.co/transformers/model_doc/auto.html?highlight=autoconfig#transformers.AutoConfig) ~~Dict[str, Any]~~ | | `mixed_precision` | Replace whitelisted ops by half-precision counterparts. Speeds up training and prediction on GPUs with [Tensor Cores](https://developer.nvidia.com/tensor-cores) and reduces GPU memory use. ~~bool~~ | | `grad_scaler_config` | Configuration to pass to `thinc.api.PyTorchGradScaler` during training when `mixed_precision` is enabled. ~~Dict[str, Any]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], FullTransformerBatch]~~ | | | | Mixed-precision support is currently an experimental feature. - The `transformer_config` argument was added in `spacy-transformers.TransformerModel.v2`. - The `mixed_precision` and `grad_scaler_config` arguments were added in `spacy-transformers.TransformerModel.v3`. The other arguments are shared between all versions. ### spacy-transformers.TransformerListener.v1 {id="TransformerListener"} > #### Example Config > > ```ini > [model] > @architectures = "spacy-transformers.TransformerListener.v1" > grad_factor = 1.0 > > [model.pooling] > @layers = "reduce_mean.v1" > ``` Create a `TransformerListener` layer, which will connect to a [`Transformer`](/api/transformer) component earlier in the pipeline. The layer takes a list of [`Doc`](/api/doc) objects as input, and produces a list of 2-dimensional arrays as output, with each array having one row per token. Most spaCy models expect a sublayer with this signature, making it easy to connect them to a transformer model via this sublayer. Transformer models usually operate over wordpieces, which usually don't align one-to-one against spaCy tokens. The layer therefore requires a reduction operation in order to calculate a single token vector given zero or more wordpiece vectors. | Name | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `pooling` | A reduction layer used to calculate the token vectors based on zero or more wordpiece vectors. If in doubt, mean pooling (see [`reduce_mean`](https://thinc.ai/docs/api-layers#reduce_mean)) is usually a good choice. ~~Model[Ragged, Floats2d]~~ | | `grad_factor` | Reweight gradients from the component before passing them upstream. You can set this to `0` to "freeze" the transformer weights with respect to the component, or use it to make some components more significant than others. Leaving it at `1.0` is usually fine. ~~float~~ | | `upstream` | A string to identify the "upstream" `Transformer` component to communicate with. By default, the upstream name is the wildcard string `"*"`, but you could also specify the name of the `Transformer` component. You'll almost never have multiple upstream `Transformer` components, so the wildcard string will almost always be fine. ~~str~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | ### spacy-transformers.Tok2VecTransformer.v3 {id="Tok2VecTransformer"} > #### Example Config > > ```ini > [model] > @architectures = "spacy-transformers.Tok2VecTransformer.v3" > name = "albert-base-v2" > tokenizer_config = {"use_fast": false} > transformer_config = {} > grad_factor = 1.0 > mixed_precision = true > grad_scaler_config = {"init_scale": 32768} > ``` Use a transformer as a [`Tok2Vec`](/api/tok2vec) layer directly. This does **not** allow multiple components to share the transformer weights and does **not** allow the transformer to set annotations into the [`Doc`](/api/doc) object, but it's a **simpler solution** if you only need the transformer within one component. | Name | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `get_spans` | Function that takes a batch of [`Doc`](/api/doc) object and returns lists of [`Span`](/api) objects to process by the transformer. [See here](/api/transformer#span_getters) for built-in options and examples. ~~Callable[[List[Doc]], List[Span]]~~ | | `tokenizer_config` | Tokenizer settings passed to [`transformers.AutoTokenizer`](https://huggingface.co/transformers/model_doc/auto.html#transformers.AutoTokenizer). ~~Dict[str, Any]~~ | | `transformer_config` | Settings to pass to the transformers forward pass. ~~Dict[str, Any]~~ | | `pooling` | A reduction layer used to calculate the token vectors based on zero or more wordpiece vectors. If in doubt, mean pooling (see [`reduce_mean`](https://thinc.ai/docs/api-layers#reduce_mean)) is usually a good choice. ~~Model[Ragged, Floats2d]~~ | | `grad_factor` | Reweight gradients from the component before passing them upstream. You can set this to `0` to "freeze" the transformer weights with respect to the component, or use it to make some components more significant than others. Leaving it at `1.0` is usually fine. ~~float~~ | | `mixed_precision` | Replace whitelisted ops by half-precision counterparts. Speeds up training and prediction on GPUs with [Tensor Cores](https://developer.nvidia.com/tensor-cores) and reduces GPU memory use. ~~bool~~ | | `grad_scaler_config` | Configuration to pass to `thinc.api.PyTorchGradScaler` during training when `mixed_precision` is enabled. ~~Dict[str, Any]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | Mixed-precision support is currently an experimental feature. - The `transformer_config` argument was added in `spacy-transformers.Tok2VecTransformer.v2`. - The `mixed_precision` and `grad_scaler_config` arguments were added in `spacy-transformers.Tok2VecTransformer.v3`. The other arguments are shared between all versions. ## Curated Transformer architectures {id="curated-trf",source="https://github.com/explosion/spacy-curated-transformers/blob/main/spacy_curated_transformers/models/architectures.py"} The following architectures are provided by the package [`spacy-curated-transformers`](https://github.com/explosion/spacy-curated-transformers). See the [usage documentation](/usage/embeddings-transformers#transformers) for how to integrate the architectures into your training config. When loading the model [from the Hugging Face Hub](/api/curatedtransformer#hf_trfencoder_loader), the model config's parameters must be same as the hyperparameters used by the pre-trained model. The [`init fill-curated-transformer`](/api/cli#init-fill-curated-transformer) CLI command can be used to automatically fill in these values. ### spacy-curated-transformers.AlbertTransformer.v1 Construct an ALBERT transformer model. | Name | Description | | ------------------------------ | ---------------------------------------------------------------------------------------- | | `vocab_size` | Vocabulary size. ~~int~~ | | `with_spans` | Callback that constructs a span generator model. ~~Callable~~ | | `piece_encoder` | The piece encoder to segment input tokens. ~~Model~~ | | `attention_probs_dropout_prob` | Dropout probability of the self-attention layers. ~~float~~ | | `embedding_width` | Width of the embedding representations. ~~int~~ | | `hidden_act` | Activation used by the point-wise feed-forward layers. ~~str~~ | | `hidden_dropout_prob` | Dropout probability of the point-wise feed-forward and embedding layers. ~~float~~ | | `hidden_width` | Width of the final representations. ~~int~~ | | `intermediate_width` | Width of the intermediate projection layer in the point-wise feed-forward layer. ~~int~~ | | `layer_norm_eps` | Epsilon for layer normalization. ~~float~~ | | `max_position_embeddings` | Maximum length of position embeddings. ~~int~~ | | `model_max_length` | Maximum length of model inputs. ~~int~~ | | `num_attention_heads` | Number of self-attention heads. ~~int~~ | | `num_hidden_groups` | Number of layer groups whose constituents share parameters. ~~int~~ | | `num_hidden_layers` | Number of hidden layers. ~~int~~ | | `padding_idx` | Index of the padding meta-token. ~~int~~ | | `type_vocab_size` | Type vocabulary size. ~~int~~ | | `mixed_precision` | Use mixed-precision training. ~~bool~~ | | `grad_scaler_config` | Configuration passed to the PyTorch gradient scaler. ~~dict~~ | | **CREATES** | The model using the architecture ~~Model~~ | ### spacy-curated-transformers.BertTransformer.v1 Construct a BERT transformer model. | Name | Description | | ------------------------------ | ---------------------------------------------------------------------------------------- | | `vocab_size` | Vocabulary size. ~~int~~ | | `with_spans` | Callback that constructs a span generator model. ~~Callable~~ | | `piece_encoder` | The piece encoder to segment input tokens. ~~Model~~ | | `attention_probs_dropout_prob` | Dropout probability of the self-attention layers. ~~float~~ | | `hidden_act` | Activation used by the point-wise feed-forward layers. ~~str~~ | | `hidden_dropout_prob` | Dropout probability of the point-wise feed-forward and embedding layers. ~~float~~ | | `hidden_width` | Width of the final representations. ~~int~~ | | `intermediate_width` | Width of the intermediate projection layer in the point-wise feed-forward layer. ~~int~~ | | `layer_norm_eps` | Epsilon for layer normalization. ~~float~~ | | `max_position_embeddings` | Maximum length of position embeddings. ~~int~~ | | `model_max_length` | Maximum length of model inputs. ~~int~~ | | `num_attention_heads` | Number of self-attention heads. ~~int~~ | | `num_hidden_layers` | Number of hidden layers. ~~int~~ | | `padding_idx` | Index of the padding meta-token. ~~int~~ | | `type_vocab_size` | Type vocabulary size. ~~int~~ | | `mixed_precision` | Use mixed-precision training. ~~bool~~ | | `grad_scaler_config` | Configuration passed to the PyTorch gradient scaler. ~~dict~~ | | **CREATES** | The model using the architecture ~~Model~~ | ### spacy-curated-transformers.CamembertTransformer.v1 Construct a CamemBERT transformer model. | Name | Description | | ------------------------------ | ---------------------------------------------------------------------------------------- | | `vocab_size` | Vocabulary size. ~~int~~ | | `with_spans` | Callback that constructs a span generator model. ~~Callable~~ | | `piece_encoder` | The piece encoder to segment input tokens. ~~Model~~ | | `attention_probs_dropout_prob` | Dropout probability of the self-attention layers. ~~float~~ | | `hidden_act` | Activation used by the point-wise feed-forward layers. ~~str~~ | | `hidden_dropout_prob` | Dropout probability of the point-wise feed-forward and embedding layers. ~~float~~ | | `hidden_width` | Width of the final representations. ~~int~~ | | `intermediate_width` | Width of the intermediate projection layer in the point-wise feed-forward layer. ~~int~~ | | `layer_norm_eps` | Epsilon for layer normalization. ~~float~~ | | `max_position_embeddings` | Maximum length of position embeddings. ~~int~~ | | `model_max_length` | Maximum length of model inputs. ~~int~~ | | `num_attention_heads` | Number of self-attention heads. ~~int~~ | | `num_hidden_layers` | Number of hidden layers. ~~int~~ | | `padding_idx` | Index of the padding meta-token. ~~int~~ | | `type_vocab_size` | Type vocabulary size. ~~int~~ | | `mixed_precision` | Use mixed-precision training. ~~bool~~ | | `grad_scaler_config` | Configuration passed to the PyTorch gradient scaler. ~~dict~~ | | **CREATES** | The model using the architecture ~~Model~~ | ### spacy-curated-transformers.RobertaTransformer.v1 Construct a RoBERTa transformer model. | Name | Description | | ------------------------------ | ---------------------------------------------------------------------------------------- | | `vocab_size` | Vocabulary size. ~~int~~ | | `with_spans` | Callback that constructs a span generator model. ~~Callable~~ | | `piece_encoder` | The piece encoder to segment input tokens. ~~Model~~ | | `attention_probs_dropout_prob` | Dropout probability of the self-attention layers. ~~float~~ | | `hidden_act` | Activation used by the point-wise feed-forward layers. ~~str~~ | | `hidden_dropout_prob` | Dropout probability of the point-wise feed-forward and embedding layers. ~~float~~ | | `hidden_width` | Width of the final representations. ~~int~~ | | `intermediate_width` | Width of the intermediate projection layer in the point-wise feed-forward layer. ~~int~~ | | `layer_norm_eps` | Epsilon for layer normalization. ~~float~~ | | `max_position_embeddings` | Maximum length of position embeddings. ~~int~~ | | `model_max_length` | Maximum length of model inputs. ~~int~~ | | `num_attention_heads` | Number of self-attention heads. ~~int~~ | | `num_hidden_layers` | Number of hidden layers. ~~int~~ | | `padding_idx` | Index of the padding meta-token. ~~int~~ | | `type_vocab_size` | Type vocabulary size. ~~int~~ | | `mixed_precision` | Use mixed-precision training. ~~bool~~ | | `grad_scaler_config` | Configuration passed to the PyTorch gradient scaler. ~~dict~~ | | **CREATES** | The model using the architecture ~~Model~~ | ### spacy-curated-transformers.XlmrTransformer.v1 Construct a XLM-RoBERTa transformer model. | Name | Description | | ------------------------------ | ---------------------------------------------------------------------------------------- | | `vocab_size` | Vocabulary size. ~~int~~ | | `with_spans` | Callback that constructs a span generator model. ~~Callable~~ | | `piece_encoder` | The piece encoder to segment input tokens. ~~Model~~ | | `attention_probs_dropout_prob` | Dropout probability of the self-attention layers. ~~float~~ | | `hidden_act` | Activation used by the point-wise feed-forward layers. ~~str~~ | | `hidden_dropout_prob` | Dropout probability of the point-wise feed-forward and embedding layers. ~~float~~ | | `hidden_width` | Width of the final representations. ~~int~~ | | `intermediate_width` | Width of the intermediate projection layer in the point-wise feed-forward layer. ~~int~~ | | `layer_norm_eps` | Epsilon for layer normalization. ~~float~~ | | `max_position_embeddings` | Maximum length of position embeddings. ~~int~~ | | `model_max_length` | Maximum length of model inputs. ~~int~~ | | `num_attention_heads` | Number of self-attention heads. ~~int~~ | | `num_hidden_layers` | Number of hidden layers. ~~int~~ | | `padding_idx` | Index of the padding meta-token. ~~int~~ | | `type_vocab_size` | Type vocabulary size. ~~int~~ | | `mixed_precision` | Use mixed-precision training. ~~bool~~ | | `grad_scaler_config` | Configuration passed to the PyTorch gradient scaler. ~~dict~~ | | **CREATES** | The model using the architecture ~~Model~~ | ### spacy-curated-transformers.ScalarWeight.v1 Construct a model that accepts a list of transformer layer outputs and returns a weighted representation of the same. | Name | Description | | -------------------- | ----------------------------------------------------------------------------- | | `num_layers` | Number of transformer hidden layers. ~~int~~ | | `dropout_prob` | Dropout probability. ~~float~~ | | `mixed_precision` | Use mixed-precision training. ~~bool~~ | | `grad_scaler_config` | Configuration passed to the PyTorch gradient scaler. ~~dict~~ | | **CREATES** | The model using the architecture ~~Model[ScalarWeightInT, ScalarWeightOutT]~~ | ### spacy-curated-transformers.TransformerLayersListener.v1 Construct a listener layer that communicates with one or more upstream Transformer components. This layer extracts the output of the last transformer layer and performs pooling over the individual pieces of each `Doc` token, returning their corresponding representations. The upstream name should either be the wildcard string '\*', or the name of the Transformer component. In almost all cases, the wildcard string will suffice as there'll only be one upstream Transformer component. But in certain situations, e.g: you have disjoint datasets for certain tasks, or you'd like to use a pre-trained pipeline but a downstream task requires its own token representations, you could end up with more than one Transformer component in the pipeline. | Name | Description | | --------------- | ---------------------------------------------------------------------------------------------------------------------- | | `layers` | The number of layers produced by the upstream transformer component, excluding the embedding layer. ~~int~~ | | `width` | The width of the vectors produced by the upstream transformer component. ~~int~~ | | `pooling` | Model that is used to perform pooling over the piece representations. ~~Model~~ | | `upstream_name` | A string to identify the 'upstream' Transformer component to communicate with. ~~str~~ | | `grad_factor` | Factor to multiply gradients with. ~~float~~ | | **CREATES** | A model that returns the relevant vectors from an upstream transformer component. ~~Model[List[Doc], List[Floats2d]]~~ | ### spacy-curated-transformers.LastTransformerLayerListener.v1 Construct a listener layer that communicates with one or more upstream Transformer components. This layer extracts the output of the last transformer layer and performs pooling over the individual pieces of each Doc token, returning their corresponding representations. The upstream name should either be the wildcard string '\*', or the name of the Transformer component. In almost all cases, the wildcard string will suffice as there'll only be one upstream Transformer component. But in certain situations, e.g: you have disjoint datasets for certain tasks, or you'd like to use a pre-trained pipeline but a downstream task requires its own token representations, you could end up with more than one Transformer component in the pipeline. | Name | Description | | --------------- | ---------------------------------------------------------------------------------------------------------------------- | | `width` | The width of the vectors produced by the upstream transformer component. ~~int~~ | | `pooling` | Model that is used to perform pooling over the piece representations. ~~Model~~ | | `upstream_name` | A string to identify the 'upstream' Transformer component to communicate with. ~~str~~ | | `grad_factor` | Factor to multiply gradients with. ~~float~~ | | **CREATES** | A model that returns the relevant vectors from an upstream transformer component. ~~Model[List[Doc], List[Floats2d]]~~ | ### spacy-curated-transformers.ScalarWeightingListener.v1 Construct a listener layer that communicates with one or more upstream Transformer components. This layer calculates a weighted representation of all transformer layer outputs and performs pooling over the individual pieces of each Doc token, returning their corresponding representations. Requires its upstream Transformer components to return all layer outputs from their models. The upstream name should either be the wildcard string '\*', or the name of the Transformer component. In almost all cases, the wildcard string will suffice as there'll only be one upstream Transformer component. But in certain situations, e.g: you have disjoint datasets for certain tasks, or you'd like to use a pre-trained pipeline but a downstream task requires its own token representations, you could end up with more than one Transformer component in the pipeline. | Name | Description | | --------------- | ---------------------------------------------------------------------------------------------------------------------- | | `width` | The width of the vectors produced by the upstream transformer component. ~~int~~ | | `weighting` | Model that is used to perform the weighting of the different layer outputs. ~~Model~~ | | `pooling` | Model that is used to perform pooling over the piece representations. ~~Model~~ | | `upstream_name` | A string to identify the 'upstream' Transformer component to communicate with. ~~str~~ | | `grad_factor` | Factor to multiply gradients with. ~~float~~ | | **CREATES** | A model that returns the relevant vectors from an upstream transformer component. ~~Model[List[Doc], List[Floats2d]]~~ | ### spacy-curated-transformers.BertWordpieceEncoder.v1 Construct a WordPiece piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers. This encoder also splits each token on punctuation characters, as expected by most BERT models. This model must be separately initialized using an appropriate loader. ### spacy-curated-transformers.ByteBpeEncoder.v1 Construct a Byte-BPE piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers. This model must be separately initialized using an appropriate loader. ### spacy-curated-transformers.CamembertSentencepieceEncoder.v1 Construct a SentencePiece piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers with CamemBERT post-processing applied. This model must be separately initialized using an appropriate loader. ### spacy-curated-transformers.CharEncoder.v1 Construct a character piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers. This model must be separately initialized using an appropriate loader. ### spacy-curated-transformers.SentencepieceEncoder.v1 Construct a SentencePiece piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers. This model must be separately initialized using an appropriate loader. ### spacy-curated-transformers.WordpieceEncoder.v1 Construct a WordPiece piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers. This encoder also splits each token on punctuation characters, as expected by most BERT models. This model must be separately initialized using an appropriate loader. ### spacy-curated-transformers.XlmrSentencepieceEncoder.v1 Construct a SentencePiece piece encoder model that accepts a list of token sequences or documents and returns a corresponding list of piece identifiers with XLM-RoBERTa post-processing applied. This model must be separately initialized using an appropriate loader. ## Pretraining architectures {id="pretrain",source="spacy/ml/models/multi_task.py"} The spacy `pretrain` command lets you initialize a `Tok2Vec` layer in your pipeline with information from raw text. To this end, additional layers are added to build a network for a temporary task that forces the `Tok2Vec` layer to learn something about sentence structure and word cooccurrence statistics. Two pretraining objectives are available, both of which are variants of the cloze task [Devlin et al. (2018)](https://arxiv.org/abs/1810.04805) introduced for BERT. For more information, see the section on [pretraining](/usage/embeddings-transformers#pretraining). ### spacy.PretrainVectors.v1 {id="pretrain_vectors"} > #### Example config > > ```ini > [pretraining] > component = "tok2vec" > > [initialize] > vectors = "en_core_web_lg" > ... > > [pretraining.objective] > @architectures = "spacy.PretrainVectors.v1" > maxout_pieces = 3 > hidden_size = 300 > loss = "cosine" > ``` Predict the word's vector from a static embeddings table as pretraining objective for a Tok2Vec layer. To use this objective, make sure that the `initialize.vectors` section in the config refers to a model with static vectors. | Name | Description | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `maxout_pieces` | The number of maxout pieces to use. Recommended values are `2` or `3`. ~~int~~ | | `hidden_size` | Size of the hidden layer of the model. ~~int~~ | | `loss` | The loss function can be either "cosine" or "L2". We typically recommend to use "cosine". ~~~str~~ | | **CREATES** | A callable function that can create the Model, given the `vocab` of the pipeline and the `tok2vec` layer to pretrain. ~~Callable[[Vocab, Model], Model]~~ | ### spacy.PretrainCharacters.v1 {id="pretrain_chars"} > #### Example config > > ```ini > [pretraining] > component = "tok2vec" > ... > > [pretraining.objective] > @architectures = "spacy.PretrainCharacters.v1" > maxout_pieces = 3 > hidden_size = 300 > n_characters = 4 > ``` Predict some number of leading and trailing UTF-8 bytes as pretraining objective for a Tok2Vec layer. | Name | Description | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `maxout_pieces` | The number of maxout pieces to use. Recommended values are `2` or `3`. ~~int~~ | | `hidden_size` | Size of the hidden layer of the model. ~~int~~ | | `n_characters` | The window of characters - e.g. if `n_characters = 2`, the model will try to predict the first two and last two characters of the word. ~~int~~ | | **CREATES** | A callable function that can create the Model, given the `vocab` of the pipeline and the `tok2vec` layer to pretrain. ~~Callable[[Vocab, Model], Model]~~ | ## Parser & NER architectures {id="parser"} ### spacy.TransitionBasedParser.v2 {id="TransitionBasedParser",source="spacy/ml/models/parser.py"} > #### Example Config > > ```ini > [model] > @architectures = "spacy.TransitionBasedParser.v2" > state_type = "ner" > extra_state_tokens = false > hidden_width = 64 > maxout_pieces = 2 > use_upper = true > > [model.tok2vec] > @architectures = "spacy.HashEmbedCNN.v2" > pretrained_vectors = null > width = 96 > depth = 4 > embed_size = 2000 > window_size = 1 > maxout_pieces = 3 > subword_features = true > ``` Build a transition-based parser model. Can apply to NER or dependency parsing. Transition-based parsing is an approach to structured prediction where the task of predicting the structure is mapped to a series of state transitions. You might find [this tutorial](https://explosion.ai/blog/parsing-english-in-python) helpful for background information. The neural network state prediction model consists of either two or three subnetworks: - **tok2vec**: Map each token into a vector representation. This subnetwork is run once for each batch. - **lower**: Construct a feature-specific vector for each `(token, feature)` pair. This is also run once for each batch. Constructing the state representation is then a matter of summing the component features and applying the non-linearity. - **upper** (optional): A feed-forward network that predicts scores from the state representation. If not present, the output from the lower model is used as action scores directly. | Name | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `tok2vec` | Subnetwork to map tokens into vector representations. ~~Model[List[Doc], List[Floats2d]]~~ | | `state_type` | Which task to extract features for. Possible values are "ner" and "parser". ~~str~~ | | `extra_state_tokens` | Whether to use an expanded feature set when extracting the state tokens. Slightly slower, but sometimes improves accuracy slightly. Defaults to `False`. ~~bool~~ | | `hidden_width` | The width of the hidden layer. ~~int~~ | | `maxout_pieces` | How many pieces to use in the state prediction layer. Recommended values are `1`, `2` or `3`. If `1`, the maxout non-linearity is replaced with a [`Relu`](https://thinc.ai/docs/api-layers#relu) non-linearity if `use_upper` is `True`, and no non-linearity if `False`. ~~int~~ | | `use_upper` | Whether to use an additional hidden layer after the state vector in order to predict the action scores. It is recommended to set this to `False` for large pretrained models such as transformers, and `True` for smaller networks. The upper layer is computed on CPU, which becomes a bottleneck on larger GPU-based models, where it's also less necessary. ~~bool~~ | | `nO` | The number of actions the model will predict between. Usually inferred from data at the beginning of training, or loaded from disk. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[List[Docs], List[List[Floats2d]]]~~ | [TransitionBasedParser.v1](/api/legacy#TransitionBasedParser_v1) had the exact same signature, but the `use_upper` argument was `True` by default. ## Tagging architectures {id="tagger",source="spacy/ml/models/tagger.py"} ### spacy.Tagger.v2 {id="Tagger"} > #### Example Config > > ```ini > [model] > @architectures = "spacy.Tagger.v2" > nO = null > normalize = false > > [model.tok2vec] > # ... > ``` Build a tagger model, using a provided token-to-vector component. The tagger model adds a linear layer with softmax activation to predict scores given the token vectors. | Name | Description | | ----------- | ------------------------------------------------------------------------------------------ | | `tok2vec` | Subnetwork to map tokens into vector representations. ~~Model[List[Doc], List[Floats2d]]~~ | | `nO` | The number of tags to output. Inferred from the data if `None`. ~~Optional[int]~~ | | `normalize` | Normalize probabilities during inference. Defaults to `False`. ~~bool~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[Floats2d]]~~ | - The `normalize` argument was added in `spacy.Tagger.v2`. `spacy.Tagger.v1` always normalizes probabilities during inference. The other arguments are shared between all versions. ## Text classification architectures {id="textcat",source="spacy/ml/models/textcat.py"} A text classification architecture needs to take a [`Doc`](/api/doc) as input, and produce a score for each potential label class. Textcat challenges can be binary (e.g. sentiment analysis) or involve multiple possible labels. Multi-label challenges can either have mutually exclusive labels (each example has exactly one label), or multiple labels may be applicable at the same time. As the properties of text classification problems can vary widely, we provide several different built-in architectures. It is recommended to experiment with different architectures and settings to determine what works best on your specific data and challenge. When the architecture for a text classification challenge contains a setting for `exclusive_classes`, it is important to use the correct value for the correct pipeline component. The `textcat` component should always be used for single-label use-cases where `exclusive_classes = true`, while the `textcat_multilabel` should be used for multi-label settings with `exclusive_classes = false`. ### spacy.TextCatEnsemble.v2 {id="TextCatEnsemble"} > #### Example Config > > ```ini > [model] > @architectures = "spacy.TextCatEnsemble.v2" > nO = null > > [model.linear_model] > @architectures = "spacy.TextCatBOW.v3" > exclusive_classes = true > length = 262144 > ngram_size = 1 > no_output_layer = false > > [model.tok2vec] > @architectures = "spacy.Tok2Vec.v2" > > [model.tok2vec.embed] > @architectures = "spacy.MultiHashEmbed.v2" > width = 64 > rows = [2000, 2000, 1000, 1000, 1000, 1000] > attrs = ["ORTH", "LOWER", "PREFIX", "SUFFIX", "SHAPE", "ID"] > include_static_vectors = false > > [model.tok2vec.encode] > @architectures = "spacy.MaxoutWindowEncoder.v2" > width = ${model.tok2vec.embed.width} > window_size = 1 > maxout_pieces = 3 > depth = 2 > ``` Stacked ensemble of a linear bag-of-words model and a neural network model. The neural network is built upon a Tok2Vec layer and uses attention. The setting for whether or not this model should cater for multi-label classification, is taken from the linear model, where it is stored in `model.attrs["multi_label"]`. | Name | Description | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `linear_model` | The linear bag-of-words model. ~~Model[List[Doc], Floats2d]~~ | | `tok2vec` | The `tok2vec` layer to build the neural network upon. ~~Model[List[Doc], List[Floats2d]]~~ | | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | [TextCatEnsemble.v1](/api/legacy#TextCatEnsemble_v1) was functionally similar, but used an internal `tok2vec` instead of taking it as argument: | Name | Description | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `exclusive_classes` | Whether or not categories are mutually exclusive. ~~bool~~ | | `pretrained_vectors` | Whether or not pretrained vectors will be used in addition to the feature vectors. ~~bool~~ | | `width` | Output dimension of the feature encoding step. ~~int~~ | | `embed_size` | Input dimension of the feature encoding step. ~~int~~ | | `conv_depth` | Depth of the tok2vec layer. ~~int~~ | | `window_size` | The number of contextual vectors to [concatenate](https://thinc.ai/docs/api-layers#expand_window) from the left and from the right. ~~int~~ | | `ngram_size` | Determines the maximum length of the n-grams in the BOW model. For instance, `ngram_size=3`would give unigram, trigram and bigram features. ~~int~~ | | `dropout` | The dropout rate. ~~float~~ | | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | ### spacy.TextCatBOW.v3 {id="TextCatBOW"} > #### Example Config > > ```ini > [model] > @architectures = "spacy.TextCatBOW.v3" > exclusive_classes = false > length = 262144 > ngram_size = 1 > no_output_layer = false > nO = null > ``` An n-gram "bag-of-words" model. This architecture should run much faster than the others, but may not be as accurate, especially if texts are short. | Name | Description | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `exclusive_classes` | Whether or not categories are mutually exclusive. ~~bool~~ | | `ngram_size` | Determines the maximum length of the n-grams in the BOW model. For instance, `ngram_size=3` would give unigram, trigram and bigram features. ~~int~~ | | `no_output_layer` | Whether or not to add an output layer to the model (`Softmax` activation if `exclusive_classes` is `True`, else `Logistic`). ~~bool~~ | | `length` | The size of the weights vector. The length will be rounded up to the next power of two if it is not a power of two. Defaults to `262144`. ~~int~~ | | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | - [TextCatBOW.v1](/api/legacy#TextCatBOW_v1) was not yet resizable. Since v2, new labels can be added to this component, even after training. - [TextCatBOW.v1](/api/legacy#TextCatBOW_v1) and [TextCatBOW.v2](/api/legacy#TextCatBOW_v2) used an erroneous sparse linear layer that only used a small number of the allocated parameters. - [TextCatBOW.v1](/api/legacy#TextCatBOW_v1) and [TextCatBOW.v2](/api/legacy#TextCatBOW_v2) did not have the `length` argument. ### spacy.TextCatParametricAttention.v1 {id="TextCatParametricAttention"} > #### Example Config > > ```ini > [model] > @architectures = "spacy.TextCatParametricAttention.v1" > exclusive_classes = true > nO = null > > [model.tok2vec] > @architectures = "spacy.Tok2Vec.v2" > > [model.tok2vec.embed] > @architectures = "spacy.MultiHashEmbed.v2" > width = 64 > rows = [2000, 2000, 1000, 1000, 1000, 1000] > attrs = ["ORTH", "LOWER", "PREFIX", "SUFFIX", "SHAPE", "ID"] > include_static_vectors = false > > [model.tok2vec.encode] > @architectures = "spacy.MaxoutWindowEncoder.v2" > width = ${model.tok2vec.embed.width} > window_size = 1 > maxout_pieces = 3 > depth = 2 > ``` A neural network model that is built upon Tok2Vec and uses parametric attention to attend to tokens that are relevant to text classification. | Name | Description | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `tok2vec` | The `tok2vec` layer to build the neural network upon. ~~Model[List[Doc], List[Floats2d]]~~ | | `exclusive_classes` | Whether or not categories are mutually exclusive. ~~bool~~ | | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | ### spacy.TextCatReduce.v1 {id="TextCatReduce"} > #### Example Config > > ```ini > [model] > @architectures = "spacy.TextCatReduce.v1" > exclusive_classes = false > use_reduce_first = false > use_reduce_last = false > use_reduce_max = false > use_reduce_mean = true > nO = null > > [model.tok2vec] > @architectures = "spacy.HashEmbedCNN.v2" > pretrained_vectors = null > width = 96 > depth = 4 > embed_size = 2000 > window_size = 1 > maxout_pieces = 3 > subword_features = true > ``` A classifier that pools token hidden representations of each `Doc` using first, max or mean reduction and then applies a classification layer. Reductions are concatenated when multiple reductions are used. `TextCatReduce` is a generalization of the older [`TextCatCNN`](/api/legacy#TextCatCNN_v2) model. `TextCatCNN` always uses a mean reduction, whereas `TextCatReduce` also supports first/max reductions. | Name | Description | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `exclusive_classes` | Whether or not categories are mutually exclusive. ~~bool~~ | | `tok2vec` | The [`tok2vec`](#tok2vec) layer of the model. ~~Model~~ | | `use_reduce_first` | Pool by using the hidden representation of the first token of a `Doc`. ~~bool~~ | | `use_reduce_last` | Pool by using the hidden representation of the last token of a `Doc`. ~~bool~~ | | `use_reduce_max` | Pool by taking the maximum values of the hidden representations of a `Doc`. ~~bool~~ | | `use_reduce_mean` | Pool by taking the mean of all hidden representations of a `Doc`. ~~bool~~ | | `nO` | Output dimension, determined by the number of different labels. If not set, the [`TextCategorizer`](/api/textcategorizer) component will set it when `initialize` is called. ~~Optional[int]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | ## Span classification architectures {id="spancat",source="spacy/ml/models/spancat.py"} ### spacy.SpanCategorizer.v1 {id="SpanCategorizer"} > #### Example Config > > ```ini > [model] > @architectures = "spacy.SpanCategorizer.v1" > scorer = {"@layers": "spacy.LinearLogistic.v1"} > > [model.reducer] > @layers = spacy.mean_max_reducer.v1" > hidden_size = 128 > > [model.tok2vec] > @architectures = "spacy.Tok2Vec.v1" > > [model.tok2vec.embed] > @architectures = "spacy.MultiHashEmbed.v1" > # ... > > [model.tok2vec.encode] > @architectures = "spacy.MaxoutWindowEncoder.v1" > # ... > ``` Build a span categorizer model to power a [`SpanCategorizer`](/api/spancategorizer) component, given a token-to-vector model, a reducer model to map the sequence of vectors for each span down to a single vector, and a scorer model to map the vectors to probabilities. | Name | Description | | ----------- | ------------------------------------------------------------------------------- | | `tok2vec` | The token-to-vector model. ~~Model[List[Doc], List[Floats2d]]~~ | | `reducer` | The reducer model. ~~Model[Ragged, Floats2d]~~ | | `scorer` | The scorer model. ~~Model[Floats2d, Floats2d]~~ | | **CREATES** | The model using the architecture. ~~Model[Tuple[List[Doc], Ragged], Floats2d]~~ | ### spacy.mean_max_reducer.v1 {id="mean_max_reducer"} Reduce sequences by concatenating their mean and max pooled vectors, and then combine the concatenated vectors with a hidden layer. | Name | Description | | ------------- | ------------------------------------- | | `hidden_size` | The size of the hidden layer. ~~int~~ | ## Entity linking architectures {id="entitylinker",source="spacy/ml/models/entity_linker.py"} An [`EntityLinker`](/api/entitylinker) component disambiguates textual mentions (tagged as named entities) to unique identifiers, grounding the named entities into the "real world". This requires 3 main components: - A [`KnowledgeBase`](/api/kb) (KB) holding the unique identifiers, potential synonyms and prior probabilities. - A candidate generation step to produce a set of likely identifiers, given a certain textual mention. - A machine learning [`Model`](https://thinc.ai/docs/api-model) that picks the most plausible ID from the set of candidates. ### spacy.EntityLinker.v2 {id="EntityLinker"} > #### Example Config > > ```ini > [model] > @architectures = "spacy.EntityLinker.v2" > nO = null > > [model.tok2vec] > @architectures = "spacy.HashEmbedCNN.v2" > pretrained_vectors = null > width = 96 > depth = 2 > embed_size = 2000 > window_size = 1 > maxout_pieces = 3 > subword_features = true > ``` The `EntityLinker` model architecture is a Thinc `Model` with a [`Linear`](https://thinc.ai/api-layers#linear) output layer. | Name | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `tok2vec` | The [`tok2vec`](#tok2vec) layer of the model. ~~Model~~ | | `nO` | Output dimension, determined by the length of the vectors encoding each entity in the KB. If the `nO` dimension is not set, the entity linking component will set it when `initialize` is called. ~~Optional[int]~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | ### spacy.EmptyKB.v1 {id="EmptyKB.v1"} A function that creates an empty `KnowledgeBase` from a [`Vocab`](/api/vocab) instance. | Name | Description | | ---------------------- | ----------------------------------------------------------------------------------- | | `entity_vector_length` | The length of the vectors encoding each entity in the KB. Defaults to `64`. ~~int~~ | ### spacy.EmptyKB.v2 {id="EmptyKB"} A function that creates an empty `KnowledgeBase` from a [`Vocab`](/api/vocab) instance. This is the default when a new entity linker component is created. It returns a `Callable[[Vocab, int], InMemoryLookupKB]`. ### spacy.KBFromFile.v1 {id="KBFromFile"} A function that reads an existing `KnowledgeBase` from file. | Name | Description | | --------- | -------------------------------------------------------- | | `kb_path` | The location of the KB that was stored to file. ~~Path~~ | ### spacy.CandidateGenerator.v1 {id="CandidateGenerator"} A function that takes as input a [`KnowledgeBase`](/api/kb) and a [`Span`](/api/span) object denoting a named entity, and returns a list of plausible [`Candidate`](/api/kb/#candidate) objects. The default `CandidateGenerator` uses the text of a mention to find its potential aliases in the `KnowledgeBase`. Note that this function is case-dependent. ### spacy.CandidateBatchGenerator.v1 {id="CandidateBatchGenerator"} A function that takes as input a [`KnowledgeBase`](/api/kb) and an `Iterable` of [`Span`](/api/span) objects denoting named entities, and returns a list of plausible [`Candidate`](/api/kb/#candidate) objects per specified [`Span`](/api/span). The default `CandidateBatchGenerator` uses the text of a mention to find its potential aliases in the `KnowledgeBase`. Note that this function is case-dependent. ## Coreference {id="coref-architectures",tag="experimental"} A [`CoreferenceResolver`](/api/coref) component identifies tokens that refer to the same entity. A [`SpanResolver`](/api/span-resolver) component infers spans from single tokens. Together these components can be used to reproduce traditional coreference models. You can also omit the `SpanResolver` if working with only token-level clusters is acceptable. ### spacy-experimental.Coref.v1 {id="Coref",tag="experimental"} > #### Example Config > > ```ini > > [model] > @architectures = "spacy-experimental.Coref.v1" > distance_embedding_size = 20 > dropout = 0.3 > hidden_size = 1024 > depth = 2 > antecedent_limit = 50 > antecedent_batch_size = 512 > > [model.tok2vec] > @architectures = "spacy-transformers.TransformerListener.v1" > grad_factor = 1.0 > upstream = "transformer" > pooling = {"@layers":"reduce_mean.v1"} > ``` The `Coref` model architecture is a Thinc `Model`. | Name | Description | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `tok2vec` | The [`tok2vec`](#tok2vec) layer of the model. ~~Model~~ | | `distance_embedding_size` | A representation of the distance between candidates. ~~int~~ | | `dropout` | The dropout to use internally. Unlike some Thinc models, this has separate dropout for the internal PyTorch layers. ~~float~~ | | `hidden_size` | Size of the main internal layers. ~~int~~ | | `depth` | Depth of the internal network. ~~int~~ | | `antecedent_limit` | How many candidate antecedents to keep after rough scoring. This has a significant effect on memory usage. Typical values would be 50 to 200, or higher for very long documents. ~~int~~ | | `antecedent_batch_size` | Internal batch size. ~~int~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], Floats2d]~~ | ### spacy-experimental.SpanResolver.v1 {id="SpanResolver",tag="experimental"} > #### Example Config > > ```ini > > [model] > @architectures = "spacy-experimental.SpanResolver.v1" > hidden_size = 1024 > distance_embedding_size = 64 > conv_channels = 4 > window_size = 1 > max_distance = 128 > prefix = "coref_head_clusters" > > [model.tok2vec] > @architectures = "spacy-transformers.TransformerListener.v1" > grad_factor = 1.0 > upstream = "transformer" > pooling = {"@layers":"reduce_mean.v1"} > ``` The `SpanResolver` model architecture is a Thinc `Model`. Note that `MentionClusters` is `List[List[Tuple[int, int]]]`. | Name | Description | | ------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `tok2vec` | The [`tok2vec`](#tok2vec) layer of the model. ~~Model~~ | | `hidden_size` | Size of the main internal layers. ~~int~~ | | `distance_embedding_size` | A representation of the distance between two candidates. ~~int~~ | | `conv_channels` | The number of channels in the internal CNN. ~~int~~ | | `window_size` | The number of neighboring tokens to consider in the internal CNN. `1` means consider one token on each side. ~~int~~ | | `max_distance` | The longest possible length of a predicted span. ~~int~~ | | `prefix` | The prefix that indicates spans to use for input data. ~~string~~ | | **CREATES** | The model using the architecture. ~~Model[List[Doc], List[MentionClusters]]~~ | --- ## File: website/docs/api/attributeruler.mdx --- title: AttributeRuler tag: class source: spacy/pipeline/attributeruler.py version: 3 teaser: 'Pipeline component for rule-based token attribute assignment' api_string_name: attribute_ruler api_trainable: false --- The attribute ruler lets you set token attributes for tokens identified by [`Matcher` patterns](/usage/rule-based-matching#matcher). The attribute ruler is typically used to handle exceptions for token attributes and to map values between attributes such as mapping fine-grained POS tags to coarse-grained POS tags. See the [usage guide](/usage/linguistic-features/#mappings-exceptions) for examples. ## Config and implementation {id="config"} The default config is defined by the pipeline component factory and describes how the component should be configured. You can override its settings via the `config` argument on [`nlp.add_pipe`](/api/language#add_pipe) or in your [`config.cfg` for training](/usage/training#config). > #### Example > > ```python > config = {"validate": True} > nlp.add_pipe("attribute_ruler", config=config) > ``` | Setting | Description | | ---------- | --------------------------------------------------------------------------------------------- | | `validate` | Whether patterns should be validated (passed to the `Matcher`). Defaults to `False`. ~~bool~~ | ```python %%GITHUB_SPACY/spacy/pipeline/attributeruler.py ``` ## AttributeRuler.\_\_init\_\_ {id="init",tag="method"} Initialize the attribute ruler. > #### Example > > ```python > # Construction via add_pipe > ruler = nlp.add_pipe("attribute_ruler") > ``` | Name | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `vocab` | The shared vocabulary to pass to the matcher. ~~Vocab~~ | | `name` | Instance name of the current pipeline component. Typically passed in automatically from the factory when the component is added. ~~str~~ | | _keyword-only_ | | | `validate` | Whether patterns should be validated (passed to the [`Matcher`](/api/matcher#init)). Defaults to `False`. ~~bool~~ | | `scorer` | The scoring method. Defaults to [`Scorer.score_token_attr`](/api/scorer#score_token_attr) for the attributes `"tag`", `"pos"`, `"morph"` and `"lemma"` and [`Scorer.score_token_attr_per_feat`](/api/scorer#score_token_attr_per_feat) for the attribute `"morph"`. ~~Optional[Callable]~~ | ## AttributeRuler.\_\_call\_\_ {id="call",tag="method"} Apply the attribute ruler to a `Doc`, setting token attributes for tokens matched by the provided patterns. | Name | Description | | ----------- | -------------------------------- | | `doc` | The document to process. ~~Doc~~ | | **RETURNS** | The processed document. ~~Doc~~ | ## AttributeRuler.add {id="add",tag="method"} Add patterns to the attribute ruler. The patterns are a list of `Matcher` patterns and the attributes are a dict of attributes to set on the matched token. If the pattern matches a span of more than one token, the `index` can be used to set the attributes for the token at that index in the span. The `index` may be negative to index from the end of the span. > #### Example > > ```python > ruler = nlp.add_pipe("attribute_ruler") > patterns = [[{"TAG": "VB"}]] > attrs = {"POS": "VERB"} > ruler.add(patterns=patterns, attrs=attrs) > ``` | Name | Description | | ---------- | --------------------------------------------------------------------------------------------------------------------------------- | | `patterns` | The `Matcher` patterns to add. ~~Iterable[List[Dict[Union[int, str], Any]]]~~ | | `attrs` | The attributes to assign to the target token in the matched span. ~~Dict[str, Any]~~ | | `index` | The index of the token in the matched span to modify. May be negative to index from the end of the span. Defaults to `0`. ~~int~~ | ## AttributeRuler.add_patterns {id="add_patterns",tag="method"} > #### Example > > ```python > ruler = nlp.add_pipe("attribute_ruler") > patterns = [ > { > "patterns": [[{"TAG": "VB"}]], "attrs": {"POS": "VERB"} > }, > { > "patterns": [[{"LOWER": "two"}, {"LOWER": "apples"}]], > "attrs": {"LEMMA": "apple"}, > "index": -1 > }, > ] > ruler.add_patterns(patterns) > ``` Add patterns from a list of pattern dicts. Each pattern dict can specify the keys `"patterns"`, `"attrs"` and `"index"`, which match the arguments of [`AttributeRuler.add`](/api/attributeruler#add). | Name | Description | | ---------- | -------------------------------------------------------------------------- | | `patterns` | The patterns to add. ~~Iterable[Dict[str, Union[List[dict], dict, int]]]~~ | ## AttributeRuler.patterns {id="patterns",tag="property"} Get all patterns that have been added to the attribute ruler in the `patterns_dict` format accepted by [`AttributeRuler.add_patterns`](/api/attributeruler#add_patterns). | Name | Description | | ----------- | -------------------------------------------------------------------------------------------- | | **RETURNS** | The patterns added to the attribute ruler. ~~List[Dict[str, Union[List[dict], dict, int]]]~~ | ## AttributeRuler.initialize {id="initialize",tag="method"} Initialize the component with data and used before training to load in rules from a file. This method is typically called by [`Language.initialize`](/api/language#initialize) and lets you customize arguments it receives via the [`[initialize.components]`](/api/data-formats#config-initialize) block in the config. > #### Example > > ```python > ruler = nlp.add_pipe("attribute_ruler") > ruler.initialize(lambda: [], nlp=nlp, patterns=patterns) > ``` > > ```ini > ### config.cfg > [initialize.components.attribute_ruler] > > [initialize.components.attribute_ruler.patterns] > @readers = "srsly.read_json.v1" > path = "corpus/attribute_ruler_patterns.json > ``` | Name | Description | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `get_examples` | Function that returns gold-standard annotations in the form of [`Example`](/api/example) objects (the training data). Not used by this component. ~~Callable[[], Iterable[Example]]~~ | | _keyword-only_ | | | `nlp` | The current `nlp` object. Defaults to `None`. ~~Optional[Language]~~ | | `patterns` | A list of pattern dicts with the keys as the arguments to [`AttributeRuler.add`](/api/attributeruler#add) (`patterns`/`attrs`/`index`) to add as patterns. Defaults to `None`. ~~Optional[Iterable[Dict[str, Union[List[dict], dict, int]]]]~~ | | `tag_map` | The tag map that maps fine-grained tags to coarse-grained tags and morphological features. Defaults to `None`. ~~Optional[Dict[str, Dict[Union[int, str], Union[int, str]]]]~~ | | `morph_rules` | The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. Defaults to `None`. ~~Optional[Dict[str, Dict[str, Dict[Union[int, str], Union[int, str]]]]]~~ | ## AttributeRuler.load_from_tag_map {id="load_from_tag_map",tag="method"} Load attribute ruler patterns from a tag map. | Name | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `tag_map` | The tag map that maps fine-grained tags to coarse-grained tags and morphological features. ~~Dict[str, Dict[Union[int, str], Union[int, str]]]~~ | ## AttributeRuler.load_from_morph_rules {id="load_from_morph_rules",tag="method"} Load attribute ruler patterns from morph rules. | Name | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `morph_rules` | The morph rules that map token text and fine-grained tags to coarse-grained tags, lemmas and morphological features. ~~Dict[str, Dict[str, Dict[Union[int, str], Union[int, str]]]]~~ | ## AttributeRuler.to_disk {id="to_disk",tag="method"} Serialize the pipe to disk. > #### Example > > ```python > ruler = nlp.add_pipe("attribute_ruler") > ruler.to_disk("/path/to/attribute_ruler") > ``` | Name | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `path` | A path to a directory, which will be created if it doesn't exist. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | ## AttributeRuler.from_disk {id="from_disk",tag="method"} Load the pipe from disk. Modifies the object in place and returns it. > #### Example > > ```python > ruler = nlp.add_pipe("attribute_ruler") > ruler.from_disk("/path/to/attribute_ruler") > ``` | Name | Description | | -------------- | ----------------------------------------------------------------------------------------------- | | `path` | A path to a directory. Paths may be either strings or `Path`-like objects. ~~Union[str, Path]~~ | | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The modified `AttributeRuler` object. ~~AttributeRuler~~ | ## AttributeRuler.to_bytes {id="to_bytes",tag="method"} > #### Example > > ```python > ruler = nlp.add_pipe("attribute_ruler") > ruler = ruler.to_bytes() > ``` Serialize the pipe to a bytestring. | Name | Description | | -------------- | ------------------------------------------------------------------------------------------- | | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The serialized form of the `AttributeRuler` object. ~~bytes~~ | ## AttributeRuler.from_bytes {id="from_bytes",tag="method"} Load the pipe from a bytestring. Modifies the object in place and returns it. > #### Example > > ```python > ruler_bytes = ruler.to_bytes() > ruler = nlp.add_pipe("attribute_ruler") > ruler.from_bytes(ruler_bytes) > ``` | Name | Description | | -------------- | ------------------------------------------------------------------------------------------- | | `bytes_data` | The data to load from. ~~bytes~~ | | _keyword-only_ | | | `exclude` | String names of [serialization fields](#serialization-fields) to exclude. ~~Iterable[str]~~ | | **RETURNS** | The `AttributeRuler` object. ~~AttributeRuler~~ | ## Serialization fields {id="serialization-fields"} During serialization, spaCy will export several data fields used to restore different aspects of the object. If needed, you can exclude them from serialization by passing in the string names via the `exclude` argument. > #### Example > > ```python > data = ruler.to_disk("/path", exclude=["vocab"]) > ``` | Name | Description | | ---------- | --------------------------------------------------------------- | | `vocab` | The shared [`Vocab`](/api/vocab). | | `patterns` | The `Matcher` patterns. You usually don't want to exclude this. | | `attrs` | The attributes to set. You usually don't want to exclude this. | | `indices` | The token indices. You usually don't want to exclude this. | --- ## File: website/docs/api/attributes.mdx --- title: Attributes teaser: Token attributes source: spacy/attrs.pyx --- [Token](/api/token) attributes are specified using internal IDs in many places including: - [`Matcher` patterns](/api/matcher#patterns), - [`Doc.to_array`](/api/doc#to_array) and [`Doc.from_array`](/api/doc#from_array) - [`Doc.has_annotation`](/api/doc#has_annotation) - [`MultiHashEmbed`](/api/architectures#MultiHashEmbed) Tok2Vec architecture `attrs` > ```python > import spacy > from spacy.attrs import DEP > > nlp = spacy.blank("en") > doc = nlp("There are many attributes.") > > # DEP always has the same internal value > assert DEP == 76 > > # "DEP" is automatically converted to DEP > assert DEP == nlp.vocab.strings["DEP"] > assert doc.has_annotation(DEP) == doc.has_annotation("DEP") > > # look up IDs in spacy.attrs.IDS > from spacy.attrs import IDS > assert IDS["DEP"] == DEP > ``` All methods automatically convert between the string version of an ID (`"DEP"`) and the internal integer symbols (`DEP`). The internal IDs can be imported from `spacy.attrs` or retrieved from the [`StringStore`](/api/stringstore). A map from string attribute names to internal attribute IDs is stored in `spacy.attrs.IDS`. The corresponding [`Token` object attributes](/api/token#attributes) can be accessed using the same names in lowercase, e.g. `token.orth` or `token.length`. For attributes that represent string values, the internal integer ID is accessed as `Token.attr`, e.g. `token.dep`, while the string value can be retrieved by appending `_` as in `token.dep_`. | Attribute | Description | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DEP` | The token's dependency label. ~~str~~ | | `ENT_ID` | The token's entity ID (`ent_id`). ~~str~~ | | `ENT_IOB` | The IOB part of the token's entity tag. Uses custom integer values rather than the string store: unset is `0`, `I` is `1`, `O` is `2`, and `B` is `3`. ~~str~~ | | `ENT_KB_ID` | The token's entity knowledge base ID. ~~str~~ | | `ENT_TYPE` | The token's entity label. ~~str~~ | | `IS_ALPHA` | Token text consists of alphabetic characters. ~~bool~~ | | `IS_ASCII` | Token text consists of ASCII characters. ~~bool~~ | | `IS_DIGIT` | Token text consists of digits. ~~bool~~ | | `IS_LOWER` | Token text is in lowercase. ~~bool~~ | | `IS_PUNCT` | Token is punctuation. ~~bool~~ | | `IS_SPACE` | Token is whitespace. ~~bool~~ | | `IS_STOP` | Token is a stop word. ~~bool~~ | | `IS_TITLE` | Token text is in titlecase. ~~bool~~ | | `IS_UPPER` | Token text is in uppercase. ~~bool~~ | | `LEMMA` | The token's lemma. ~~str~~ | | `LENGTH` | The length of the token text. ~~int~~ | | `LIKE_EMAIL` | Token text resembles an email address. ~~bool~~ | | `LIKE_NUM` | Token text resembles a number. ~~bool~~ | | `LIKE_URL` | Token text resembles a URL. ~~bool~~ | | `LOWER` | The lowercase form of the token text. ~~str~~ | | `MORPH` | The token's morphological analysis. ~~MorphAnalysis~~ | | `NORM` | The normalized form of the token text. ~~str~~ | | `ORTH` | The exact verbatim text of a token. ~~str~~ | | `POS` | The token's universal part of speech (UPOS). ~~str~~ | | `SENT_START` | Token is start of sentence. ~~bool~~ | | `SHAPE` | The token's shape. ~~str~~ | | `SPACY` | Token has a trailing space. ~~bool~~ | | `TAG` | The token's fine-grained part of speech. ~~str~~ |