spaCy Documentation

Package naming conventions {id="conventions"} Copy Chapter

In general, spaCy expects all pipeline packages to follow the naming convention
of [lang]_[name]. For spaCy's pipelines, we also chose to divide the name into
three components:

1. Type: Capabilities (e.g. core for general-purpose pipeline with
   tagging, parsing, lemmatization and named entity recognition, or dep for
   only tagging, parsing and lemmatization).

2. Genre: Type of text the pipeline is trained on, e.g. web or news.

3. Size: Package size indicator, sm, md, lg or trf.

   sm and trf pipelines have no static word vectors.

   For pipelines with default vectors, md has a reduced word vector table with
   20k unique vectors for ~500k words and lg has a large word vector table
   with ~500k entries.

   For pipelines with floret vectors, md vector tables have 50k entries and
   lg vector tables have 200k entries.

For example, en_core_web_sm is a small English
pipeline trained on written web text (blogs, news, comments), that includes
vocabulary, syntax and entities.

Package versioning {id="model-versioning"} Copy Section

Additionally, the pipeline package versioning reflects both the compatibility
with spaCy, as well as the model version. A package version a.b.c translates
to:

• a: spaCy major version. For example, 2 for spaCy v2.x.
• b: spaCy minor version. For example, 3 for spaCy v2.3.x.
• c: Model version. Different model config: e.g. from being trained on
  different data, with different parameters, for different numbers of
  iterations, with different vectors, etc.

For a detailed compatibility overview, see the
compatibility.json.
This is also the source of spaCy's internal compatibility check, performed when
you run the download command.

Trained pipeline design {id="design"} Copy Chapter

The spaCy v3 trained pipelines are designed to be efficient and configurable.
For example, multiple components can share a common "token-to-vector" model and
it's easy to swap out or disable the lemmatizer. The pipelines are designed to
be efficient in terms of speed and size and work well when the pipeline is run
in full.

When modifying a trained pipeline, it's important to understand how the
components depend on each other. Unlike spaCy v2, where the tagger,
parser and ner components were all independent, some v3 components depend on
earlier components in the pipeline. As a result, disabling or reordering
components can affect the annotation quality or lead to warnings and errors.

Main changes from spaCy v2 models:

• The Tok2Vec component may be a separate, shared component. A
  component like a tagger or parser can
  listen to an earlier tok2vec or
  transformer rather than having its own separate tok2vec layer.
• Rule-based exceptions move from individual components to the
  attribute_ruler. Lemma and POS exceptions move from the tokenizer exceptions
  to the attribute ruler and the tag map and morph rules move from the tagger to
  the attribute ruler.
• The lemmatizer tables and processing move from the vocab and tagger to a
  separate lemmatizer component.

CNN/CPU pipeline design {id="design-cnn"} Copy Section

In the sm/md/lg models:

• The tagger, morphologizer and parser components listen to the tok2vec
  component. If the lemmatizer is trainable (v3.3+), lemmatizer also listens
  to tok2vec.
• The attribute_ruler maps token.tag to token.pos if there is no
  morphologizer. The attribute_ruler additionally makes sure whitespace is
  tagged consistently and copies token.pos to token.tag if there is no
  tagger. For English, the attribute ruler can improve its mapping from
  token.tag to token.pos if dependency parses from a parser are present,
  but the parser is not required.
• The lemmatizer component for many languages requires token.pos annotation
  from either tagger+attribute_ruler or morphologizer.
• The ner component is independent with its own internal tok2vec layer.

CNN/CPU pipelines with floret vectors

The Croatian, Finnish, Korean, Slovenian, Swedish and Ukrainian md and lg
pipelines use floret vectors instead of default vectors.
If you're running a trained pipeline on texts and working with Doc
objects, you shouldn't notice any difference with floret vectors. With floret
vectors no tokens are out-of-vocabulary, so
Token.is_oov will return False for all tokens.

If you access vectors directly for similarity comparisons, there are a few
differences because floret vectors don't include a fixed word list like the
vector keys for default vectors.

• If your workflow iterates over the vector keys, you need to use an external
  word list instead:

  diff Copy

  - lexemes = [nlp.vocab[orth] for orth in nlp.vocab.vectors]
  + lexemes = [nlp.vocab[word] for word in external_word_list]

• Vectors.most_similar is not supported because
  there's no fixed list of vectors to compare your vectors to.

Transformer pipeline design {id="design-trf"} Copy Section

In the transformer (trf) pipelines, the tagger, parser and ner (if
present) all listen to the transformer component. The attribute_ruler and
lemmatizer have the same configuration as in the CNN models.

For spaCy v3.0-v3.6, trf pipelines use
spacy-transformers and the
transformer output in doc._.trf_data is a
TransformerData object.

For spaCy v3.7+, trf pipelines use
spacy-curated-transformers
and doc._.trf_data is a
DocTransformerOutput object.

Modifying the default pipeline {id="design-modify"} Copy Section

For faster processing, you may only want to run a subset of the components in a
trained pipeline. The disable and exclude arguments to
spacy.load let you control which components are
loaded and run. Disabled components are loaded in the background so it's
possible to reenable them in the same pipeline in the future with
nlp.enable_pipe. To skip loading a component
completely, use exclude instead of disable.

Disable part-of-speech tagging and lemmatization

To disable part-of-speech tagging and lemmatization, disable the tagger,
morphologizer, attribute_ruler and lemmatizer components.

# Note: English doesn't include a morphologizer
nlp = spacy.load("en_core_web_sm", disable=["tagger", "attribute_ruler", "lemmatizer"])
nlp = spacy.load("en_core_web_trf", disable=["tagger", "attribute_ruler", "lemmatizer"])

The lemmatizer depends on tagger+attribute_ruler or morphologizer for a
number of languages. If you disable any of these components, you'll see
lemmatizer warnings unless the lemmatizer is also disabled.

v3.3: Catalan, English, French, Russian and Spanish

v3.0-v3.2: Catalan, Dutch, English, French, Greek, Italian, Macedonian,
Norwegian, Polish, Russian and Spanish

Use senter rather than parser for fast sentence segmentation

If you need fast sentence segmentation without dependency parses, disable the
parser use the senter component instead:

nlp = spacy.load("en_core_web_sm")
nlp.disable_pipe("parser")
nlp.enable_pipe("senter")

The senter component is ~10× faster than the parser and more accurate
than the rule-based sentencizer.

Switch from trainable lemmatizer to default lemmatizer

Since v3.3, a number of pipelines use a trainable lemmatizer. You can check
whether the lemmatizer is trainable:

nlp = spacy.load("de_core_web_sm")
assert nlp.get_pipe("lemmatizer").is_trainable

If you'd like to switch to a non-trainable lemmatizer that's similar to v3.2 or
earlier, you can replace the trainable lemmatizer with the default non-trainable
lemmatizer:

# Requirements: pip install spacy-lookups-data
nlp = spacy.load("de_core_web_sm")
# Remove existing lemmatizer
nlp.remove_pipe("lemmatizer")
# Add non-trainable lemmatizer from language defaults
# and load lemmatizer tables from spacy-lookups-data
nlp.add_pipe("lemmatizer").initialize()

Switch from rule-based to lookup lemmatization

For the Dutch, English, French, Greek, Macedonian, Norwegian and Spanish
pipelines, you can swap out a trainable or rule-based lemmatizer for a lookup
lemmatizer:

# Requirements: pip install spacy-lookups-data
nlp = spacy.load("en_core_web_sm")
nlp.remove_pipe("lemmatizer")
nlp.add_pipe("lemmatizer", config={"mode": "lookup"}).initialize()

Disable everything except NER

For the non-transformer models, the ner component is independent, so you can
disable everything else:

nlp = spacy.load("en_core_web_sm", disable=["tok2vec", "tagger", "parser", "attribute_ruler", "lemmatizer"])

In the transformer models, ner listens to the transformer component, so you
can disable all components related tagging, parsing, and lemmatization.

nlp = spacy.load("en_core_web_trf", disable=["tagger", "parser", "attribute_ruler", "lemmatizer"])

Move NER to the end of the pipeline

As of v3.1, the NER component is at the end of the pipeline by default.

For access to POS and LEMMA features in an entity_ruler, move ner to the
end of the pipeline after attribute_ruler and lemmatizer:

# load without NER
nlp = spacy.load("en_core_web_sm", exclude=["ner"])

# source NER from the same pipeline package as the last component
nlp.add_pipe("ner", source=spacy.load("en_core_web_sm"))

# insert the entity ruler
nlp.add_pipe("entity_ruler", before="ner")

File: website/docs/api/architectures.mdx Copy Chapter

title: Model Architectures
teaser: Pre-defined model architectures included with the core library
source: spacy/ml/models
menu:

• ['Tok2Vec', 'tok2vec-arch']
• ['Transformers', 'transformers']
• ['Pretraining', 'pretrain']
• ['Parser & NER', 'parser']
• ['Tagging', 'tagger']
• ['Text Classification', 'textcat']
• ['Span Classification', 'spancat']
• ['Entity Linking', 'entitylinker']
• ['Coreference', 'coref-architectures']

A model architecture is a function that wires up a
Model instance, which you can then use in a
pipeline component or as a layer of a larger network. This page documents
spaCy's built-in architectures that are used for different NLP tasks. All
trainable built-in components expect a model
argument defined in the config and document their the default architecture.
Custom architectures can be registered using the
@spacy.registry.architectures decorator and used as
part of the training config. Also see the
usage documentation on
layers and model architectures.

Tok2Vec architectures {id="tok2vec-arch",source="spacy/ml/models/tok2vec.py"} Copy Chapter

spacy.Tok2Vec.v2 {id="Tok2Vec"} Copy Section

Example config

ini Copy

[model]
@architectures = "spacy.Tok2Vec.v2"

[model.embed]
@architectures = "spacy.CharacterEmbed.v2"
# ...

[model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
# ...

Construct a tok2vec model out of two subnetworks: one for embedding and one for
encoding. See the
"Embed, Encode, Attend, Predict"
blog post for background.

spacy.HashEmbedCNN.v2 {id="HashEmbedCNN"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy.HashEmbedCNN.v2"
pretrained_vectors = null
width = 96
depth = 4
embed_size = 2000
window_size = 1
maxout_pieces = 3
subword_features = true

Build spaCy's "standard" tok2vec layer. This layer is defined by a
MultiHashEmbed embedding layer that uses
subword features, and a
MaxoutWindowEncoder encoding layer
consisting of a CNN and a layer-normalized maxout activation function.

spacy.Tok2VecListener.v1 {id="Tok2VecListener"} Copy Section

Example config

ini Copy

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.HashEmbedCNN.v2"
width = 342

[components.tagger]
factory = "tagger"

[components.tagger.model]
@architectures = "spacy.Tagger.v2"

[components.tagger.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.width}

A listener is used as a sublayer within a component such as a
DependencyParser,
EntityRecognizeror
TextCategorizer. Usually you'll have multiple
listeners connecting to a single upstream Tok2Vec component
that's earlier in the pipeline. The listener layers act as proxies, passing
the predictions from the Tok2Vec component into downstream components, and
communicating gradients back upstream.

Instead of defining its own Tok2Vec instance, a model architecture like
Tagger can define a listener as its tok2vec
argument that connects to the shared tok2vec component in the pipeline.

Listeners work by caching the Tok2Vec output for a given batch of Docs. This
means that in order for a component to work with the listener, the batch of
Docs passed to the listener must be the same as the batch of Docs passed to
the Tok2Vec. As a result, any manipulation of the Docs which would affect
Tok2Vec output, such as to create special contexts or remove Docs for which
no prediction can be made, must happen inside the model, after the call to
the Tok2Vec component.

spacy.MultiHashEmbed.v2 {id="MultiHashEmbed"} Copy Section

Example config

ini Copy

[model]
@architectures = "spacy.MultiHashEmbed.v2"
width = 64
attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"]
rows = [2000, 1000, 1000, 1000]
include_static_vectors = true

Construct an embedding layer that separately embeds a number of lexical
attributes using hash embedding, concatenates the results, and passes it through
a feed-forward subnetwork to build a mixed representation. The features used can
be configured with the attrs argument. The suggested attributes are NORM,
PREFIX, SUFFIX and SHAPE. This lets the model take into account some
subword information, without construction a fully character-based
representation. If pretrained vectors are available, they can be included in the
representation as well, with the vectors table kept static (i.e. it's not
updated).

spacy.CharacterEmbed.v2 {id="CharacterEmbed"} Copy Section

Example config

ini Copy

[model]
@architectures = "spacy.CharacterEmbed.v2"
width = 128
rows = 7000
nM = 64
nC = 8

Construct an embedded representation based on character embeddings, using a
feed-forward network. A fixed number of UTF-8 byte characters are used for each
word, taken from the beginning and end of the word equally. Padding is used in
the center for words that are too short.

For instance, let's say nC=4, and the word is "jumping". The characters used
will be "jung" (two from the start, two from the end). If we had nC=8, the
characters would be "jumpping": 4 from the start, 4 from the end. This ensures
that the final character is always in the last position, instead of being in an
arbitrary position depending on the word length.

The characters are embedded in a embedding table with a given number of rows,
and the vectors concatenated. A hash-embedded vector of the NORM of the word
is also concatenated on, and the result is then passed through a feed-forward
network to construct a single vector to represent the information.

spacy.MaxoutWindowEncoder.v2 {id="MaxoutWindowEncoder"} Copy Section

Example config

ini Copy

[model]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 128
window_size = 1
maxout_pieces = 3
depth = 4

Encode context using convolutions with maxout activation, layer normalization
and residual connections.

spacy.MishWindowEncoder.v2 {id="MishWindowEncoder"} Copy Section

Example config

ini Copy

[model]
@architectures = "spacy.MishWindowEncoder.v2"
width = 64
window_size = 1
depth = 4

Encode context using convolutions with
Mish activation, layer normalization
and residual connections.

spacy.TorchBiLSTMEncoder.v1 {id="TorchBiLSTMEncoder"} Copy Section

Example config

ini Copy

[model]
@architectures = "spacy.TorchBiLSTMEncoder.v1"
width = 64
depth = 2
dropout = 0.0

Encode context using bidirectional LSTM layers. Requires
PyTorch.

spacy.StaticVectors.v2 {id="StaticVectors"} Copy Section

Example config

ini Copy

[model]
@architectures = "spacy.StaticVectors.v2"
nO = null
nM = null
dropout = 0.2
key_attr = "ORTH"

[model.init_W]
@initializers = "glorot_uniform_init.v1"

Embed Doc objects with their vocab's vectors table, applying a
learned linear projection to control the dimensionality. Unknown tokens are
mapped to a zero vector. See the documentation on
static vectors for details.

spacy.FeatureExtractor.v1 {id="FeatureExtractor"} Copy Section

Example config

ini Copy

[model]
@architectures = "spacy.FeatureExtractor.v1"
columns = ["NORM", "PREFIX", "SUFFIX", "SHAPE", "ORTH"]

Extract arrays of input features from Doc objects. Expects a list
of feature names to extract, which should refer to token attributes.

Transformer architectures {id="transformers",source="github.com/explosion/spacy-transformers/blob/master/spacy_transformers/architectures.py"} Copy Chapter

The following architectures are provided by the package
spacy-transformers. See the
usage documentation for how to
integrate the architectures into your training config.

Note that in order to use these architectures in your config, you need to
install the
spacy-transformers. See the
installation docs
for details and system requirements.

spacy-transformers.TransformerModel.v3 {id="TransformerModel"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "roberta-base"
tokenizer_config = {"use_fast": true}
transformer_config = {}
mixed_precision = true
grad_scaler_config = {"init_scale": 32768}

[model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

Load and wrap a transformer model from the
HuggingFace transformers library. You
can use any transformer that has pretrained weights and a PyTorch
implementation. The name variable is passed through to the underlying library,
so it can be either a string or a path. If it's a string, the pretrained weights
will be downloaded via the transformers library if they are not already
available locally.

In order to support longer documents, the
TransformerModel layer allows you to pass
in a get_spans function that will divide up the Doc objects
before passing them through the transformer. Your spans are allowed to overlap
or exclude tokens. This layer is usually used directly by the
Transformer component, which allows you to share the
transformer weights across your pipeline. For a layer that's configured for use
in other components, see
Tok2VecTransformer.

Mixed-precision support is currently an experimental feature.

spacy-transformers.TransformerListener.v1 {id="TransformerListener"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[model.pooling]
@layers = "reduce_mean.v1"

Create a TransformerListener layer, which will connect to a
Transformer component earlier in the pipeline. The layer
takes a list of Doc objects as input, and produces a list of
2-dimensional arrays as output, with each array having one row per token. Most
spaCy models expect a sublayer with this signature, making it easy to connect
them to a transformer model via this sublayer. Transformer models usually
operate over wordpieces, which usually don't align one-to-one against spaCy
tokens. The layer therefore requires a reduction operation in order to calculate
a single token vector given zero or more wordpiece vectors.

spacy-transformers.Tok2VecTransformer.v3 {id="Tok2VecTransformer"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy-transformers.Tok2VecTransformer.v3"
name = "albert-base-v2"
tokenizer_config = {"use_fast": false}
transformer_config = {}
grad_factor = 1.0
mixed_precision = true
grad_scaler_config = {"init_scale": 32768}

Use a transformer as a Tok2Vec layer directly. This does
not allow multiple components to share the transformer weights and does
not allow the transformer to set annotations into the Doc
object, but it's a simpler solution if you only need the transformer within
one component.

Mixed-precision support is currently an experimental feature.

Curated Transformer architectures {id="curated-trf",source="https://github.com/explosion/spacy-curated-transformers/blob/main/spacy_curated_transformers/models/architectures.py"} Copy Chapter

The following architectures are provided by the package
spacy-curated-transformers.
See the usage documentation for
how to integrate the architectures into your training config.

When loading the model
from the Hugging Face Hub, the
model config's parameters must be same as the hyperparameters used by the
pre-trained model. The
init fill-curated-transformer CLI
command can be used to automatically fill in these values.

spacy-curated-transformers.AlbertTransformer.v1 Copy Section

Construct an ALBERT transformer model.

spacy-curated-transformers.BertTransformer.v1 Copy Section

Construct a BERT transformer model.

spacy-curated-transformers.CamembertTransformer.v1 Copy Section

Construct a CamemBERT transformer model.

spacy-curated-transformers.RobertaTransformer.v1 Copy Section

Construct a RoBERTa transformer model.

spacy-curated-transformers.XlmrTransformer.v1 Copy Section

Construct a XLM-RoBERTa transformer model.

spacy-curated-transformers.ScalarWeight.v1 Copy Section

Construct a model that accepts a list of transformer layer outputs and returns a
weighted representation of the same.

spacy-curated-transformers.TransformerLayersListener.v1 Copy Section

Construct a listener layer that communicates with one or more upstream
Transformer components. This layer extracts the output of the last transformer
layer and performs pooling over the individual pieces of each Doc token,
returning their corresponding representations. The upstream name should either
be the wildcard string '*', or the name of the Transformer component.

In almost all cases, the wildcard string will suffice as there'll only be one
upstream Transformer component. But in certain situations, e.g: you have
disjoint datasets for certain tasks, or you'd like to use a pre-trained pipeline
but a downstream task requires its own token representations, you could end up
with more than one Transformer component in the pipeline.

spacy-curated-transformers.LastTransformerLayerListener.v1 Copy Section

Construct a listener layer that communicates with one or more upstream
Transformer components. This layer extracts the output of the last transformer
layer and performs pooling over the individual pieces of each Doc token,
returning their corresponding representations. The upstream name should either
be the wildcard string '*', or the name of the Transformer component.

In almost all cases, the wildcard string will suffice as there'll only be one
upstream Transformer component. But in certain situations, e.g: you have
disjoint datasets for certain tasks, or you'd like to use a pre-trained pipeline
but a downstream task requires its own token representations, you could end up
with more than one Transformer component in the pipeline.

spacy-curated-transformers.ScalarWeightingListener.v1 Copy Section

Construct a listener layer that communicates with one or more upstream
Transformer components. This layer calculates a weighted representation of all
transformer layer outputs and performs pooling over the individual pieces of
each Doc token, returning their corresponding representations.

Requires its upstream Transformer components to return all layer outputs from
their models. The upstream name should either be the wildcard string '*', or
the name of the Transformer component.

In almost all cases, the wildcard string will suffice as there'll only be one
upstream Transformer component. But in certain situations, e.g: you have
disjoint datasets for certain tasks, or you'd like to use a pre-trained pipeline
but a downstream task requires its own token representations, you could end up
with more than one Transformer component in the pipeline.

spacy-curated-transformers.BertWordpieceEncoder.v1 Copy Section

Construct a WordPiece piece encoder model that accepts a list of token sequences
or documents and returns a corresponding list of piece identifiers. This encoder
also splits each token on punctuation characters, as expected by most BERT
models.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.ByteBpeEncoder.v1 Copy Section

Construct a Byte-BPE piece encoder model that accepts a list of token sequences
or documents and returns a corresponding list of piece identifiers.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.CamembertSentencepieceEncoder.v1 Copy Section

Construct a SentencePiece piece encoder model that accepts a list of token
sequences or documents and returns a corresponding list of piece identifiers
with CamemBERT post-processing applied.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.CharEncoder.v1 Copy Section

Construct a character piece encoder model that accepts a list of token sequences
or documents and returns a corresponding list of piece identifiers.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.SentencepieceEncoder.v1 Copy Section

Construct a SentencePiece piece encoder model that accepts a list of token
sequences or documents and returns a corresponding list of piece identifiers.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.WordpieceEncoder.v1 Copy Section

Construct a WordPiece piece encoder model that accepts a list of token sequences
or documents and returns a corresponding list of piece identifiers. This encoder
also splits each token on punctuation characters, as expected by most BERT
models.

This model must be separately initialized using an appropriate loader.

spacy-curated-transformers.XlmrSentencepieceEncoder.v1 Copy Section

Construct a SentencePiece piece encoder model that accepts a list of token
sequences or documents and returns a corresponding list of piece identifiers
with XLM-RoBERTa post-processing applied.

This model must be separately initialized using an appropriate loader.

Pretraining architectures {id="pretrain",source="spacy/ml/models/multi_task.py"} Copy Chapter

The spacy pretrain command lets you initialize a Tok2Vec layer in your
pipeline with information from raw text. To this end, additional layers are
added to build a network for a temporary task that forces the Tok2Vec layer to
learn something about sentence structure and word cooccurrence statistics. Two
pretraining objectives are available, both of which are variants of the cloze
task Devlin et al. (2018) introduced for
BERT.

For more information, see the section on
pretraining.

spacy.PretrainVectors.v1 {id="pretrain_vectors"} Copy Section

Example config

ini Copy

[pretraining]
component = "tok2vec"

[initialize]
vectors = "en_core_web_lg"
...

[pretraining.objective]
@architectures = "spacy.PretrainVectors.v1"
maxout_pieces = 3
hidden_size = 300
loss = "cosine"

Predict the word's vector from a static embeddings table as pretraining
objective for a Tok2Vec layer. To use this objective, make sure that the
initialize.vectors section in the config refers to a model with static
vectors.

spacy.PretrainCharacters.v1 {id="pretrain_chars"} Copy Section

Example config

ini Copy

[pretraining]
component = "tok2vec"
...

[pretraining.objective]
@architectures = "spacy.PretrainCharacters.v1"
maxout_pieces = 3
hidden_size = 300
n_characters = 4

Predict some number of leading and trailing UTF-8 bytes as pretraining objective
for a Tok2Vec layer.

Parser & NER architectures {id="parser"} Copy Chapter

spacy.TransitionBasedParser.v2 {id="TransitionBasedParser",source="spacy/ml/models/parser.py"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true

[model.tok2vec]
@architectures = "spacy.HashEmbedCNN.v2"
pretrained_vectors = null
width = 96
depth = 4
embed_size = 2000
window_size = 1
maxout_pieces = 3
subword_features = true

Build a transition-based parser model. Can apply to NER or dependency parsing.
Transition-based parsing is an approach to structured prediction where the task
of predicting the structure is mapped to a series of state transitions. You
might find this tutorial
helpful for background information. The neural network state prediction model
consists of either two or three subnetworks:

• tok2vec: Map each token into a vector representation. This subnetwork is
  run once for each batch.
• lower: Construct a feature-specific vector for each (token, feature)
  pair. This is also run once for each batch. Constructing the state
  representation is then a matter of summing the component features and applying
  the non-linearity.
• upper (optional): A feed-forward network that predicts scores from the
  state representation. If not present, the output from the lower model is used
  as action scores directly.

Tagging architectures {id="tagger",source="spacy/ml/models/tagger.py"} Copy Chapter

spacy.Tagger.v2 {id="Tagger"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false

[model.tok2vec]
# ...

Build a tagger model, using a provided token-to-vector component. The tagger
model adds a linear layer with softmax activation to predict scores given the
token vectors.

Text classification architectures {id="textcat",source="spacy/ml/models/textcat.py"} Copy Chapter

A text classification architecture needs to take a Doc as input,
and produce a score for each potential label class. Textcat challenges can be
binary (e.g. sentiment analysis) or involve multiple possible labels.
Multi-label challenges can either have mutually exclusive labels (each example
has exactly one label), or multiple labels may be applicable at the same time.

As the properties of text classification problems can vary widely, we provide
several different built-in architectures. It is recommended to experiment with
different architectures and settings to determine what works best on your
specific data and challenge.

When the architecture for a text classification challenge contains a setting for
exclusive_classes, it is important to use the correct value for the correct
pipeline component. The textcat component should always be used for
single-label use-cases where exclusive_classes = true, while the
textcat_multilabel should be used for multi-label settings with
exclusive_classes = false.

spacy.TextCatEnsemble.v2 {id="TextCatEnsemble"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy.TextCatEnsemble.v2"
nO = null

[model.linear_model]
@architectures = "spacy.TextCatBOW.v3"
exclusive_classes = true
length = 262144
ngram_size = 1
no_output_layer = false

[model.tok2vec]
@architectures = "spacy.Tok2Vec.v2"

[model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = 64
rows = [2000, 2000, 1000, 1000, 1000, 1000]
attrs = ["ORTH", "LOWER", "PREFIX", "SUFFIX", "SHAPE", "ID"]
include_static_vectors = false

[model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = ${model.tok2vec.embed.width}
window_size = 1
maxout_pieces = 3
depth = 2

Stacked ensemble of a linear bag-of-words model and a neural network model. The
neural network is built upon a Tok2Vec layer and uses attention. The setting for
whether or not this model should cater for multi-label classification, is taken
from the linear model, where it is stored in model.attrs["multi_label"].

spacy.TextCatBOW.v3 {id="TextCatBOW"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy.TextCatBOW.v3"
exclusive_classes = false
length = 262144
ngram_size = 1
no_output_layer = false
nO = null

An n-gram "bag-of-words" model. This architecture should run much faster than
the others, but may not be as accurate, especially if texts are short.

spacy.TextCatParametricAttention.v1 {id="TextCatParametricAttention"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy.TextCatParametricAttention.v1"
exclusive_classes = true
nO = null

[model.tok2vec]
@architectures = "spacy.Tok2Vec.v2"

[model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = 64
rows = [2000, 2000, 1000, 1000, 1000, 1000]
attrs = ["ORTH", "LOWER", "PREFIX", "SUFFIX", "SHAPE", "ID"]
include_static_vectors = false

[model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = ${model.tok2vec.embed.width}
window_size = 1
maxout_pieces = 3
depth = 2

A neural network model that is built upon Tok2Vec and uses parametric attention
to attend to tokens that are relevant to text classification.

spacy.TextCatReduce.v1 {id="TextCatReduce"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy.TextCatReduce.v1"
exclusive_classes = false
use_reduce_first = false
use_reduce_last = false
use_reduce_max = false
use_reduce_mean = true
nO = null

[model.tok2vec]
@architectures = "spacy.HashEmbedCNN.v2"
pretrained_vectors = null
width = 96
depth = 4
embed_size = 2000
window_size = 1
maxout_pieces = 3
subword_features = true

A classifier that pools token hidden representations of each Doc using first,
max or mean reduction and then applies a classification layer. Reductions are
concatenated when multiple reductions are used.

TextCatReduce is a generalization of the older
TextCatCNN model. TextCatCNN always uses a mean
reduction, whereas TextCatReduce also supports first/max reductions.

Span classification architectures {id="spancat",source="spacy/ml/models/spancat.py"} Copy Chapter

spacy.SpanCategorizer.v1 {id="SpanCategorizer"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy.SpanCategorizer.v1"
scorer = {"@layers": "spacy.LinearLogistic.v1"}

[model.reducer]
@layers = spacy.mean_max_reducer.v1"
hidden_size = 128

[model.tok2vec]
@architectures = "spacy.Tok2Vec.v1"

[model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v1"
# ...

[model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v1"
# ...

Build a span categorizer model to power a
SpanCategorizer component, given a token-to-vector
model, a reducer model to map the sequence of vectors for each span down to a
single vector, and a scorer model to map the vectors to probabilities.

spacy.mean_max_reducer.v1 {id="mean_max_reducer"} Copy Section

Reduce sequences by concatenating their mean and max pooled vectors, and then
combine the concatenated vectors with a hidden layer.

Entity linking architectures {id="entitylinker",source="spacy/ml/models/entity_linker.py"} Copy Chapter

An EntityLinker component disambiguates textual mentions
(tagged as named entities) to unique identifiers, grounding the named entities
into the "real world". This requires 3 main components:

• A KnowledgeBase (KB) holding the unique identifiers, potential
  synonyms and prior probabilities.
• A candidate generation step to produce a set of likely identifiers, given a
  certain textual mention.
• A machine learning Model that picks the
  most plausible ID from the set of candidates.

spacy.EntityLinker.v2 {id="EntityLinker"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy.EntityLinker.v2"
nO = null

[model.tok2vec]
@architectures = "spacy.HashEmbedCNN.v2"
pretrained_vectors = null
width = 96
depth = 2
embed_size = 2000
window_size = 1
maxout_pieces = 3
subword_features = true

The EntityLinker model architecture is a Thinc Model with a
Linear output layer.

spacy.EmptyKB.v1 {id="EmptyKB.v1"} Copy Section

A function that creates an empty KnowledgeBase from a Vocab
instance.

spacy.EmptyKB.v2 {id="EmptyKB"} Copy Section

A function that creates an empty KnowledgeBase from a Vocab
instance. This is the default when a new entity linker component is created. It
returns a Callable[[Vocab, int], InMemoryLookupKB].

spacy.KBFromFile.v1 {id="KBFromFile"} Copy Section

A function that reads an existing KnowledgeBase from file.

spacy.CandidateGenerator.v1 {id="CandidateGenerator"} Copy Section

A function that takes as input a KnowledgeBase and a
Span object denoting a named entity, and returns a list of
plausible Candidate objects. The default
CandidateGenerator uses the text of a mention to find its potential aliases in
the KnowledgeBase. Note that this function is case-dependent.

spacy.CandidateBatchGenerator.v1 {id="CandidateBatchGenerator"} Copy Section

A function that takes as input a KnowledgeBase and an Iterable of
Span objects denoting named entities, and returns a list of
plausible Candidate objects per specified
Span. The default CandidateBatchGenerator uses the text of a
mention to find its potential aliases in the KnowledgeBase. Note that this
function is case-dependent.

Coreference {id="coref-architectures",tag="experimental"} Copy Chapter

A CoreferenceResolver component identifies tokens that refer to
the same entity. A SpanResolver component infers spans
from single tokens. Together these components can be used to reproduce
traditional coreference models. You can also omit the SpanResolver if working
with only token-level clusters is acceptable.

spacy-experimental.Coref.v1 {id="Coref",tag="experimental"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy-experimental.Coref.v1"
distance_embedding_size = 20
dropout = 0.3
hidden_size = 1024
depth = 2
antecedent_limit = 50
antecedent_batch_size = 512

[model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "transformer"
pooling = {"@layers":"reduce_mean.v1"}

The Coref model architecture is a Thinc Model.

spacy-experimental.SpanResolver.v1 {id="SpanResolver",tag="experimental"} Copy Section

Example Config

ini Copy

[model]
@architectures = "spacy-experimental.SpanResolver.v1"
hidden_size = 1024
distance_embedding_size = 64
conv_channels = 4
window_size = 1
max_distance = 128
prefix = "coref_head_clusters"

[model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "transformer"
pooling = {"@layers":"reduce_mean.v1"}

The SpanResolver model architecture is a Thinc Model. Note that
MentionClusters is List[List[Tuple[int, int]]].

File: website/docs/api/attributeruler.mdx Copy Chapter

title: AttributeRuler tag: class source: spacy/pipeline/attributeruler.py version: 3 teaser: 'Pipeline component for rule-based token attribute assignment' api_string_name: attribute_ruler api_trainable: false Copy Chapter

The attribute ruler lets you set token attributes for tokens identified by
Matcher patterns. The attribute ruler is
typically used to handle exceptions for token attributes and to map values
between attributes such as mapping fine-grained POS tags to coarse-grained POS
tags. See the usage guide for
examples.

Config and implementation {id="config"} Copy Chapter

The default config is defined by the pipeline component factory and describes
how the component should be configured. You can override its settings via the
config argument on nlp.add_pipe or in your
config.cfg for training.

Example

python Copy

config = {"validate": True}
nlp.add_pipe("attribute_ruler", config=config)

%%GITHUB_SPACY/spacy/pipeline/attributeruler.py

AttributeRuler.\_\_init\_\_ {id="init",tag="method"} Copy Chapter

Initialize the attribute ruler.

Example

python Copy

# Construction via add_pipe
ruler = nlp.add_pipe("attribute_ruler")

AttributeRuler.\_\_call\_\_ {id="call",tag="method"} Copy Chapter

Apply the attribute ruler to a Doc, setting token attributes for tokens
matched by the provided patterns.

AttributeRuler.add {id="add",tag="method"} Copy Chapter

Add patterns to the attribute ruler. The patterns are a list of Matcher
patterns and the attributes are a dict of attributes to set on the matched
token. If the pattern matches a span of more than one token, the index can be
used to set the attributes for the token at that index in the span. The index
may be negative to index from the end of the span.

Example

python Copy

ruler = nlp.add_pipe("attribute_ruler")
patterns = [[{"TAG": "VB"}]]
attrs = {"POS": "VERB"}
ruler.add(patterns=patterns, attrs=attrs)

AttributeRuler.add_patterns {id="add_patterns",tag="method"} Copy Chapter

Example

python Copy

ruler = nlp.add_pipe("attribute_ruler")
patterns = [
  {
    "patterns": [[{"TAG": "VB"}]], "attrs": {"POS": "VERB"}
  },
  {
    "patterns": [[{"LOWER": "two"}, {"LOWER": "apples"}]],
    "attrs": {"LEMMA": "apple"},
    "index": -1
  },
]
ruler.add_patterns(patterns)

Add patterns from a list of pattern dicts. Each pattern dict can specify the
keys "patterns", "attrs" and "index", which match the arguments of
AttributeRuler.add.

AttributeRuler.patterns {id="patterns",tag="property"} Copy Chapter

Get all patterns that have been added to the attribute ruler in the
patterns_dict format accepted by
AttributeRuler.add_patterns.

AttributeRuler.initialize {id="initialize",tag="method"} Copy Chapter

Initialize the component with data and used before training to load in rules
from a file. This method is typically called by
Language.initialize and lets you customize
arguments it receives via the
[initialize.components] block in the
config.

Example

python Copy

ruler = nlp.add_pipe("attribute_ruler")
ruler.initialize(lambda: [], nlp=nlp, patterns=patterns)

ini Copy

### config.cfg
[initialize.components.attribute_ruler]

[initialize.components.attribute_ruler.patterns]
@readers = "srsly.read_json.v1"
path = "corpus/attribute_ruler_patterns.json

AttributeRuler.load_from_tag_map {id="load_from_tag_map",tag="method"} Copy Chapter

Load attribute ruler patterns from a tag map.

AttributeRuler.load_from_morph_rules {id="load_from_morph_rules",tag="method"} Copy Chapter

Load attribute ruler patterns from morph rules.

AttributeRuler.to_disk {id="to_disk",tag="method"} Copy Chapter

Serialize the pipe to disk.

Example

python Copy

ruler = nlp.add_pipe("attribute_ruler")
ruler.to_disk("/path/to/attribute_ruler")

AttributeRuler.from_disk {id="from_disk",tag="method"} Copy Chapter

Load the pipe from disk. Modifies the object in place and returns it.

Example

python Copy

ruler = nlp.add_pipe("attribute_ruler")
ruler.from_disk("/path/to/attribute_ruler")

AttributeRuler.to_bytes {id="to_bytes",tag="method"} Copy Chapter

Example

python Copy

ruler = nlp.add_pipe("attribute_ruler")
ruler = ruler.to_bytes()

Serialize the pipe to a bytestring.

AttributeRuler.from_bytes {id="from_bytes",tag="method"} Copy Chapter

Load the pipe from a bytestring. Modifies the object in place and returns it.

Example

python Copy

ruler_bytes = ruler.to_bytes()
ruler = nlp.add_pipe("attribute_ruler")
ruler.from_bytes(ruler_bytes)

Serialization fields {id="serialization-fields"} Copy Chapter

During serialization, spaCy will export several data fields used to restore
different aspects of the object. If needed, you can exclude them from
serialization by passing in the string names via the exclude argument.

Example

python Copy

data = ruler.to_disk("/path", exclude=["vocab"])

File: website/docs/api/attributes.mdx Copy Chapter

title: Attributes teaser: Token attributes source: spacy/attrs.pyx Copy Chapter

Token attributes are specified using internal IDs in many places
including:

• Matcher patterns,
• Doc.to_array and
  Doc.from_array
• Doc.has_annotation
• MultiHashEmbed Tok2Vec architecture
  attrs

python Copy

import spacy
from spacy.attrs import DEP

nlp = spacy.blank("en")
doc = nlp("There are many attributes.")

# DEP always has the same internal value
assert DEP == 76

# "DEP" is automatically converted to DEP
assert DEP == nlp.vocab.strings["DEP"]
assert doc.has_annotation(DEP) == doc.has_annotation("DEP")

# look up IDs in spacy.attrs.IDS
from spacy.attrs import IDS
assert IDS["DEP"] == DEP

All methods automatically convert between the string version of an ID ("DEP")
and the internal integer symbols (DEP). The internal IDs can be imported from
spacy.attrs or retrieved from the StringStore. A map
from string attribute names to internal attribute IDs is stored in
spacy.attrs.IDS.

The corresponding Token object attributes can be
accessed using the same names in lowercase, e.g. token.orth or token.length.
For attributes that represent string values, the internal integer ID is accessed
as Token.attr, e.g. token.dep, while the string value can be retrieved by
appending _ as in token.dep_.