CoreNLP Documentation

Technical Documentation: stanfordnlp/CoreNLP

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md)

Stanford CoreNLP

Stanford CoreNLP provides a set of natural language analysis tools written in Java. It can take raw human language text input and give the base forms of words, their parts of speech, whether they are names of companies, people, etc., normalize and interpret dates, times, and numeric quantities, mark up the structure of sentences in terms of syntactic phrases or dependencies, and indicate which noun phrases refer to the same entities. It was originally developed for English, but now also provides varying levels of support for (Modern Standard) Arabic, (mainland) Chinese, French, German, Hungarian, Italian, and Spanish. Stanford CoreNLP is an integrated framework, which makes it very easy to apply a bunch of language analysis tools to a piece of text. Starting from plain text, you can run all the tools with just two lines of code. Its analyses provide the foundational building blocks for higher-level and domain-specific text understanding applications. Stanford CoreNLP is a set of stable and well-tested natural language processing tools, widely used by various groups in academia, industry, and government. The tools variously use rule-based, probabilistic machine learning, and deep learning components.

The Stanford CoreNLP code is written in Java and licensed under the GNU General Public License (v2 or later). Note that this is the full GPL, which allows many free uses, but not its use in proprietary software that you distribute to others.

Build Instructions

Several times a year we distribute a new version of the software, which corresponds to a stable commit.

During the time between releases, one can always use the latest, under development version of our code.

Here are some helpful instructions to use the latest code:

Provided build

Sometimes we will provide updated jars here which have the latest version of the code.

At present, the current released version of the code is our most recent released jar, though you can always build the very latest from GitHub HEAD yourself.

Build with Ant

Make sure you have Ant installed, details here: http://ant.apache.org/
Compile the code with this command: cd CoreNLP ; ant
Then run this command to build a jar with the latest version of the code: cd CoreNLP/classes ; jar -cf ../stanford-corenlp.jar edu
This will create a new jar called stanford-corenlp.jar in the CoreNLP folder which contains the latest code
The dependencies that work with the latest code are in CoreNLP/lib and CoreNLP/liblocal, so make sure to include those in your CLASSPATH.
When using the latest version of the code make sure to download the latest versions of the corenlp-models, english-models, and english-models-kbp and include them in your CLASSPATH. If you are processing languages other than English, make sure to download the latest version of the models jar for the language you are interested in.

Build with Maven

Make sure you have Maven installed, details here: https://maven.apache.org/
If you run this command in the CoreNLP directory: mvn package , it should run the tests and build this jar file: CoreNLP/target/stanford-corenlp-4.6.0.jar
When using the latest version of the code make sure to download the latest versions of the corenlp-models, english-extra-models, and english-kbp-models and include them in your CLASSPATH. If you are processing languages other than English, make sure to download the latest version of the models jar for the language you are interested in.
If you want to use Stanford CoreNLP as part of a Maven project you need to install the models jars into your Maven repository. Below is a sample command for installing the Spanish models jar. For other languages just change the language name in the command. To install stanford-corenlp-models-current.jar you will need to set -Dclassifier=models. Here is the sample command for Spanish: mvn install:install-file -Dfile=/location/of/stanford-spanish-corenlp-models-current.jar -DgroupId=edu.stanford.nlp -DartifactId=stanford-corenlp -Dversion=4.6.0 -Dclassifier=models-spanish -Dpackaging=jar

Models

The models jars that correspond to the latest code can be found in the table below.

Some of the larger (English) models -- like the shift-reduce parser and WikiDict -- are not distributed with our default models jar.
These require downloading the English (extra) and English (kbp) jars. Resources for other languages require usage of the corresponding
models jar.

The best way to get the models is to use git-lfs and clone them from Hugging Face Hub.

For instance, to get the French models, run the following commands:

text

# Make sure you have git-lfs installed
# (https://git-lfs.github.com/)
git lfs install

git clone https://huggingface.co/stanfordnlp/corenlp-french

The jars can be directly downloaded from the links below or the Hugging Face Hub page as well.

Language	Model Jar	Last Updated
Arabic	download (HF Hub)	4.6.0
Chinese	download (HF Hub)	4.6.0
English (extra)	download (HF Hub)	4.6.0
English (KBP)	download (HF Hub)	4.6.0
French	download (HF Hub)	4.6.0
German	download (HF Hub)	4.6.0
Hungarian	download (HF Hub)	4.6.0
Italian	download (HF Hub)	4.6.0
Spanish	download (HF Hub)	4.6.0

Thank you to Hugging Face for helping with our hosting!

Install by Gradle

If you don't know Gradle itself, see official site: https://gradle.org

Write the following in your build.gradle according to Maven Central:

gradle

dependencies {
    implementation 'edu.stanford.nlp:stanford-corenlp:4.5.5'
}

If you want to analyse English, add following:

gradle

implementation "edu.stanford.nlp:stanford-corenlp:4.5.5:models"
    implementation "edu.stanford.nlp:stanford-corenlp:4.5.5:models-english"
    implementation "edu.stanford.nlp:stanford-corenlp:4.5.5:models-english-kbp"

If you use another version, replace "4.5.5" to a version you use.

Useful resources

You can find releases of Stanford CoreNLP on Maven Central.

You can find more explanation and documentation on the Stanford CoreNLP homepage.

For information about making contributions to Stanford CoreNLP, see the file CONTRIBUTING.md.

Questions about CoreNLP can either be posted on StackOverflow with the tag stanford-nlp,
or on the mailing lists.

Doc Classify README (`doc/classify/README.txt`)

Stanford Classifier v4.2.0 - 2020-11-17

Original core classifier code and command line interface by Dan Klein
and Chris Manning. Support code, additional features, etc. by
Kristina Toutanova, Jenny Finkel, Galen Andrew, Joseph Smarr, Chris
Cox, Roger Levy, Rajat Raina, Pi-Chuan Chang, Marie-Catherine de
Marneffe, Eric Yeh, Anna Rafferty, and John Bauer. This release
prepared by John Bauer.

This package contains a maximum entropy classifier.

For more information about the classifier, point a web browser at the included javadoc directory, starting at the Package page for the edu.stanford.nlp.classify package, and looking also at the ColumnDataClassifier class documentation therein.

This software requires Java 8 (JDK 1.8.0+). (You must have installed it
separately. Check the command "java -version".)

QUICKSTART

COMMAND LINE INTERFACE
To classify the included example dataset cheeseDisease (in the examples directory), type the following at the command line while in the main classifier directory:

java -cp "*:." edu.stanford.nlp.classify.ColumnDataClassifier -prop examples/cheese2007.prop

This will classify the included test data, cheeseDisease.test, based on the probability that each example is a cheese or a disease, as calculated by a linear classifier trained on cheeseDisease.train.

The cheese2007.prop file demonstrates how features are specified. The first feature in the file, useClassFeature,
indicates that a feature should be used based on class frequency in the training set. Most other features are
calculated on specific columns of data in your tab-delimited text file. For example, "1.useNGrams=true" indicates
that n-gram features should be created for the values in column 1 (numbering begins at 0!). Note that you must
specify, for example, "true" in "1.useNGrams=true"; "1.useNGrams" alone will not cause n-gram features to be created.
N-gram features are character subsequences of the string in the column, for example, "t", "h", "e", "th", "he",
"the" from the word "the". You can also specify various other kinds of features such as just using the string value
as a categorical feature (1.useString=true) or splitting up a longer string into bag-of-words features
(1.splitWordsRegexp=[ ] 1.useSplitWords=true). The prop file also allows a choice of printing and optimization
options, and allows you to specify training and test files (e.g., in cheese2007.prop under the "Training input"
comment). See the javadoc for ColumnDataClassifier within the edu.stanford.nlp.classify package for more information
on these and other options.

Another included dataset is the iris dataset which uses numerical features to separate types of irises. To specify the use of a real-valued rather than categorical feature, you can use one or more of "realValued", "logTransform", or "logitTransform" for a given column. "realValued" adds the number in the given column as a feature value, while the transform options perform either a log or a logit transform on the value first. The format of these feature options is the same as for categorical features; for instance, iris2007.prop shows the use of real valued features such as "2.realValued=true".

CLASSIFYING YOUR OWN DATA FILES
To classify your own data files, they should be in tab-delimited text from which to make features as shown above, SVMLight format, or as tab-delimited text with the exact feature values you would like. Then specify the train and test files on the command line or in a .prop file with "trainFile=/myPath/myTrainFile.train" and "testFile==/myPath/myTestFile.test". You can also create a serialized classifier using the serializeTo option followed by a file path.

CODE EXAMPLES
You can also directly use the classes in this package to train classifiers within other programs. An example of this is shown in ClassifierExample, in the package edu.stanford.nlp.classify. This class demonstrates how to build a classifier factory, creating a classifier and setting various parameters in the classifier, training the classifier, and finally testing the classifier on a different data set.

NO GUI
This package does not provide a graphical user interface. The
classifier is accessible only via the command line or programmatically.

LICENSE

// Stanford Classifier
// Copyright (c) 2003-2007 The Board of Trustees of
// The Leland Stanford Junior University. All Rights Reserved.
//
// This program is free software; you can redistribute it and/or
// modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation; either version 2
// of the License, or (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see http://www.gnu.org/licenses/ .
//
// For more information, bug reports, fixes, contact:
// Christopher Manning
// Dept of Computer Science, Gates 2A
// Stanford CA 94305-9020
// USA
// [email protected]
// https://nlp.stanford.edu/software/classifier.html

CHANGES

2020-11-17 4.2.0 Update for compatibility

2020-05-10 4.0.0 Update for compatibility

2018-10-16 3.9.2 Update for compatibility

2018-02-27 3.9.1 Updated for compatibility

2016-10-31 3.7.0 Update for compatibility

2015-12-09 3.6.0 Update for compatibility

2015-04-20 3.5.2 Update for compatibility

2015-01-29 3.5.1 New input/output options, support for GloVe
word vectors

2014-10-26 3.5.0 Upgrade to Java 1.8

2014-08-27 3.4.1 Update for compatibility

2014-06-16 3.4 Update for compatibility

2014-01-04 3.3.1 Bugfix release

2013-11-12 3.3.0 Update for compatibility

2013-06-19 3.2.0 Update for compatibility

2013-04-04 2.1.8 Update to maintain compatibility

2012-11-11 2.1.7 new pair-of-words features

2012-07-09 2.1.6 Minor bug fixes

2012-05-22 2.1.5 Re-release to maintain compatibility
with other releases

2012-03-09 2.1.4 Bugfix for svmlight format

2011-12-16 2.1.3 Re-release to maintain compatibility
with other releases

2011-09-14 2.1.2 Change ColumnDataClassifier to be an object
with API rather than static methods;
ColumnDataClassifier thread safe

2011-06-15 2.1.1 Re-release to maintain compatibility
with other releases

2011-05-15 2.1 Updated with more documentation

2007-08-15 2.0 New command line interface, substantial
increase in options and features
(updated on 2007-09-28 with a bug fix)

2003-05-26 1.0 Initial release

Doc Corenlp README (`doc/corenlp/README.txt`)

Stanford CoreNLP - Stanford's Suite of NLP Tools

DOCUMENTATION

Please look at the URL below for documentation for Stanford CoreNLP:

https://nlp.stanford.edu/software/corenlp.html

LICENSE

//
// StanfordCoreNLP -- a suite of NLP tools
// Copyright © 2009-2020 The Board of Trustees of
// The Leland Stanford Junior University. All Rights Reserved.
//
// This program is free software; you can redistribute it and/or
// modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation; either version 2
// of the License, or (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see http://www.gnu.org/licenses/ .
//
// For more information, bug reports, fixes, contact:
// Christopher Manning
// Dept of Computer Science, Gates 2A
// Stanford CA 94305-9020
// USA
//

CHANGES

2026-09-24 4.6.0 Security fixes, enhanced-graph Semgrex
queries, SemgrexStats, SUTime bugfixes, and
dependency upgrades

2025-06-06 4.5.10 Additional Semgrex / SSurgeon features and
remove patterns/lucene because of security
warning

2025-03-23 4.5.9 Additional Semgrex / SSurgeon features and a
security patch recommended for deserialization

2024-12-11 4.5.8 Update paths to reflect an internal file
server dying. Add brat demo files to distrib
to reduce dependencies on nlp.stanford.edu.
Update the con->dep converter for UD. Update
some external libraries for security
vulnerabilities

2024-04-18 4.5.7 Add the f1 scores to the parser output,
improve the dependency converter

2024-01-31 4.5.6 Minor lemmatizer and tokenizer fixes

2023-09-06 4.5.5 Fix up some SD and UD conversion errors. Add
SceneGraph to the server. Fix Tregex optional
bug. "fourty" and forty (40) days in SUTime.

2023-03-15 4.5.4 Minor updates to Ssurgeon, include the
Morphology CLI

2023-03-10 4.5.3 Fix the collinizer. Add an Ssurgeon interface

2023-01-19 4.5.2 Update package dependencies to avoid security
problems. Add protobuf CLI tools for use in
Python or elsewhere

2022-08-29 4.5.1 Bugfixes to tokenizer and semgrex

2022-07-21 4.5.0 Tokenizer and lemmatizer upgrades, along with
a new tsurgeon operation and some bugfixes

2022-01-20 4.4.0 Fix issue with Italian depparse, tsurgeon CLI,
fix security issues, bug fixes

2021-10-14 4.3.1 Minor bugfixes to German umlaut tokenization,
CDC tokenizer offset, Hungarian model package

2021-09-26 4.3.0 Add trained tokenizer from corenlp-it, add
Italian and Hungarian pipelines using data
from FBK, UD, Szeged, NYTK, and SPMRL. Better
emoji support in the PTB tokenizer

2021-05-14 4.2.2 Fix issue with demo.

2021-05-05 4.2.1 Fix Turkish locale bug, QuoteAnnotator crash
fixes, smaller srparser models, improvements
to enhanced UD converter, Updated dependencies
(istack, protobuf), batch processing of
semgrex & enhancer requests when using stanza

2020-11-16 4.2.0 Bug fixes, Retrained English parser models
with improved trees, Updated dependencies
(ejml, junit, jflex), Speed up loading
Wikidict annotator, New features for server
handling of tokensregex and tregex requests,
Release built directly from GitHub repo

2020-07-31 4.1.0 Improved server interface, improved memory
usage of sutime, spanish tokenization upgrades

2020-04-19 4.0.0 Changed to UDv2 tokenization ("new" LDC Treebank,
for English); handles multi-word-tokens;
improved UDv2-based taggers and parsers for
English, French, German, Spanish; new French NER;
new Chinese segmenter; library updates, bug fixes

2018-10-05 3.9.2 improved NER pipeline and entity mention
confidences; support for Java 11; new POS
models for English; 4 methods for setting
document dates; tokenizer improvements;
CoreNLP runs as filter from stdin to stdout;
bug fixes

2018-02-27 3.9.1 Bug fixes, minor enhancements

2018-01-31 3.9.0 Spanish KBP and new dependency parse model,
wrapper API for data, quote attribution
improvements, easier use of coref info, bug
fixes

2017-06-09 3.8.0 Web service annotator, discussion forum
handling, new French and Spanish models

2016-10-31 3.7.0 KBP Annotator, improved coreference, Arabic
pipeline

2015-12-09 3.6.0 Improved coreference, OpenIE integration,
Stanford CoreNLP server

2015-04-20 3.5.2 Switch to Universal dependencies, add Chinese
coreference system to CoreNLP

2015-01-29 3.5.1 NER, dependency parser, SPIED improvements;
general bugfixes

2014-10-26 3.5.0 Upgrade to Java 1.8; add annotators for
dependency parsing and relation extraction

2014-08-27 3.4.1 Add Spanish models

2014-06-16 3.4 Add shift reduce parser

2014-01-04 3.3.1 Bugfix release

2013-11-12 3.3.0 Add sentiment model, minor sutime improvements

2013-06-19 3.2.0 New RNN parser model, more efficient tagger

2013-04-04 1.3.5 Speed improvements, coref improvements,
Chinese version, -nthreads option

2012-11-12 1.3.4 Improved ner model and dependency code,
now possible to change annotator pool for
later StanfordCoreNLP objects

2012-07-09 1.3.3 Minor bug fixes

2012-05-22 1.3.2 Improvements to sutime

2012-03-09 1.3.1 Now supports caseless models (available as DLC)

2011-12-16 1.3.0 Threadsafe!
Bugs in time annotation fixed

2011-09-14 1.2.0 Time expression recognizer added to ner annotator
Output bugfixes
Parser can now substitute for tagger

2011-06-19 1.1.0 Improved coref release

2011-05-15 1.0.4 More efficient dcoref data structure
Supports already-tokenized input text

2011-04-17 1.0.3 Compatible with other releases
Support loading arbitrary annotators
Tagger bug fixes, such as "EOS" token

2010-11-11 1.0.2 Remove wn.jar

2010-11-11 1.0.1 Add xml removal

2010-10-07 1.0 Initial release

Doc Corenlp Input (`doc/corenlp/input.txt`)

Stanford University is located in California. It is a great university, founded in 1891.

Doc Lexparser README (`doc/lexparser/README.txt`)

Stanford Lexicalized Parser v4.2.1 - 2021-05-05

Original core parser code by Dan Klein. Support code, additional
modules, languages, features, internationalization, compaction, typed
dependencies, etc. by Christopher Manning, Roger Levy, Teg Grenager,
Galen Andrew, Marie-Catherine de Marneffe, Jenny Finkel, Spence Green,
Bill MacCartney, Anna Rafferty, Huihsin Tseng, Pi-Chuan Chang,
Wolfgang Maier, Richard Eckart, Richard Socher, John Bauer,
Sebastian Schuster, and Jon Gauthier.

This release was prepared by Jason Bolton.

This package contains 6 parsers: a high-accuracy unlexicalized PCFG; a
lexicalized dependency parser; a factored model, where the estimates
of dependencies and an unlexicalized PCFG are jointly optimized to
give a lexicalized PCFG treebank parser; a TreeRNN parser, where
recursive neural networks trained with semantic word vectors are used
to score parse trees; a Shift-Reduce Constituency Parser;
and a transition-based neural dependency parser.
Also included are grammars for various languages for use with these parsers.

For more information about the parser API, point a web browser at the
included javadoc directory (use the browser's Open File command to open
the index.html file inside the javadoc folder). Start by looking at the
Package page for the edu.stanford.nlp.parser.lexparser package, and then
look at the page for the LexicalizedParser class documentation therein,
particularly documentation of the main method.

Secondly, you should also look at the Parser FAQ on the web:

text

https://nlp.stanford.edu/software/parser-faq.html

This software requires Java 8 (JDK 1.8.0+). (You must have installed it
separately. Check that the command "java -version" works and gives 1.8+.)

QUICKSTART

UNIX COMMAND-LINE USAGE

On a Unix system you should be able to parse the English test file with the
following command:

text

./lexparser.sh data/testsent.txt

This uses the PCFG parser, which is quick to load and run, and quite accurate.

[Notes: it takes a few seconds to load the parser data before parsing
begins; continued parsing is quicker. To use the lexicalized parser, replace
englishPCFG.ser.gz with englishFactored.ser.gz in the lexparser.sh script
and use the flag -mx600m to give more memory to java.]

WINDOWS GUI USAGE

On a Windows system, assuming that java is on your PATH, you should be able
to run a parsing GUI by double-clicking on the lexparser-gui.bat icon,
or giving the command lexparser-gui in this directory from a command prompt.

Click Load File, Browse, and navigate to and select testsent.txt in
the top directory of the parser distribution. Click Load Parser,
Browse, and select the models jar, also in the top directory of the
parser distribution. From the models jar, select englishPCFG.ser.gz.
Click Parse to parse the first sentence.

NEURAL NETWORK DEPENDENCY PARSER USAGE

To use the neural net dependency parser, issue the following command:

text

java -Xmx2g -cp "*" edu.stanford.nlp.parser.nndep.DependencyParser \
-model edu/stanford/nlp/models/parser/nndep/english_UD.gz \
-textFile data/english-onesent.txt -outFile data/english-onesent.txt.out

The output will be written to data/english-onesent.txt.out

If you want to run on a language other than English, you will need to use
a language specific POS tagger. Here is an example for Chinese:

text

java -Xmx2g -cp "*" edu.stanford.nlp.parser.nndep.DependencyParser \
-model edu/stanford/nlp/models/parser/nndep/UD_Chinese.gz \
-tagger.model edu/stanford/nlp/models/pos-tagger/chinese-distsim.tagger \
-textFile data/chinese-onesent-utf8.txt -outFile data/chinese-onesent-utf8.txt.out

OTHER USE CASES

The GUI is also available under Unix:

text

lexparser-gui.sh

Under Mac OS X, you can double-click on lexparser-gui.command to invoke the
GUI. The command-line version works on all platforms. Use lexparser.bat
to run it under Windows. The GUI is only for exploring the parser. It does
not allow you to save output. You need to use the command-line program or
programmatic API to do serious work with the parser.

ADDITIONAL GRAMMARS

The parser is supplied with several trained grammars. There are English
grammars based on the standard LDC Penn Treebank WSJ training sections 2-21
(wsj*), and ones based on an augmented data set, better for questions,
commands, and recent English and biomedical text (english*).

All grammars are located in the included models jar. (If you'd like to have
grammar files like in older versions of the parser, you can get them by
extracting them from the jar file with the 'jar -xf' command.)

MULTILINGUAL PARSING
In addition to the English grammars, the parser comes with trained grammars
for Arabic, Chinese, French, and German. To parse with these grammars, run

text

lexparser-lang.sh

with no arguments to see usage instructions. You can change language-specific
settings passed to the parser by modifying lexparser_lang.def.

You can also train and evaluate new grammars using:

text

lexparser-lang-train-test.sh

To see how we trained the grammars supplied in this distribution, see

text

bin/makeSerialized.csh

You will not be able to run this script (since it uses Stanford-specific file
paths), but you should be able to see what we did.

Arabic

Trained on parts 1-3 of the Penn Arabic Treebank (ATB) using the
pre-processing described in (Green and Manning, 2010). The default input
encoding is UTF-8 Arabic script. You can convert text in Buckwalter encoding to UTF-8
with the package edu.stanford.nlp.international.arabic.Buckwalter which is included
in stanford-parser.jar.

The parser requires segmentation and tokenization of raw text per the ATB standard
prior to parsing. You can generate this segmentation and tokenization with the Stanford
Word Segmenter, which is available separately at:

https://nlp.stanford.edu/software/segmenter.html

Chinese

There are Chinese grammars trained just on mainland material from
Xinhua and more mixed material from the LDC Chinese Treebank. The default
input encoding is GB18030.

French

The standalone parser distribution comes with a neural dependency parser
model trained on the French-GSD data set (version 2.2). The standalone
parser distribution does not include functionality for producing UD 2.2
tokenization, so pre-tokenized text (text tokenized by whitespace) must
be provided when running the neural dependency parser, and the "-tokenized"
flag must be used.

Example command:

text

java -Xmx2g -cp "*" edu.stanford.nlp.parser.nndep.DependencyParser \
-model edu/stanford/nlp/models/parser/nndep/UD_French.gz \
-tagger.model edu/stanford/nlp/models/pos-tagger/french-ud.tagger \
-tokenized -textFile example.txt -outFile example.txt.out

Note that "example.txt" should contain UD 2.2 tokens, separated by whitespace.

The only provided French constituency parser is a shift-reduce parser. At this
time running the shift-reduce parser on French text requires running a pipeline
with the full Stanford CoreNLP package.

To use the shift-reduce constituency parser on text and the UD 2.2 tokenization,
upgrade to the full Stanford CoreNLP package and run a French pipeline.

German

The constituency parser was trained on the Negra corpus. Details are included in
(Rafferty and Manning, 2008). This parser expects UD 2.2 tokenization. Input
text files must be UD 2.2 tokens separated by whitespace. The "-tokenized" flag
must be used.

The neural dependency parser was trained on the German-GSD data set (version 2.2).
The standalone parser distribution does not include functionality for producing
UD 2.2 tokenization, so pre-tokenized text (text tokenized by whitespace) must
be provided when running the neural dependency parser, and the "-tokenized"
flag must be used.

Example command:

text

java -Xmx2g -cp "*" edu.stanford.nlp.parser.nndep.DependencyParser \
-model edu/stanford/nlp/models/parser/nndep/UD_German.gz \
-tagger.model edu/stanford/nlp/models/pos-tagger/german-ud.tagger \
-tokenized -textFile example.txt -outFile example.txt.out

German shift reduce parsers are available, but running them on input German text
requires the use of a full Stanford CoreNLP pipeline. The lower accuracy lexicalized
parsers have options for running on input text.

To use the shift-reduce constituency parser on text and the UD 2.2 tokenization, upgrade
to the full Stanford CoreNLP package and run a German pipeline.

Spanish

The constituency parser was trained on the Spanish AnCora treebank and
LDC provided discussion forum and newswire treebanks. This parser expects UD 2.0
tokenization.

The neural dependency parser was trained on the Spanish AnCora data set (version 2.0).
The standalone parser distribution does not include functionality for producing
UD 2.0 tokenization, so pre-tokenized text (text tokenized by whitespace) must
be provided when running the neural dependency parser, and the "-tokenized"
flag must be used.

Example command:

text

java -Xmx2g -cp "*" edu.stanford.nlp.parser.nndep.DependencyParser \
-model edu/stanford/nlp/models/parser/nndep/UD_Spanish.gz \
-tagger.model edu/stanford/nlp/models/pos-tagger/spanish-ud.tagger \
-tokenized -textFile example.txt -outFile example.txt.out

Spanish shift reduce parsers are available, but running them on input Spanish text
requires the use of a full Stanford CoreNLP pipeline. The lower accuracy lexicalized
parsers have options for running on input text.

To use the shift-reduce constituency parser on text and the UD 2.2 tokenization, upgrade
to the full Stanford CoreNLP package and run a Spanish pipeline.

TREEBANK PREPROCESSING

The pre-processed versions of the ATB described
in (Green and Manning, 2010) and the FTB described in (Green et al.,
2011) can be reproduced using the TreebankPreprocessor included in this
release. The configuration files are located in /conf. For example,
to create the ATB data, run:

text

bin/run-tb-preproc -v conf/atb-latest.conf

Note that you'll need to update the conf file paths to your local treebank
distributions as the data is not distributed with the parser. You'll
also need to set the classpath in the cmd_line variable of run-tb-preproc.

The TreebankPreprocessor conf files support various options, which are
documented in

text

edu.stanford.nlp.international.process.ConfigParser

EVALUATION METRICS

The Stanford parser comes with Java implementations of the following
evaluation metrics:

text

Dependency Labeled Attachment

Evalb         (Collins, 1997)
  -Includes per-category evaluation with the -c option

Leaf Ancestor (Sampson and Babarczy, 2003)
  -Both micro- and macro-averaged score

Tagging Accuracy

See the usage instructions and javadocs in the requisite classes located in
edu.stanford.nlp.parser.metrics.

UNIVERSAL DEPENDENCIES vs. STANFORD DEPENDENCIES

Since v3.5.2 the default dependency representation is the new Universal Dependencies
representation. Universal Dependencies were developed with the goal of being a
cross-linguistically valid representation. Note that some constructs such as prepositional
phrases are now analyzed differently and that the set of relations was updated. Please
look at the Universal Dependencies documentation for more information:

text

http://www.universaldependencies.org

The parser also still supports the original Stanford Dependencies representation
as described in the StanfordDependenciesManual.pdf. Use the flag

text

-originalDependencies

to obtain original Stanford Dependencies.

LICENSE

// StanfordLexicalizedParser -- a probabilistic lexicalized NL CFG parser
// Copyright (c) 2002-2020 The Board of Trustees of
// The Leland Stanford Junior University. All Rights Reserved.
//
// This program is free software; you can redistribute it and/or
// modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation; either version 2
// of the License, or (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see http://www.gnu.org/licenses/ .
//
// For more information, bug reports, fixes, contact:
// Christopher Manning
// Dept of Computer Science, Gates 2A
// Stanford CA 94305-9020
// USA
// [email protected]
// https://nlp.stanford.edu/downloads/lex-parser.html

CHANGES

2021-05-05 4.2.1 Reduce size of srparser models

2020-11-17 4.2.0 Retrain English models with treebank fixes

2020-05-22 4.0.0 Model tokenization updated to UDv2.0

2018-10-16 3.9.2 Update for compatibility

2018-02-27 3.9.1 new French and Spanish UD models, misc. UD
enhancements, bug fixes

2017-06-09 3.8.0 Updated for compatibility

2016-10-31 3.7.0 new UD models

2015-12-09 3.6.0 Updated for compatibility

2015-04-20 3.5.2 Switch to universal dependencies

2015-01-29 3.5.1 Dependency parser improvements; general
bugfixes

2014-10-26 3.5.0 Upgrade to Java 1.8; add neural-network
dependency parser

2014-08-27 3.4.1 Add Spanish models

2014-06-16 3.4 Shift-reduce parser

2014-01-04 3.3.1 Bugfix release, dependency improvements

2013-11-12 3.3.0 Remove the attr dependency, add imperatives to
English training data

2013-06-19 3.2.0 New RNN model for WSJ and English with
improved test set accuracy, rel dependency
removed

2013-04-05 2.0.5 Dependency improvements, ctb7 model, -nthreads
option

2012-11-12 2.0.4 Dependency speed improvements; other
dependency changes

2012-07-09 2.0.3 Minor bug fixes

2012-05-22 2.0.2 Supports adding extra data in non-tree format

2012-03-09 2.0.1 Caseless English model added, ready for maven

2012-01-11 2.0.0 Threadsafe!

2011-09-14 1.6.9 Added some imperatives to the English
training data; added root dependency.

2011-06-15 1.6.8 Added French parser and leaf ancestor
evaluation metric; reorganized distribution;
new data preparation scripts; rebuilt grammar
models; other bug fixes

2011-05-15 1.6.7 Minor bug fixes

2011-04-17 1.6.6 Compatible with tagger, corenlp and tregex.

2010-10-30 1.6.5 Further improvements to English Stanford
Dependencies and other minor changes

2010-08-16 1.6.4 More minor bug fixes and improvements to English
Stanford Dependencies and question parsing

2010-07-09 1.6.3 Improvements to English Stanford Dependencies and
question parsing, minor bug fixes

2010-02-25 1.6.2 Improvements to Arabic parser models,
and to English and Chinese Stanford Dependencies

2008-10-19 1.6.1 Slightly improved Arabic, German and
Stanford Dependencies

2007-08-18 1.6 Added Arabic, k-best PCCFG parsing;
improved English grammatical relations

2006-05-30 1.5.1 Improved English and Chinese grammatical relations;
fixed UTF-8 handling

2005-07-20 1.5 Added grammatical relations output;
fixed bugs introduced in 1.4

2004-03-24 1.4 Made PCFG faster again (by FSA minimization);
added German support

2003-09-06 1.3 Made parser over twice as fast;
added tokenization options

2003-07-20 1.2 Halved PCFG memory usage;
added support for Chinese

2003-03-25 1.1 Improved parsing speed; included GUI,
improved PCFG grammar

2002-12-05 1.0 Initial release

Doc Lexparser README Dependencies (`doc/lexparser/README_dependencies.txt`)

UNIVERSAL/STANFORD DEPENDENCIES. Stanford Parser v3.7.0

IMPORTANT: Starting with version 3.5.2 the default dependencies
representation output by the Stanford Parser is the new Universal
Dependencies Representation. Universal Dependencies were developed
with the goal of being a cross-linguistically valid representation.
Note that some constructions such as prepositional phrases are now
analyzed differently and that the set of relations was updated. The
online documentation of English Universal Dependencies at

text

http://www.universaldependencies.org

should be consulted for the current set of dependency relations.

The parser and converter also still support the original
Stanford Dependencies as described in the Stanford Dependencies
manual. Use the flag

text

-originalDependencies

to obtain the original Stanford Dependencies. Note, however, that we
are no longer maintaining the SD converter or representation and we
therefore recommend to use the Universal Dependencies representation
for any new projects.

The manual for the English version of the Stanford Dependencies
representation:

text

StanfordDependenciesManual.pdf

should be consulted for the set of dependency relations in the original
Stanford Dependencies representation and the correct commands for
generating Stanford Dependencies together with any of the Stanford Parser,
another parser, or a treebank.

A typed dependencies representation is also available for Chinese. For
the moment the documentation consists of the code, and a brief
presentation in this paper:

Pi-Chuan Chang, Huihsin Tseng, Dan Jurafsky, and Christopher
D. Manning. 2009. Discriminative Reordering with Chinese Grammatical
Relations Features. Third Workshop on Syntax and Structure in Statistical
Translation. http://nlp.stanford.edu/pubs/ssst09-chang.pdf

DEPENDENCIES SCHEMES

For an overview of the original English Universal Dependencies schemes, please look
at:

Marie-Catherine de Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen,
Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Universal Stanford
dependencies: A cross-linguistic typology. 9th International Conference on
Language Resources and Evaluation (LREC 2014).
http://nlp.stanford.edu/~manning/papers/USD_LREC14_UD_revision.pdf

and

Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič,
Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira,
Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A Multilingual
Treebank Collection. In Proceedings of the Tenth International Conference on Language
Resources and Evaluation (LREC 2016).
http://nlp.stanford.edu/pubs/nivre2016ud.pdf

Please note, though, that some of the relations discussed in the first paper
were subsequently updated and please refer to the online documentation at

text

http://www.universaldependencies.org

for an up to date documention of the set of relations.

For an overview of the enhanced and enhanced++ dependency representations, please look
at:

Sebastian Schuster and Christopher D. Manning. 2016. Enhanced English Universal
Dependencies: An Improved Representation for Natural Language Understanding Tasks.
In Proceedings of the Tenth International Conference on Language Resources and
Evaluation (LREC 2016).
http://nlp.stanford.edu/~sebschu/pubs/schuster-manning-lrec2016.pdf

For an overview of the original typed dependencies scheme, please look
at:

Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D.
Manning. 2006. Generating Typed Dependency Parses from Phrase
Structure Parses. 5th International Conference on Language Resources
and Evaluation (LREC 2006).
http://nlp.stanford.edu/~manning/papers/LREC_2.pdf

For more discussion of the design principles, please see:

Marie-Catherine de Marneffe and Christopher D. Manning. 2008. The
Stanford typed dependencies representation. In Proceedings of the
workshop on Cross-Framework and Cross-Domain Parser Evaluation, pp. 1-8.
http://nlp.stanford.edu/~manning/papers/dependencies-coling08.pdf

These papers can be cited as references for the original English Stanford
Dependencies and Enlgish Universal Dependencies.

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.7.0

Implementation of enhanced and enhanced++ dependency
representations as described in Schuster and Manning (2016).

Fixed concurrency issue.

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.5.2

Switch to Universal Dependencies as the default representation.
Please see the Universal Dependencies documentation at

text

http://www.universaldependencies.org

for more information on the new relations.

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.5.1

A couple of small fixes were made, leading to ccomp and advcl being
recognized in a couple of new environments.

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.4

One major change was made to the dependency taxonomy:

We decided to collapse together the two dependencies partmod and infmod,
since they have similar function and mainly differ in the form of the verbal
head, which is anyways recorded in the POS tag. Those two relations are
removed from the taxonomy, and a new relation vmod covering the union of both
was added.

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.3.1

A couple of fixes/improvements were made in the dependency conversion,
and one change was made to the taxonomy of relations.

The partmod and infmod relations were deleted, and replaced with
vmod for reduced, non-finite verbal modifiers. The distinction between
these two relations can be recovered from the POS tag of the dependent.
A couple of improvements were made to the conversion, the largest
one being recognizing pobj inside a PP not headed by something tagged
as IN or TO.

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.3

Some fixes/improvements were made in the dependency conversion, and one
change was made to the taxonomy of relations.

For currency amount expressions with a currency symbol like "$", it
had previously been the case that "$" was the head, and then each
number word modified it as a number. We realized that this was
unnecessarily inconsistent. For the expression "two thousand dollars",
"dollars" is the head, but "thousand" is a num modifier of it, and
number is used for the parts of a number multi-word expression only.
This analysis is now also used for cases with a currency symbol. E.g.,
"for $ 52.7 million": prep(for, $) num($, million) number(million, 52.7).
Similarly, for "the $ 2.29 billion value", we changed the analysis from
num(value, $) number($, billion) to amod(value, $) num($, billion).
This corresponds to hwat you got for "a two dollar value".
This is actually the most common change (at least on WSJ newswire!).
Remove the attr relation. Some cases disappear by making the question
phrase of WHNP be NP questions the root. Others (predicative NP
complements) become xcomp.
Less aggressive labeling of participial form VPs as xcomp. More of them
are correctly labeled partmod (but occasionally a true xcomp is also
mislabeled as partmod).
Small rule changes to recognize a few more ccomp and parataxis.

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.2, JUNE 2013

An improved dependency conversion means that our dependency trees are
not always projective, one deletion was made from the taxonomy of
relations, and various small converter fixes were made:

rel was removed. rel was originally used as the relation for an
overt relativizer in a relative clause. But it was never a real
grammatical relation, and we gradually started labeling easy cases
as nsubj or dobj. In this release, rel is removed, pobj cases are
also labeled, and the remaining hard cases are labeled as dep.
As a result of correctly labeling a pobj in questions and relative
clauses, the converter now sometimes produces non-projective dependency
trees (ones with crossing dependencies, if the words are laid out in
their normal order in a line, and all dependency arcs are drawn above
them). This is not a bug, it's an improvement in the generated
dependencies, but you should be aware that Stanford Dependencies
trees are now occasionally non-projective. (Some simple dependency
parsing algorithms only produce projective dependency trees.)

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v2.0.5, MARCH 2013

We have begun a more major effort to improve the suitability and coverage of
Stanford Dependencies on less formal text types, and to clean up a couple of
the more quirky dependencies in the original set. These changes are still
ongoing, but in this first installment, we have removed 3 dependencies and
added 2:

abbrev was removed, and is now viewed as just a case of appos.
complm was removed, and is now viewed as just a case of mark.
(This is consistent with an HPSG-like usage of mark.)
purpcl was removed, and is now viewed as just a case of advcl.
discourse was added. The lack of a dependency type for
interjections was an omission even in the early versions, but it
became essential as we expanded our consideration of informal
text types. It is used for interjections, fillers, discourse markers
and emoticons.
goeswith was added. In badly edited text, it is used to join the
two parts of a word.

A few other changes and improvements were also made, including improvements
in the recognition of advcl. There has been a reduction of "dep" dependencies
of about 14% on newswire (and higher on more informal text genres).

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v2.0.4, NOVEMBER 2012

A few minor changes and fixes were made: HYPH is now recognized, and treated
as punctuation and clausal complements of adjectives (including comparatives)
are recognized as ccomp.

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v1.6.9

This version adds an explicit root dependency in the set of dependencies
returned. In the past, there had been no explicit representation of the
root of the sentence in the set of dependencies returned, except in the
CoNLL format output, which always showed the root. Now, there is always
an explicit extra dependency that marks the sentence root, using a fake
ROOT pseudoword with index 0. That is, the root is marked in this way:
root(ROOT-0, depends-3)
Otherwise there were only a couple of minute changes in the dependencies
produced (appositions are now recognized in WHNPs!).

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v1.6.8

This version includes only small fixes, principally addressing some gaps
in the correct treatment of dependencies in inverted sentence (SQ and SINV)
constructions, and some errors in the treatment of copulas in the presence of
temporal NPs.

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- NOVEMBER 2010 - JANUARY 2011

Two changes were made to the taxonomy of dependencies.

measure (phrase modifier) was generalized and replaced by
npadvmod (noun phrase adverbial modifier) which includes measure
phrases and other adverbial uses of noun phrases. Temporal NPs
(tmod) are now a subtype of npadvmod in the dependency hierarchy.
mwe (multi-word expression) is introduced for certain common
function word dependencies for which another good analysis isn't
easy to come by (and which were frequently dep before) such as
"instead of" or "rather than".

A new option has ben added to allow the copula to be treated as
the head when it has an adjective or noun complement.

The conversion software will now work fairly well with the
David Vadas version of the treebank with extra noun phrase
structure. (A few rare cases that are handled with the standard
treebank aren't yet handled, but you will get better dependencies
for compound nouns and multiword adjectival modifiers, etc.)

Considerable improvements were made in the coverage of named
dependencies. You should expect to see only about half as many generic
"dep" dependencies as in version 1.6.4.

CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- JUNE-AUGUST 2010

No new dependency relations have been introduced.

There have been some significant improvements in the generated
dependencies, principally covering:

Better resolution of nsubj and dobj long distance dependencies
(but v1.6.4 fixes the overpercolation of dobj in v1.6.3)
Better handling of conjunction distribution in CCprocessed option
Correction of bug in v1.6.2 that made certain verb dependents noun
dependents.
Better dependencies are generated for question structures (v1.6.4)
Other minor improvements in recognizing passives, adverbial
modifiers, etc.

Doc Lexparser Pos Sentences (`doc/lexparser/pos-sentences.txt`)

People can butter their bread with a knife .
People can butter/VB their bread with a knife .
People can butter/NN their bread with a knife .
People/NNS can/MD butter/VB their/PRP$ bread/NN with/IN a/DT knife/NN ./.
People/NNS can/VB butter/NN their/PRP$ bread/NN with/IN a/DT knife/NN ./.
People/NNS can/NN butter/NN their/PRP$ bread/NN with/IN a/DT knife/NN ./.
People/NN can/NN butter/NN their/NN bread/NN with/NN a/NN knife/NN ./NN

Doc Lexparser Testsent (`doc/lexparser/testsent.txt`)

Scores of properties are under extreme fire threat as a huge blaze
continues to advance through Sydney's north-western suburbs. Fires
have also shut down the major road and rail links between Sydney and
Gosford.

The promotional stop in Sydney was everything to be expected for a
Hollywood blockbuster - phalanxes of photographers, a stretch limo to
a hotel across the Quay - but with one difference. A line-up of
masseurs was waiting to take the media in hand. Never has the term
"massaging the media" seemed so accurate.

Doc Loglinear ARCH (`doc/loglinear/ARCH.txt`)

Architectural Overview of edu.stanford.nlp.loglinear:

The goal of this package is to provide fast, general structure log-linear modelling that's easy to use and to extend.

The package is broken into three parts: model, inference, and learning

Model contains all of the basic storage elements, as well as means to serialize and deserialize for both storage and
network transit. Inference depends on model, and provides an implementation of the clique tree message passing algorithm
for efficient exact inference in tree-structured graphs. Learning depends on inference and model, and provides a simple
interface to efficient multithreaded batch learning, with an implementation of AdaGrad guarded by backtracking.

We will go over model, then inference, then learning.

#####################################################

Model module overview:

#####################################################

ConcatVector:

The key to the speed of loglinear is the ConcatVector class. ConcatVector provides a useful abstraction for NLP machine
learning: a concatenation of vectors, treated as a single vector. The basic idea is to have each feature output a vector,
which are then stored in a ConcatVector (or 'concatenated vector'). When the dot-product is taken of two ConcatVectors,
the result is the sum of the dot product of each of the concatenated components of each vector in sequence. To write
that out explicitly, if a feature ConcatVector f is composed of a number of vector f_i's, and a weight ConcatVector w is
composed of a number of vector w_i's, then dot(f,w) is:

\sum_i dot(f_i, w_i)

This leaves us with two key advantages over a regular vector: each component can be individually tuned for sparsity, and
each component has an isolated namespace and so can an individual feature vector can grow after training begins (say
discovering a new word in a one-hot feature vector), and the weight vector will behave appropriately without hassle.

NDArray

We have a basic NDArray, which allows a standard iterator over possible assignments that creates a lot of int[] arrays
on the heap, and a more elaborate iterator that saves GC by mutating a single array passed over. You'll see this used
throughout the code in hot loops marked by an "//OPTIMIZATION" comment

ConcatVectorTable

ConcatVectorTable is a subclass of NDArray that we use to store factor tables for the log-linear graphical model, where
each element of the table represents the features for one joint assignment to the variables the factor is associated
with. In order to get a factor like you learned about in CS 228, each element of the table is dot-producted with weights.
We don't do this at construction to allow a single set of GraphicalModel objects to be used throughout training.

GraphicalModel

GraphicalModel is a super stripped down implementation of a graphical model. It holds factors, represented by lists of
neighbor indices and a ConcatVectorTable for features. It was deliberate to make all downstream annotations on the model
(like observations for inference or observations for training) go into a HashMap. This is to maintain easy backwards
compatibility with previous serialized versions as features change, and to make life more convenient for downstream
algorithms that may be passing GraphicalModel objects across module or network boundaries, and don't want to create tons
of little 'ride-along' objects that add annotations to the GraphicalModel.

#####################################################

Inference module overview:

#####################################################

TableFactor

This is the traditional 'factor' datatype that you're used to hearing about from Daphne in 228 and "Probabilistic
Graphical Models". It's a subclass of NDArray, and has fast operations for product and marginalize dataflows. It's the
key building block for inference.

CliqueTree

This object takes a GraphicalModel at creation and provides high speed tree-shaped message passing inference for both
exact marginals and exact MAP estimates. It exists as a new object for each GraphicalModel, rather than a static call
for each model, to allow for cacheing some messages when repeated marginals are needed on only slightly changing models.

#####################################################

Learning module overview:

#####################################################

AbstractDifferentiableFunction

This follows the Optimize.jl package convention of providing both gradient and function value in a single return value.

LogLikelihoodFunction

An implementation of AbstractDifferentiableFunction for calculating the log-likelihood of a log-linear model as given by
a GraphicalModel.

AbstractOnlineOptimizer

This is the basic interface for online optimizers to follow. It is sketched out right now, but no implementations have
been made yet.

AbstractBatchOptimizer

There is a fair amount of redundant complexity involved in writing an optimizer that needs to calculate the gradient
on the entire batch of examples every update step. The work between threads must be carefully balanced so that the
time between the first thread finishing, and the last, during which the CPU utilization is far less than 100%, is
minimized. This is managed through rough estimating of the amount of work each item represents, and a perceptron style
updating once the system is running, based on CPU time used for each thread. We also implement a convenience function
here to allow the user to interrupt training early if they are happy with convergence to this point, since that involves
some tricky Java threading to make work.

BacktrackingAdaGradOptimizer

This subclasses AbstractBatchOptimizer, and implements a simple AdaGrad gradient descent guarded by backtracking line
search to maximize an AbstractDifferentiableFunction.

Doc Loglinear OPTIMIZATION (`doc/loglinear/OPTIMIZATION.txt`)

Optimization for loglinear was done last, and driven by realistic benchmarks wherever possible.

There are 2 major benchmarks for loglinear so far:

Training CoNLL linear chain model with a mixture of dense and sparse features:
org.stanford.nlp.loglinear.learning.CoNLLBenchmark
A bunch of microbenchmarks for ConcatVector:
org.stanford.nlp.loglinear.model.ConcatVectorBenchmark

The general findings so far have been:

The JNI doesn't speed up ConcatVector operations, even with AVX assembly, since the JIT seems to vectorize automatically,
and occasionally will give the C code a copy of the arrays to mutate, which is absurdly slow.
The vast majority of training time is spent on feature vector related operations. Message passing takes less than 5%
of total time. The rest of the time is evenly split between the initial dot product with weights to get factor values
and the final summation of feature vectors to get the derivative of the log-likelihood.
Huge heap wastage occurs if you allow the featurizing code to run once and keep around ConcatVector's uselessly, which
results in general slowdowns and long GC waits. Instead keeping featurizing code as thunks is much faster.
Making ConcatVector copy-on-write is a very valuable way to keep the GC from working too hard. A very common case is
a clone vector that is only ever read from, so optimizing for that yielded a 50% drop in GC load.
A non-trivial fraction of time is wasted on poorly balanced work queues for different threads during batch gradient
descent. Balancing more carefully yielded a 10% speedup.

Doc Loglinear QUICKSTART (`doc/loglinear/QUICKSTART.txt`)

loglinear package quickstart:

First, read the ConcatVector section in ARCH.txt.

To jump straight into working code, go read generateSentenceModel() in edu.stanford.nlp.loglinear.CoNLLBenchmark.

#####################################################

Creating and featurizing a GraphicalModel

#####################################################

To construct a GraphicalModel, which you'll need for training and inference, do the following:

GraphicalModel model = new GraphicalModel();

Now, to add a factor to the model, you'll need to know two things: Who are the neighbor variables of this factor, and
how many states do each of those variables possess? As an example, let's add a factor between variable 2 and variable 7,
where variable 2 has 3 states, and variable 7 has 2 states.

int[] neighbors = new int[]{2,7};
int[] neighborSizes = new int[]{3,2}; // This must appear in the same order as neighbors

model.addFactor(neighbors, neighborSizes, (int[] assignment) -> {
// TODO: In the next paragraph we'll discuss featurizing each possible assignment to neighbors
return new ConcatVector(0);

});

You'll also need to know how to featurize your new factors. That happens inside the closure you pass into the factor.
The closure takes as an argument an assignment to the factor (in the same order as neighbors and neighborSizes) and
expects in response a ConcatVector of features.

Make sure your closures are idempotent! The system actually stores these closures, rather than their resulting
ConcatVector's. This is an optimization. The GC in modern JVMs is tuned to collect large numbers of young objects, so
creating new ConcatVectors every gradient calculation and then immediately disposing of them turns out to increase
speed and dramatically decrease heap footprint and GC load.

Here's how to create a feature closure:

int[] neighbors = new int[]{2,7};
int[] neighborSizes = new int[]{3,2}; // This must appear in the same order as neighbors

model.addFactor(neighbors, neighborSizes, (int[] assignment) -> {

text

// This is how assignment[] is structured

int variable2Assignment = assignment[0];
int variable7Assignment = assignment[1];

// Create a new ConcatVector with 2 segments:

ConcatVector features = new ConcatVector();

// Add a dense feature as feature 0, of length 2
// (Dense features in ConcatVector's are mostly used for embeddings)

features.setDenseComponent(0, new double[]{
    variable2Assignment * 2 + variable7Assignment,
    variable2Assignment + variable7Assignment * 2,
});

// Add a sparse (one-hot) feature as feature 1

int sparseIndex = variable2Assignment;
double sparseValue = 1.0;
features.setSparseComponent(1, sparseIndex, sparseValue);

// Return our feature set to complete the closure

return features;

});

And that's all there is to it. Just repeat this several times to populate a GraphicalModel for whatever problem you've
got.

#####################################################

Training a set of weights

#####################################################

Assuming you've got a bunch of GraphicalModel objects, there's not much you need to do to train a system. First, you
need to provide labels to all your variables that the training system understands. To do this, you need to get the
HashMap<String,String> object that represents metadata, and put in labels that the LogLikelihood system understands.
You must label every variable that is mentioned in any factor in your model.

GraphicalModel model = new GraphicalModel();

// Omitted model construction, see previous section

// Tell LogLikelihood to treat variable 2 as having the assignment "1" in training labels

model.getVariableMetadataByReference(2).put(LogLikelihoodFunction.VARIABLE_TRAINING_VALUE, "1");

// Tell LogLikelihood to treat variable 7 as having the assignment "0" in training labels

model.getVariableMetadataByReference(7).put(LogLikelihoodFunction.VARIABLE_TRAINING_VALUE, "0");

Once you've got an array of labeled models, training is pretty straightforward. We create an optimizer, pass it a
LogLikelihoodFunction as the function to optimize, and the array of models as the data to optimize over. That returns
us the optimal set of weights.

GraphicalModel[] trainingSet = //omitted dataset construction;

// Create the optimizer we will use

AbstractBatchOptimizer opt = new BacktrackingAdaGradOptimizer();

// Call the optimizer, with a dataset, a function to optimize, initial weights, and l2 regularization constant

ConcatVector weights = opt.optimize(trainingSet, new LogLikelihoodFunction(), new ConcatVector(0), 0.1);

We can then use these weights for inference.

#####################################################

Inference

#####################################################

Inference is easy once we have a set of weights we want to use. We simply create a CliqueTree for the model we're trying
to optimize and the weights we want, and then ask it for inference results.

GraphicalModel model = // see previous section;
ConcatVector weights = // see first section;

CliqueTree tree = new CliqueTree(model, weights);

int[] mapAssignment = tree.calculateMAP(); double[][] marginalProbabilities = tree.calculateMarginals();

The MAP assignment comes back as an array of assignments, where the assignment for variable 0 is at index 0, variable 1
is at index 1, and so forth. The marginalProbilities array is organized in the same way, except instead of an int for
an assignment, there is an array of doubles, one for each possible assignment for the variable it represents, which
represent global marginals.

Doc Loglinear README (`doc/loglinear/README.txt`)

For an explanation of how everything fits together, see ARCH.txt

For a quick runnable object, go run edu.stanford.nlp.loglinear.CoNLLBenchmark in core's test package.

For a tutorial, see QUICKSTART.txt

For a brief overview of testing for this package (which was quite thorough), see TESTING.txt

Look at OPTIMIZATION.txt for an overview of what exists, what's been done, and what we know about further optimizations.

Doc Loglinear TESTING (`doc/loglinear/TESTING.txt`)

The testing for the loglinear package uses functional invariants over randomly generated inputs, which in general yields
much more durable software.

This is aided by the JUnit port of Quickcheck. The general tactic for testing is to randomly generate inputs, then use
the slow definitional approach on tractably small inputs and test that the output of our algorithms always matches
exactly. The GitHub for the Quickcheck port is https://github.com/pholser/junit-quickcheck. The dependencies for that
are in the lib/ folder in test1.

Some of the general testing approaches are listed below:

message passing -> tested against brute force factor multiplication and marginalization
partition function -> tested against brute force multiplication and summation
log likelihood gradient -> tested against definition of derivative
optimization -> tested by making thousands of random perturbations around the function to check if any values are better
concatVector -> tested against a non-sparse version
table factor -> tested against functional invariants of results, and re-implementations using different algorithms

Doc Ner README (`doc/ner/README.txt`)

Stanford NER - v4.2.0 - 2020-11-17

This package provides a high-performance machine learning based named
entity recognition system, including facilities to train models from
supervised training data and pre-trained models for English.

Original CRF code by Jenny Finkel.
Additional modules, features, internationalization, compaction, and
support code by Christopher Manning, Dan Klein, Christopher Cox, Huy Nguyen
Shipra Dingare, Anna Rafferty, and John Bauer.
This release prepared by Jason Bolton.

LICENSE

The software is licensed under the full GPL v2+. Please see the file LICENCE.txt

For more information, bug reports, and fixes, contact:
Christopher Manning
Dept of Computer Science, Gates 2A
Stanford CA 94305-9020
USA
[email protected]
https://nlp.stanford.edu/software/CRF-NER.html

CONTACT

For questions about this distribution, please contact Stanford's JavaNLP group
at [email protected]. We provide assistance on a best-effort
basis.

TUTORIAL

Quickstart guidelines, primarily for end users who wish to use the included NER
models, are below. For further instructions on training your own NER model,
go to https://nlp.stanford.edu/software/crf-faq.html.

INCLUDED SERIALIZED MODELS / TRAINING DATA

The basic included serialized model is a 3 class NER tagger that can
label: PERSON, ORGANIZATION, and LOCATION entities. It is included as
english.all.3class.distsim.crf.ser.gz. It is trained on data from
CoNLL, MUC6, MUC7, ACE, OntoNotes, and Wikipedia.
Because this model is trained on both US
and UK newswire, it is fairly robust across the two domains.

We have also included a 4 class NER tagger trained on the CoNLL 2003
Shared Task training data that labels for PERSON, ORGANIZATION,
LOCATION, and MISC. It is named
english.conll.4class.distsim.crf.ser.gz .

A third model is trained only on data from MUC and
distinguishes between 7 different classes:
english.muc.7class.distsim.crf.ser.gz.

All of the serialized classifiers come in two versions, one trained to
basically expected standard written English capitalization, and the other
to ignore capitalization information. The case-insensitive versions
of the three models available on the Stanford NER webpage.
These models use a distributional similarity lexicon to improve performance
(by between 1.5%-3% F-measure). The distributional similarity features
make the models perform substantially better, but they require rather
more memory. The distsim models are included in the release package.
The nodistsim versions of the same models may be available on the
Stanford NER webpage.

Finally, we have models for other languages, including two German models,
a Chinese model, and a Spanish model. The files for these models can be
found at:

http://nlp.stanford.edu/software/CRF-NER.html

QUICKSTART INSTRUCTIONS

This NER system requires Java 1.8 or later.

Providing java is on your PATH, you should be able to run an NER GUI
demonstration by just clicking. It might work to double-click on the
stanford-ner.jar archive but this may well fail as the operating system
does not give Java enough memory for our NER system, so it is safer to
instead double click on the ner-gui.bat icon (Windows) or ner-gui.sh
(Linux/Unix/MacOSX). Then, using the top option from the Classifier
menu, load a CRF classifier from the classifiers directory of the
distribution. You can then `either load a text file or web page from
the File menu, or decide to use the default text in the window. Finally,
you can now named entity tag the text by pressing the Run NER button.

From a command line, you need to have java on your PATH and the
stanford-ner.jar file and the lib directory in your CLASSPATH. (The way of doing this depends on
your OS/shell.) The supplied ner.bat and ner.sh should work to allow
you to tag a single file. For example, for Windows:

text

ner file

Or on Unix/Linux you should be able to parse the test file in the distribution
directory with the command:

java -mx600m -cp stanford-ner.jar:lib/* edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier classifiers/english.all.3class.distsim.crf.ser.gz -textFile sample.txt

Here's an output option that will print out entities and their class to
the first two columns of a tab-separated columns output file:

java -mx600m -cp stanford-ner.jar:lib/* edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier classifiers/english.all.3class.distsim.crf.ser.gz -outputFormat tabbedEntities -textFile sample.txt > sample.tsv

When run from a jar file, you also have the option of using a serialized
classifier contained in the jar file.

USING FULL STANFORD CORENLP NER FUNCTIONALITY

This standalone distribution also allows access to the full NER
capabilities of the Stanford CoreNLP pipeline. These capabilities
can be accessed via the NERClassifierCombiner class.
NERClassifierCombiner allows for multiple CRFs to be used together,
and has options for recognizing numeric sequence patterns and time
patterns with the rule-based NER of SUTime.

Suppose one combines three CRF's CRF-1,CRF-2, and CRF-3 with the
NERClassifierCombiner. When the NERClassiferCombiner runs, it will
first apply the NER tags of CRF-1 to the text, then it will apply
CRF-2's NER tags to any tokens not tagged by CRF-1 and so on. If
the option ner.combinationMode is set to NORMAL (default), any label
applied by CRF-1 cannot be applied by subsequent CRF's. For instance
if CRF-1 applies the LOCATION tag, no other CRF's LOCATION tag will be
used. If ner.combinationMode is set to HIGH_RECALL, this limitation
will be deactivated.

To use NERClassifierCombiner at the command-line, the jars in lib
and stanford-ner.jar must be in the CLASSPATH. Here is an example command:

Let's break this down a bit. The flag "-ner.model" should be followed by a
list of CRF's to be combined by the NERClassifierCombiner. Some serialized
CRF's are provided in the classifiers directory. In this example the CRF's
trained on the CONLL 4 class data and the MUC 7 class data are being combined.

When the flag "-ner.useSUTime" is followed by "false", SUTime is shut off. You should
note that when the "false" is omitted, the text "4 days ago" suddenly is
tagged with DATE. These are the kinds of phrases SUTime can identify.

NERClassifierCombiner can be run on different types of input as well. Here is
an example which is run on CONLL style input:

It is crucial to include the "-map word=0,answer=1" , which is specifying that
the input test file has the words in the first column and the answer labels
in the second column.

It is also possible to serialize and load an NERClassifierCombiner.

This command loads the three sample crfs with combinationMode=HIGH_RECALL
and SUTime=false, and dumps them to a file named
test_serialized_ncc.ncc.ser.gz

java -mx2g edu.stanford.nlp.ie.NERClassifierCombiner -ner.model
classifiers/english.conll.4class.distsim.crf.ser.gz,classifiers/english.muc.7class.distsim.crf.ser.gz,
classifiers/english.all.3class.distsim.crf.ser.gz -ner.useSUTime false
-ner.combinationMode HIGH_RECALL -serializeTo test.serialized.ncc.ncc.ser.gz

An example serialized NERClassifierCombiner with these settings is supplied in
the classifiers directory. Here is an example of loading that classifier and
running it on the sample CONLL data:

java -mx2g edu.stanford.nlp.ie.NERClassifierCombiner -loadClassifier
classifiers/example.serialized.ncc.ncc.ser.gz -map word=0,answer=1
-testFile sample-conll-file.txt

For a more exhaustive description of NERClassifierCombiner go to
http://nlp.stanford.edu/software/ncc-faq.html

PROGRAMMATIC USE

The NERDemo file illustrates a couple of ways of calling the system
programatically. You should get the same results from

java -cp stanford-ner.jar:lib/*:. -mx300m NERDemo classifiers/english.all.3class.distsim.crf.ser.gz sample.txt

as from using CRFClassifier. For more information on API calls, look in
the enclosed javadoc directory: load index.html in a browser and look
first at the edu.stanford.nlp.ie.crf package and CRFClassifier class.
If you wish to train your own NER systems, look also at the
edu.stanford.nlp.ie package NERFeatureFactory class.

SERVER VERSION

The NER code may also be run as a server listening on a socket:

java -mx1000m -cp stanford-ner.jar:lib/* edu.stanford.nlp.ie.NERServer 1234

You can specify which model to load with flags, either one on disk:

java -mx1000m -cp stanford-ner.jar:lib/* edu.stanford.nlp.ie.NERServer -loadClassifier classifiers/all.3class.crf.ser.gz 1234

Or if you have put a model inside the jar file, as a resource under, say, models:

java -mx1000m -cp stanford-ner.jar:lib/* edu.stanford.nlp.ie.NERServer -loadClassifier models/all.3class.crf.ser.gz 1234

RUNNING CLASSIFIERS FROM INSIDE A JAR FILE

The software can run any serialized classifier from within a jar file by
following the -loadClassifier flag by some resource available within a
jar file on the CLASSPATH. An end user can make
their own jar files with the desired NER models contained inside.
This allows single jar file deployment.

PERFORMANCE GUIDELINES

Performance depends on many factors. Speed and memory use depend on
hardware, operating system, and JVM. Accuracy depends on the data
tested on. Nevertheless, in the belief that something is better than
nothing, here are some statistics from one machine on one test set, in
semi-realistic conditions (where the test data is somewhat varied).

ner-eng-ie.crf-3-all2006-distsim.ser.gz (older version of ner-eng-ie.crf-3-all2008-distsim.ser.gz)
Memory: 320MB (on a 32 bit machine)
PERSON ORGANIZATION LOCATION
91.88 82.91 88.21

CHANGES

2020-11-17 4.2.0 Update for compatibility

2020-05-10 4.0.0 Update to UDv2.0 tokenization

2018-10-16 3.9.2 Update for compatibility

2018-02-27 3.9.1 KBP ner models for Chinese and Spanish

2017-06-09 3.8.0 Updated for compatibility

2016-10-31 3.7.0 Improved Chinese NER

2015-12-09 3.6.0 Updated for compatibility

2015-04-20 3.5.2 synch standalone and CoreNLP functionality

2015-01-29 3.5.1 Substantial accuracy improvements

2014-10-26 3.5.0 Upgrade to Java 1.8

2014-08-27 3.4.1 Add Spanish models

2014-06-16 3.4 Fix serialization bug

2014-01-04 3.3.1 Bugfix release

2013-11-12 3.3.0 Update for compatibility

2013-06-19 3.2.0 Improve handling of line-by-line input

2013-04-04 1.2.8 nthreads option

2012-11-11 1.2.7 Improved English 3 class model by including
data from Wikipedia, release Chinese model

2012-07-09 1.2.6 Minor bug fixes

2012-05-22 1.2.5 Fix encoding issue

2012-04-07 1.2.4 Caseless version of English models supported

2012-01-06 1.2.3 Minor bug fixes

2011-09-14 1.2.2 Improved thread safety

2011-06-19 1.2.1 Models reduced in size but on average improved
in accuracy (improved distsim clusters)

2011-05-16 1.2 Normal download includes 3, 4, and 7
class models. Updated for compatibility
with other software releases.

2009-01-16 1.1.1 Minor bug and usability fixes, changed API

2008-05-07 1.1 Additional feature flags, various code updates

2006-09-18 1.0 Initial release

Doc Ner TODO 2009 (`doc/ner/TODO-2009.txt`)

Add German NER
Add BionNER
Add all uppercase/uncased NER model
Make it possible to have multiple NER matching threads sharing one data resource of features, etc.

Doc Ner Ner Manifest (`doc/ner/ner-manifest.txt`)

Manifest-Version: 1.0
Created-By: Stanford JavaNLP
Main-Class: edu.stanford.nlp.ie.crf.NERGUI

Doc Ner Performance (`doc/ner/performance.txt`)

PERFORMANCE NOTES:

Note that these results are underestimates for DATE, TIME, etc. because
not all the test data (mcdm.test) was marked for them. But they're not
bad undercounts, because the classifiers have as much trouble based on
the "all" data on recall as precision. For example:

Sept 2006 F1 scores on mcdm.test

ner-eng-ie.crf-3-all2006.ser.gz
Memory: 100 MB
PERSON ORGANIZATION LOCATION
89.19 80.15 85.48

ner-eng-ie.crf-3-all2006-distsim.ser.gz
Memory: 320MB
PERSON ORGANIZATION LOCATION
91.88 82.91 88.21

ner-eng-ie.crf-7-muc.ser.gz
Memory: 120MB
PERSON ORGANIZATION LOCATION DATE TIME MONEY PERCENT
74.45 59.93 76.27 55.59 73.12 64.96 71.05

ner-eng-ie.crf-7-muc-distsim.ser.gz
Memory: 350MB
PERSON ORGANIZATION LOCATION DATE TIME MONEY PERCENT
84.09 65.20 83.13 56.68 72.19 66.01 70.82

ner-eng-ie.crf-8-all2006.ser.gz
Memory: 120MB
PERSON ORGANIZATION LOCATION DATE TIME MONEY PERCENT
89.15 79.93 85.07 47.43 69.79 59.10 29.87

ner-eng-ie.crf-8-all2006-distsim.ser.gz
Memory: 350MB
PERSON ORGANIZATION LOCATION DATE TIME MONEY PERCENT
92.11 82.55 87.65 47.95 69.59 58.82 31.17

train_all.8.distsim_test_mcdm.test.results
processed 125957 tokens with 11947 phrases; found: 11465 phrases; correct: 9437.
accuracy: 95.16%; precision: 82.31%; recall: 78.99%; FB1: 80.62
DATE: precision: 85.23%; recall: 33.36%; FB1: 47.95 535
LOCATION: precision: 85.06%; recall: 90.41%; FB1: 87.65 3293
MISC: precision: 0.00%; recall: 0.00%; FB1: 0.00 518
MONEY: precision: 71.09%; recall: 50.17%; FB1: 58.82 211
ORGANIZATION: precision: 84.04%; recall: 81.12%; FB1: 82.55 3846
PERCENT: precision: 64.86%; recall: 20.51%; FB1: 31.17 37
PERSON: precision: 91.39%; recall: 92.83%; FB1: 92.11 2905
TIME: precision: 99.17%; recall: 53.60%; FB1: 69.59 120

Doc Ner Sample W Time (`doc/ner/sample-w-time.txt`)

Barack Obama was born on August 4, 1961 in Honolulu, Hawaii which was 4 days
ago.

Doc Ner Sample (`doc/ner/sample.txt`)

The fate of Lehman Brothers, the beleaguered investment bank, hung in the balance on Sunday as Federal Reserve officials and the leaders of major financial institutions continued to gather in emergency meetings trying to complete a plan to rescue the stricken bank. Several possible plans emerged from the talks, held at the Federal Reserve Bank of New York and led by Timothy R. Geithner, the president of the New York Fed, and Treasury Secretary Henry M. Paulson Jr.

Doc Openie README (`doc/openie/README.txt`)

Stanford Open Domain Relation Extractor, v3.5.2 - 2015-04-20
Copyright (c) 2002-2012 The Board of Trustees of
The Leland Stanford Junior University. All Rights Reserved.

Original author: Gabor Angeli
Code contributions:

This package contains an open domain relation extractor, producing
<subject, relation, object> triples where each element can be an arbitrary
plain-text element.

Doc Openie Openie Manifest (`doc/openie/openie-manifest.txt`)

Manifest-Version: 1.0
Created-By: Stanford JavaNLP
Main-class: edu.stanford.nlp.naturalli.OpenIE

Doc Segmenter README Arabic (`doc/segmenter/README-Arabic.txt`)

Stanford Arabic Segmenter - v4.2.0 - 2020-11-17

Arabic segmenter by Spence Green
CRF code by Jenny Finkel
Support code by Stanford JavaNLP members

The system requires Java 6 (JDK1.6) or higher.

The Arabic word segmenter is based on a conditional random field (CRF) sequence
classifier. The included TreebankPreprocessor package can be used to generate
training data for the model, assuming a copy of Penn Arabic Treebank (ATB) is available.

INSTALLATION

The segmenter does not require any platform-specific installation. Unpack the
gzip'd tar file in any convenient location. To invoke the segmenter, you'll need
to add the ".jar" files in the unpacked directory to your Java classpath. If you
don't know how to modify your classpath, read this tutorial:

http://docs.oracle.com/javase/tutorial/essential/environment/paths.html

The commands in the remainder of this README assume that you've added the jars
to your classpath.

USAGE

The segmenter assumes that you have a newline delimited text file with UTF-8
encoding. If you're working with Arabic script, chances are that your file is
already encoded in UTF-8. If it is in some other encoding, then you should
check Google for a tutorial on "iconv," a handy tool that allows you to
automatically change file encodings.

Suppose that your raw Arabic file is called "my_arabic_file.txt". You can
segment the tokens in this file with the following command:

java -mx1g edu.stanford.nlp.international.arabic.process.ArabicSegmenter -loadClassifier data/arabic-segmenter-atb+bn+arztrain.ser.gz -textFile my_arabic_file.txt > my_arabic_file.txt.segmented

Additional command line options are available to mark proclitics and enclitics
that were split by the segmenter. Suppose that the raw token "AABBBCC" was split
into three segments "AA BBB CC", where "AA" is a proclitic and "CC" is an
enclitic. You can direct the segmenter to mark these clitics with these command-line
options:

-prefixMarker #
-suffixMarker #

In this case, the segmenter would produce "AA# BBB #CC". Here '#' is the segment marker,
but the options accept any character as an argument. You can use different markers
for proclitics and enclitics.

ORTHOGRAPHIC NORMALIZATION

The segmenter contains a deterministic orthographic normalization package:

edu.stanford.nlp.international.arabic.process.ArabicTokenizer

ArabicTokenizer supports various orthographic normalization options that can be configured
in ArabicSegmenter using the -orthoOptions flag. The argument to -orthoOptions is a comma-separated list of
normalization options. The following options are supported:

useUTF8Ellipsis : Replaces sequences of three or more full stops with \u2026
normArDigits : Convert Arabic digits to ASCII equivalents
normArPunc : Convert Arabic punctuation to ASCII equivalents
normAlif : Change all alif forms to bare alif
normYa : Map ya to alif maqsura
removeDiacritics : Strip all diacritics
removeTatweel : Strip tatweel elongation character
removeQuranChars : Remove diacritics that appear in the Quran
removeProMarker : Remove the ATB null pronoun marker
removeSegMarker : Remove the ATB clitic segmentation marker
removeMorphMarker : Remove the ATB morpheme boundary markers
removeLengthening : Replace sequences of three identical characters with one
atbEscaping : Replace left/right parentheses with ATB escape characters

The orthographic normalization options must match at both training and test time!
Consequently, if you want to apply an orthographic normalization that differs from the
default, then you'll need to retrain ArabicSegmenter.

SEGMENTING DIALECTAL TEXT

The segmenter supports segmenting dialectal Arabic using domain adaptation.
[Hal Daumé III, Frustratingly Easy Domain Adaptation, ACL 2007] The model that
comes with this distribution is trained to support Egyptian dialect. To
indicate that the provided text is in Egyptian dialect, add the command-line
option:

-domain arz

You can also construct a file that specifies a dialect for each
newline-separated sentence, by adding "atb" [MSA] or "arz" [Egyptian] at the
beginning of each line followed by a space character. This feature is enabled
with the flag:

-withDomains

See the bottom of the next section for information about training the
segmenter on your own dialectal data.

TRAINING THE SEGMENTER

The current model is trained on parts 1-3 of the ATB, parts 1-8 of the ARZ treebank,
and the Broadcast News treebank. To train a new model, you need to create a data file from
the unpacked LDC distributions. You can create this data file with
the script tb-preproc, which is included in the segmenter release.

You'll need:

Python 2.7 (for running the preprocessing scripts)
an unpacked LDC distribution with files in integrated format
a directory with four text files called dev, train, test, and all, each of
which lists filenames of integrated files (these usually end in .su.txt),
one per line. dev, train, and test can be left empty if you have no need
for a train/test split.

The splits that we use are included in the distribution. You can also find
them at

http://nlp.stanford.edu/software/parser-arabic-data-splits.shtml

Once you have these, run the tb-preproc script, providing the necessary
arguments:

atb_base - the most specific directory that is a parent of all
integrated files you wish to include. Files will be located
recursively by name in this directory; it is not recommended
to have several copies of the same distribution within this
directory (though all that will happen is that you will
train on redundant data).

splits_dir - the directory containing dev, train, and test listings

output_prefix - the location and filename prefix that will identify the
output files. The preprocessor appends "-all.utf8.txt" to
this argument to give the name of the output file for the
"all" split (and similarly for dev, train, and test).

domain - [optional] a label for the Arabic dialect/genre that this
data is in. Our model uses "atb" for ATB1-3, "bn" for
Broadcast News, and "arz" for Egyptian. If a domain is
given, additional files will be generated (named e.g.
"output_prefix-withDomains-all.utf8.txt") for training the
domain adaptation model.

Suppose your output_prefix is "./atb". You should see files in the current
working directory named

atb-dev.utf8.txt
atb-train.utf8.txt
atb-test.utf8.txt

You can use the train file to retrain the segmenter with this command:

java -Xmx12g -Xms12g edu.stanford.nlp.international.arabic.process.ArabicSegmenter -trainFile atb-train.utf8.txt -serializeTo my_trained_segmenter.ser.gz

This command will produce the serialized model "my_trained_segmenter.ser.gz"
that you can use for raw text processing as described in the "USAGE" section above.

The command above train the model with L2 regularization. For the model
included in the distribution has been trained using L1 regularization, which
decreases the model file size dramatically in exchange for a usually
negligible drop in accuracy. To use L1 regularization, add the options

-useOWLQN -priorLambda 0.05

TRAINING FOR DIALECT

To train a model with domain adaptation, first make sure you have generated a
training file with domain labels. You can create this using the preprocessing
script with the optional domain argument, or do it yourself with a simple sed
script (the -withDomains files differ from the simple training files only in
the presence of a domain identifier prepended to each line followed by a
space). These domain labels can be arbitrary strings, as long as they don't
contain whitespace characters; thus, if you have data available for other
dialects in ATB format, it is possible to train your own system that can
support these dialects. For best results, include MSA data as well as your
dialect data in your training. You can do this by simply concatenating the
ATB1-3 -withDomains file and the dialect -withDomains file. (Adding data from
dialects other than your target dialect should not hurt performance, as long
as they are marked as different domains--it may even help!)

The training command for domain-labeled data is:

java -Xmx64g -Xms64g edu.stanford.nlp.international.arabic.process.ArabicSegmenter -withDomains -trainFile atb+arz-withDomains-train.utf8.txt -serializeTo my_trained_segmenter.ser.gz

Warning: training with lots of data from several domains requires a lot of
memory and processor time. If you have enough memory to fit all of the
weights for the entire dataset in RAM (this is a bit less than 64G for ATB1-3

BN + ARZ), training will take about ten days of single-threaded processor
time. This can be parallelized by adding the option

-multiThreadGrad

If you are not running on a machine with 64G of RAM, the training is likely to
take much longer. You have been warned.

Doc Segmenter README Chinese (`doc/segmenter/README-Chinese.txt`)

Stanford Chinese Segmenter - v4.2.0 - 2020-11-17

Chinese segmenter by Pi-Chuan Chang, Huihsin Tseng, and Galen Andrew
CRF code by Jenny Finkel
Support code by Stanford JavaNLP members

The system requires Java 6 (JDK1.6) or higher.

USAGE

Unix:

segment.sh [-k] [ctb|pku]
ctb : Chinese Treebank
pku : Beijing Univ.

filename: The file you want to segment. Each line is a sentence.
encoding: UTF-8, GB18030, etc.
(This must be a character encoding name known by Java)
size: size of the n-best list (just put '0' to print the best hypothesis
without probabilities).
-k: keep all white spaces in the input

Sample usage: segment.sh ctb test.simp.utf8 UTF-8
Note: Large test file requires large memory usage. For processing
large data files, you may want to change memory allocation in Java
(e.g., to be able to use 8Gb of memory, you need to change "-mx2g"
to "-mx8g" inside segment.sh). Another solution is to split the test
file to smaller ones to reduce memory usage.
In addition to the command line scripts, there is a Java class
"SegDemo" which shows how to call the segmenter in Java code.
Usage:
java -mx2g -cp "*:." SegDemo test.simp.utf8

SegDemo as supplied assumes that it is running in the home directory of the
installation, and to run anywhere else, you need to set the path to the
dictionaries.

SEGMENTATION MODELS

Two segmentation models are provided. The "ctb" model was trained with Chinese
treebank (CTB) segmentation, and the "pku" model was trained with Beijing
University's (PKU) segmentation. PKU models provide smaller vocabulary
sizes and OOV rates on test data than CTB models.

For both CTB and PKU, we provide two models representing slightly different
feature sets:

Models "ctb" and "pku" incorporate lexicon features to increase consistency in
segmentation.

DATA

[Segmentation standard]
(Chinese Penn Treebank)
The supplied segmenter segments according to Chinese Penn Treebank
segmentation conventions. For more information, see:

http://www.cis.upenn.edu/~chinese/segguide.3rd.ch.pdf

(Beijing University)
This segmenter segments according to the Peking University standard.
For more information, see:

http://sighan.cs.uchicago.edu/bakeoff2005/data/pku_spec.pdf

[Training data]
(Chinese Penn Treebank)
"data/ctb.gz" is trained with the training data in the LDC Chinese Treebank 7

(Beijing University)
"data/pku.gz" is trained with the data provided by Peking University
for the Second International Chinese Word Segmentation Bakeoff.
See:

http://sighan.cs.uchicago.edu/bakeoff2005/

MORE INFORMATION

The details of the segmenter can be found in this paper:

Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky
and Christopher Manning.
"A Conditional Random Field Word Segmenter."
In Fourth SIGHAN Workshop on Chinese Language Processing. 2005.

http://nlp.stanford.edu/pubs/sighan2005.pdf

(Notice that the training data, features and normalizations
used in this distribution are not exactly the same as the systems
described in the paper.)

The description of the lexicon features can be found in:

Pi-Chuan Chang, Michel Gally and Christopher Manning.
"Optimizing Chinese Word Segmentation for Machine Translation Performance"
In ACL 2008 Third Workshop on Statistical Machine Translation.

http://nlp.stanford.edu/pubs/acl-wmt08-cws.pdf

For more information, look in the included Javadoc, starting with the
edu.stanford.nlp.ie.crf.CRFClasifier class documentation.

Send any questions or feedback to [email protected].

Doc Segmenter README (`doc/segmenter/README.txt`)

The Stanford segmenter distribution includes tools for segmenting
Chinese and Arabic text. See README-Chinese.txt and README-Arabic.txt
for more details.

LICENSE

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/ .

For more information, bug reports, fixes, contact:
Christopher Manning
Dept of Computer Science, Gates 2A
Stanford CA 94305-9020
USA
[email protected]

CHANGES

2020-11-17 4.2.0 Update for compatibility

2020-05-10 4.0.0 New Chinese segmenter trained off of CTB 9.0

2018-10-16 3.9.2 Update for compatibility

2018-02-27 3.9.1 Updated for compatibility

2016-10-31 3.7.0 Update for compatibility

2015-12-09 3.6.0 Update for compatibility

2015-04-20 3.5.2 Update for compatibility

2015-01-29 3.5.1 Update for compatibility

2014-10-26 3.5.0 Upgrade to Java 1.8

2014-08-27 3.4.1 Update for compatibility

2014-06-16 3.4 Update Arabic segmenter

2014-01-04 3.3.1 Bugfix release

2013-11-12 3.3.0 Update for compatibility

2013-06-19 3.2.0 Improve handling of line-by-line input

2013-04-04 1.6.8 ctb7 model, -nthreads option

2012-11-11 1.6.7 Bugfixes to both Arabic and Chinese
segmenters; Chinese segmenter can now load
files from jar

2012-07-09 1.6.6 Improved Arabic model

2012-05-22 1.6.5 Supports stdin.

2012-03-09 1.6.4 Arabic segmenter now included.

2011-12-16 1.6.3 Updated code to maintain compatibility

2011-09-14 1.6.2 Updated code to maintain compatibility

2011-06-15 1.6.1 Updated code to maintain compatibility

2011-05-15 1.6 Updated models, code to be compatible with
other current releases

2008-05-21 1.5 The models in distribution incorporate
training lexicon features. In addition, the
segmenter now supports k-best output.

2006-05-12 1.0 This distribution includes models of two
segmentation standards -- CTB and PK
(Beijing Univ.)

2006-04-10 0.9 Add normalization for punctuation/symbol
characters from the "ASCII" (U+0021-U+0075)
code point range.

Doc Tagger README Models (`doc/tagger/README-Models.txt`)

Stanford POS Tagger, v4.2.0 - 2020-11-17
Copyright (c) 2002-2020 The Board of Trustees of
The Leland Stanford Junior University. All Rights Reserved.

This document contains (some) information about the models included in
this release and that may be downloaded for the POS tagger website at
http://nlp.stanford.edu/software/tagger.html . All of the models mentioned
in this document are in the downloaded package in the same directory as this
readme. All taggers are accompanied by the props files used to create
them; please examine these files for more detailed information about the
creation of the taggers.

For English, the bidirectional taggers are slightly more accurate, but
tag much more slowly; choose the appropriate tagger based on your
speed/performance needs.

English taggers

english-left3words-distsim.tagger
Trained on WSJ sections 0-18 and extra parser training data using the
left3words architecture and includes word shape and distributional
similarity features. Penn tagset. UDv2.0 tokenization standard.

english-bidirectional-distsim.tagger
Trained on WSJ sections 0-18 using a bidirectional architecture and
including word shape and distributional similarity features.
Penn Treebank tagset. UDv2.0 tokenization standard.

english-caseless-left3words-distsim.tagger
Trained on WSJ sections 0-18 and extra parser training data using the
left3words architecture and includes word shape and distributional
similarity features. Penn tagset. Ignores case. UDv2.0 tokenization
standard.

Chinese tagger

chinese-nodistsim.tagger
Trained on a combination of CTB7 texts from Chinese and Hong Kong
sources.
LDC Chinese Treebank POS tag set.

chinese-distsim.tagger
Trained on a combination of CTB7 texts from Chinese and Hong Kong
sources with distributional similarity clusters.
LDC Chinese Treebank POS tag set.

Arabic tagger

arabic.tagger
Trained on the entire ATB p1-3.
When trained on the train part of the ATB p1-3 split done for the 2005
JHU Summer Workshop (Diab split), using (augmented) Bies tags, it gets

French tagger

french-ud.tagger
Trained on the French GSD (UDv2.2) data set

German tagger

german-ud.tagger
Trained on the German GSD (UDv2.2) data set

Spanish tagger

spanish-ud.tagger
Trained on the Spanish AnCora (UDv2.0) data set

Doc Tagger README (`doc/tagger/README.txt`)

Stanford POS Tagger, v4.2.0 - 2020-11-17
Copyright (c) 2002-2020 The Board of Trustees of
The Leland Stanford Junior University. All Rights Reserved.

Original tagger author: Kristina Toutanova
Code contributions: Christopher Manning, Dan Klein, William Morgan,
Huihsin Tseng, Anna Rafferty, John Bauer
Major rewrite for version 2.0 by Michel Galley.
Current release prepared by: Jason Bolton

This package contains a Maximum Entropy part of speech tagger.

A Part-Of-Speech Tagger (POS Tagger) is a piece of software that reads
text in some language and assigns parts of speech to each word (and
other tokens), such as noun, verb, adjective, etc. Generally
computational applications use more fine-grained POS tags like
'noun-plural'. This software is a Java implementation of the log-linear
part-of-speech (POS) taggers described in:

Kristina Toutanova and Christopher D. Manning. 2000. Enriching the
Knowledge Sources Used in a Maximum Entropy Part-of-Speech
Tagger. Proceedings of the Joint SIGDAT Conference on Empirical Methods
in Natural Language Processing and Very Large Corpora (EMNLP/VLC-2000),
Hong Kong.

Kristina Toutanova, Dan Klein, Christopher Manning, and Yoram
Singer. 2003. Feature-Rich Part-of-Speech Tagging with a Cyclic
Dependency Network. In Proceedings of HLT-NAACL 2003 pages 252-259.

The system requires Java 1.8+ to be installed. About 60 MB of memory is
required to run a trained tagger, depending on the OS, tagging model
chosen, etc. (i.e., you may need to give to java an option like java
-mx120m). Plenty of memory is needed to train a tagger. It depends on
the complexity of the model but at least 1GB is recommended (java
-mx1g). Two trained tagger models for English are included with the
tagger, along with some caseless versions, and we provide models for
some other languages. The tagger can be retrained on other languages
based on POS-annotated training text.

QUICKSTART

The Stanford POS Tagger is designed to be used from the command line or
programmatically via its API.

There is a GUI interface, but it is for
demonstration purposes only; most features of the tagger can only be
accessed via the command line. To run the demonstration GUI you should
be able to use any of the following 2 methods:

java -mx200m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTaggerGUI models/wsj-0-18-left3words-distsim.tagger

Running the appropriate script for your operating system:
stanford-postagger-gui.bat
./stanford-postagger-gui.sh

To run the tagger from the command line, you can start with the provided
script appropriate for you operating system:
./stanford-postagger.sh models/wsj-0-18-left3words-distsim.tagger sample-input.txt
stanford-postagger models\wsj-0-18-left3words-distsim.tagger sample-input.txt
The output should match what is found in sample-output.txt

The tagger has three modes: tagging, training, and testing. Tagging
allows you to use a pretrained model (two English models are included)
to assign part of speech tags to unlabeled text. Training allows you to
save a new model based on a set of tagged data that you provide.
Testing allows you to see how well a tagger performs by tagging labeled
data and evaluating the results against the correct tags.

Many options are available for training, tagging, and testing. These
options can be set using a properties file. To start, you can generate a
default properties file by:

java -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -genprops > myPropsFile.prop

This will create the file myPropsFile.prop with descriptions of each
option for the tagger and the default values for these options
specified. Any properties you can specify in a properties file can be
specified on the command line or vice versa. For further information,
please consult the Javadocs (start with the entry for MaxentTagger,
which includes a table of all options which may be set to configure the
tagger and descriptions of those options).

To tag a file using the pre-trained bidirectional model

java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/wsj-0-18-bidirectional-distsim.tagger -textFile sample-input.txt > sample-tagged.txt

Tagged output will be printed to standard out, which you can redirect
as above. Note that the bidirectional model is slightly more accurate
but significantly slower than the left3words model.

To train a simple model

java -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -prop propertiesFile -model modelFile -trainFile trainingFile

To test a model

java -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -prop propertiesFile -model modelFile -testFile testFile

Using models for French, German, and Spanish

Starting with version 4.0.0, French, German, and Spanish are tokenized according to the UD 2.0 standard. This includes creating
multiword tokens. This functionality requires the pipeline functionality only available in the full Stanford CoreNLP distribution.
To tag French, German, or Spanish, one must provide UD 2.0 tokenized text, or upgrade to the full Stanford CoreNLP package to get
UD 2.0 tokenization for these languages.

To run on pretokenized text, add "-tokenize false" to your command.

Example:

java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/french-ud.tagger -tokenize false -textFile sample-input.txt > sample-tagged.txt

README.txt

This file.

LICENSE.txt

Stanford POS Tagger is licensed under the GNU General Public License (v2+).

stanford-postagger.jar
stanford-postagger-YYYY-MM-DD.jar

This is a JAR file containing all the Stanford classes necessary to
run the Stanford POS Tagger. The two jar files are identical. You can use
either the one with a version (date) indication or without, as you prefer.

src

A directory containing the Java 1.8 source code for the Stanford POS
Tagger distribution.

build.xml, Makefile

Files for building the distribution (with ant and make, respectively)

models

A directory containing trained POS taggers; the taggers end in ".tagger"
and the props file used to make the taggers end in ".props". The
".props" files cannot be directly used on your own machine as they use
paths on the Stanford NLP machines, but they may serve as examples for
your own properties files. Included in the full version are other
English taggers, a German tagger, an Arabic tagger, and a Chinese
tagger. If you chose to download the smaller version of the tagger,
you have only two English taggers (left3words is faster but slightly
less accurate than bidirectional-distsim) - feel free to download any
other taggers you need from the POS tagger website. More information
about the models can be found in the README-Models.txt file in this
directory.

sample-input.txt

A sample text file that you can tag to demonstrate the tagger.

sample-output.txt

Tagged output of the tagger (using the left3words model)

stanford-postagger-gui.sh
stanford-postagger-gui.bat

Scripts for invoking the GUI demonstration version of the tagger.

stanford-postagger.sh
stanford-postagger.bat

Scripts for running the command-line version of the tagger.

javadoc

Javadocs for the distribution. In particular, look at the javadocs
for the class edu.stanford.nlp.tagger.maxent.MaxentTagger.

TaggerDemo.java

A sample file for how to call the tagger in your own program. You
should be able to compile and run it with:

javac -cp stanford-postagger.jar TaggerDemo.java
java -cp ".:stanford-postagger.jar" TaggerDemo models/wsj-0-18-left3words-distsim.tagger sample-input.txt

(If you are on Windows, you need to replace the ":" with a ";" in the
-cp argument, and should use a "" in place of the "/" in the filename....)

THANKS

Thanks to the members of the Stanford Natural Language Processing Lab
for great collaborative work on Java libraries for natural language
processing.

http://nlp.stanford.edu/javanlp/

CHANGES

2020-11-17 4.2.0 Add currency data for English models.

2020-08-06 4.1.0 Add missing extractor, spanish tokenization
upgrades

2020-05-22 4.0.0 Model tokenization updated to UDv2.0

2018-10-16 3.9.2 New English models, better currency symbol
handling

2018-02-27 3.9.1 new French UD model

2017-06-09 3.8.0 new Spanish and French UD models

2016-10-31 3.7.0 Update for compatibility, German UD model

2015-12-09 3.6.0 Updated for compatibility

2015-04-20 3.5.2 Update for compatibility

2015-01-29 3.5.1 General bugfixes

2014-10-26 3.5.0 Upgrade to Java 1.8

2014-08-27 3.4.1 Add Spanish models

2014-06-16 3.4 Using CC tagset for French

2014-01-04 3.3.1 Bugfix release

2013-11-12 3.3.0 Add imperatives to English training data

2013-06-19 3.2.0 Decrease size and improve speed of tagger
models for all languages

2013-04-04 3.1.5 Speed improvements, ctb7 model, -nthreads
option

2012-11-11 3.1.4 Updated Chinese model

2012-07-09 3.1.3 Minor bug fixes

2012-05-22 3.1.2 Updated for compatibility with other releases

2012-03-09 3.1.1 Caseless models added

2012-01-06 3.1.0 French tagger added, tagging speed improved

2011-09-14 3.0.4 Updated for compatibility with other releases

2011-06-15 3.0.3 Updated for compatibility with other releases

2011-05-15 3.0.2 Can read training files in TSV format

2011-04-17 3.0.1 Improved German and Arabic models
Compatible with other Stanford releases

2010-05-21 3.0.0 Re-entrant

LICENSE

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/ .

For more information, bug reports, fixes, contact:
Christopher Manning
Dept of Computer Science, Gates 2A
Stanford CA 94305-9020
USA
Support/Questions: [email protected]
Licensing: [email protected]
http://nlp.stanford.edu/software/tagger.html

CONTACT

For questions about the Stanford POS tagger, please feel free to contact
the Stanford JavaNLP user community at the mailing list
[email protected]. You need to be a member of this
mailing list to be able to post to it. Join the list either by emailing
[email protected] (leave the subject and message
body empty) or by using the web interface at:

text

https://mailman.stanford.edu/mailman/listinfo/java-nlp-user

This is the best list to post to in order to ask questions, make
announcements, or for discussion among Stanford JavaNLP tool users. We
provide assistance on a best-effort basis. You can also look at the list
archives via https://mailman.stanford.edu/pipermail/java-nlp-user/. For
licensing questions, please see the tagger webpage or contact Stanford
JavaNLP at [email protected].

Doc Tagger Postagger Manifest (`doc/tagger/postagger-manifest.txt`)

Manifest-Version: 1.0
Created-By: Stanford JavaNLP
Main-class: edu.stanford.nlp.tagger.maxent.MaxentTaggerGUI

Doc Tagger Sample Input (`doc/tagger/sample-input.txt`)

A passenger plane has crashed shortly after take-off from Kyrgyzstan's
capital, Bishkek, killing a large number of those on board. The head of
Kyrgyzstan's civil aviation authority said that out of about 90
passengers and crew, only about 20 people have survived. The Itek Air
Boeing 737 took off bound for Mashhad, in north-eastern Iran, but turned
round some 10 minutes later.

Doc Tagger Sample Output (`doc/tagger/sample-output.txt`)

A_DT passenger_NN plane_NN has_VBZ crashed_VBN shortly_RB after_IN take-off_NN from_IN Kyrgyzstan_NNP 's_POS capital_NN ,, Bishkek_NNP ,, killing_VBG a_DT large_JJ number_NN of_IN those_DT on_IN board_NN ..
The_DT head_NN of_IN Kyrgyzstan_NNP 's_POS civil_JJ aviation_NN authority_NN said_VBD that_IN out_IN of_IN about_IN 90_CD passengers_NNS and_CC crew_NN ,, only_RB about_IN 20_CD people_NNS have_VBP survived_VBN ..
The_DT Itek_NNP Air_NNP Boeing_NNP 737_CD took_VBD off_RP bound_VBN for_IN Mashhad_NNP ,, in_IN north-eastern_JJ Iran_NNP ,, but_CC turned_VBD round_NN some_DT 10_CD minutes_NNS later_RB ..

Doc Tregex README Gui (`doc/tregex/README-gui.txt`)

Tregex GUI v4.2.0 - 2020-11-17

Original core Tregex code by Roger Levy and Galen Andrew.
Original core Tsurgeon code by Roger Levy.
TregexGUI by Anna Rafferty
Support code, additional features, etc. by Chris Manning
This release prepared by John Bauer.

TREGEX GRAPHICAL USER INTERFACE (GUI) README

The Tregex GUI is a graphical user interface for Tregex and Tsurgeon.
You can access it by double- clicking on the jar file tregex.jar. For
searching large treebanks, you may need to use more memory; the script
run-tregex-gui.command includes this allocation of memory and can be run
from the command line or double-click to run on a Mac. If you still have
memory problems, you can allot more memory by opening the script in a
text editor and changing "-mx300m" to include a bigger number (e.g.,
"-mx512m"). Tregex requires Java 1.5+. Further documentation for
Tregex and Tsurgeon can be found in README-tregex.txt and
README-tsurgeon.txt, respectively.

LOADING TREEBANKS/TREE FILES

To load a file containing Penn Treebank formatted trees, choose "Load trees..." from the file menu.
Multiple tree files and/or directories may be selected. After selecting the tree files you wish to
load, press "Load with file filters" to choose what filters you would like to apply to the files.
All filters are run based on the name of the file. Possible filtering options are:

Prefix: Load only files that start with the given character sequence
Extension: Load only files that end with the given character sequence
Has number in range: Loads only numbered files such that the number falls in the given range, inclusive.
Ranges can be disjoint as long as multiple ranges are comma-separated (e.g., "100-500,550-700")

File filters are combined such that all loaded files must obey all of
the filters; only one filter of any given type should be specified.

Once the tree files are loaded, their names appear in the upper left hand panel "Tree files:".
Unchecking the check boxes next to the files causes the unchecked files not to be included in
searches/tsurgeon operations. To remove all files from the tree panel, choose "Clear all files"
from the Edit menu.

PERFORMING TREGEX SEARCHES

To perform a Tregex search, load the files you would like to search and type a Tregex pattern
in the "Pattern:" box in the top middle of the window. Press "Help" beneath the Pattern box
for information about Tregex syntax. After you have typed the pattern, press "Search" to
find all matches to the given pattern.

By default, trees that contain at least one match are displayed in the "Matches:" panel in the
top right of the window, and the first matching tree is graphically displayed in the bottom
portion of the window. Click on a match in the Match panel to display it graphically. In the
graphical display, matched nodes in the tree are displayed in a different color than other nodes.
To display only the matched subtrees, choose "Preferences..." (Mac, from the Application menu) or
"Options..." (other OS, under Tools), and check "Show only matched portions of the tree". You must
rerun the search to switch between showing only matched portions and showing full trees.

In preferences, other display options can also be set, such as the colors, size, and font used by
the graphical display.

USING TSURGEON

Tsurgeon modifications can also be performed using Interactive Tregex. To enable Tsurgeon, choose
"Preferences..." from the File menu and check "Enable Tsurgeon". You can now run Tsurgeon scripts.
Tsurgeon commands must be paired with a Tregex pattern that names the nodes on which modifications
will be performed. Type the Tregex pattern in the Pattern box, and type the modifications you would
like to make in the "Tsurgeon script:" box. Then click "Run script" to perform the modifications.
Each Tsurgeon operation must appear on a separate line in the Tsurgeon script box. Press "Help" for
some information about Tsurgeon operation syntax.

SAVING RESULTS

You can save the results of a Tregex search or Tsurgeon operation by choosing "Save matches..." from the
File menu. This saves all trees in the Matches panel in Penn Treebank form. "Save matched sentences..." saves
the matches in sentence String form, just as they show up in the matches panel.

You can also save a log of the number of matches found for each pattern you have searched. By clicking the
"Statistics" button in the middle of the screen, below the Tsurgeon buttons, you can see a table of the patterns
for which you have searched, the number of trees that each matched, and the number of overall matches that were
found. To save this information in a tab delimited text file, choose "Save statistics..." from the File menu.

All three save options save files in the encoding specified in the Preferences panel for loading tree files.

MULTILANGUAGE SUPPORT

Some multilanguage support is built into Tregex GUI, and most languages can be read by the GUI. To enable
this support, choose go to Preferences (Mac, under the application menu) or Options (other OS, under the Tools menu).
Several options may need to be changed: tree reader factory, head finder, font, and encoding. Several possible
tree reader factories and head finders are provided; you may also specify your own. Two common languages you may be
trying to use are Chinese or Arabic; any head finder or tree reader factory beginning with "Chinese" or "Arabic" will
work for these languages, and additionally, CTBTreeReaderFactory is compatible with many Chinese treebanks. Based on
your choice of head finder and tree reader factory, the Tregex GUI will guess if you may need a different font and/or
text encoding. If a different text encoding is usually used for your selections, you will be prompted as to what text
encoding you would like to use. This may also be specified directly in the Preferences panel.

QUESTIONS

For more information on Tregex or Tsurgeon, read README-tregex.txt and README-tsurgeon.txt, and also look at the javadocs
suggested in those files. For questions about this distribution, please contact Stanford's JavaNLP group at
[email protected]. We provide assistance on a best-effort basis.

LICENSE

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/ .

For more information, bug reports, fixes, contact:
Christopher Manning
Dept of Computer Science, Gates 2A
Stanford CA 94305-9020
USA
[email protected]
http://nlp.stanford.edu/software/tregex.html

CHANGES

2020-11-17 4.2.0 Update for compatibility

2020-05-10 4.0.0 Update for compatibility

2018-10-16 3.9.2 Update for compatibility

2018-02-27 3.9.1 Updated for compatibility

2017-06-09 3.8.0 Updated for compatibility

2016-10-31 3.7.0 Update for compatibility

2015-12-09 3.6.0 Updated for compatibility

2015-04-20 3.5.2 Update for compatibility

2015-01-29 3.5.1 General bugfixes

2014-10-26 3.5.0 Upgrade to Java 1.8

2014-08-27 3.4.1 Fix bug by adding matcher object for tsurgeon

2014-06-16 3.4 Improved efficiency for some operations, new
tregex and tsurgeon patterns added

2014-01-04 3.3.1 Bugfix release, new createSubtree tsurgeon
operation

2013-11-12 3.3.0 Allow a TregexMatcher to have its own
HeadFinder, useful for the dependencies

2013-06-19 3.2.0 Fix for tsurgeon number reading bug

2013-04-04 2.0.6 Update to maintain compatibility

2012-11-11 2.0.5 Efficiency improvements

2012-07-09 2.0.4 Minor bug fixes

2012-05-22 2.0.3 Rebuilt to be compatible with everything.

2012-03-09 2.0.2 Efficiency improvements

2011-12-16 2.0.1 Fix bug in matchesAt, fix bug in category
function, add macros

2011-09-14 2.0.0 Efficiency improvements, include semgrex.

2011-05-15 1.4.4 Rebuilt to be compatible with everything.

2011-05-15 1.4.3 Rebuilt to be compatible with everything.

2011-04-17 1.4.2 Rebuilt to be compatible with tagger, parser,
and corenlp.

2010-11-18 1.4.1 Small fixes and improvements (improved help
screens, multipattern Tsurgeon scripts with
comments introduced by % supported, unclosed
regex no longer crashes GUI, support character
encodings in script files, fix bug in tregex
matching immediate domination path, TregexGUI
now shows filename and line number of each
match in matches panel)

2009-09-30 1.4 GUI slider for tree size, generalized relabel
command (incompatibly), __ and @ now supported
in path constraints; bugfixes.

2008-05-06 1.1 Several bug fixes; addition of browse trees
function, improved copy/paste and drag and
drop support; misc. feature additions

2007-09-20 1.0 Initial release

Doc Tregex README Tregex (`doc/tregex/README-tregex.txt`)

Tregex v4.2.0 - 2020-11-17

Original core Tregex code by Roger Levy and Galen Andrew.
Original core Tsurgeon code by Roger Levy.
GUI by Anna Rafferty
Support code, additional features, etc. by Chris Manning
This release prepared by John Bauer.

This package contains Tregex and Tsurgeon.

Tregex is a Tgrep2-style utility for matching patterns in trees. It can
be run in a graphical user interface, from the command line using the
TregexPattern main method, or used programmatically in java code via the
TregexPattern, TregexMatcher and TregexPatternCompiler classes.

As of version 1.2, the Tsurgeon tree-transformation utility is bundled
together with Tregex. See the file README.tsurgeon for details.

Java version 1.8 is required to use Tregex. If you really want to use
Tregex under an earlier version of Java, look into RetroWeaver:

http://retroweaver.sourceforge.net/

QUICKSTART

Programmatic use, command-line use, and GUI-use are supported. To access the
graphical interface for Tsurgeon and Tregex, double-click the tregex.jar file.
Some help (particularly with syntax) is provided within the program; for further
assistance, see README-gui.txt and the documentation mentioned below.

A full explanation of pattern syntax and usage is given in the javadocs
(particularly TregexPattern), and some of this information is also presented in
the TREGEX SYNTAX section below. As a quick example of usage,
the following line will scan an English PennTreebank annotated corpus
and print all nodes representing a verb phrase dominating a past-tense
verb and a noun phrase.

./tregex.sh 'VP < VBD < NP' corpus_dir

README-tregex.txt

This file.

README-tsurgeon.txt

Documentation for Tsurgeon, a tool for modifying trees.

README-gui.txt

Documentation for the graphical interface for Tregex and Tsurgeon tools.

LICENSE.txt

Tregex is licensed under the GNU General Public License.

stanford-tregex.jar

This is a JAR file containing all the Stanford classes necessary to
run tregex.

src

A directory containing the Java 1.8 source code for the Tregex
distribution.

javadoc

Javadocs for the distribution. In particular, look at the javadocs
for the class edu.stanford.nlp.trees.tregex.TregexPattern. The
first part of that class's javadoc describes syntax and semantics
for relations, node labels, node names, and variable groups. The
docs for the main method describe command-line options.

tregex.sh

a shell script for invoking the Tregex tree search tool.

tsurgeon.sh

a shell script for invoking the Tsurgeon tree transformation tool.

run-tregex-gui.command

A command file that can be double-clicked on a Mac to start the gui.

run-tregex-gui.bat

A bat file that can be double-clicked on a PC to start the gui.

examples

a directory containing several sample files to show Tsurgeon operation:

atree
a sample natural-language tree in Penn Treebank annotation style.
exciseNP
renameVerb
relabelWithGroupName
Sample tree-transformation operation files for Tsurgeon. See
README-tsurgeon.txt for more information about the contents of these
files.

TREGEX

Tregex Pattern Syntax and Uses

Using a Tregex pattern, you can find only those trees that match the pattern you're
looking for. The following table shows the symbols that are allowed in the pattern,
and below there is more information about using these patterns.

Table of Symbols and Meanings:
A << B
A dominates B
A >> B
A is dominated by B
A < B
A immediately dominates B
A > B
A is immediately dominated by B
A $ B
A is a sister of B (and not equal to B)
A .. B
A precedes B
A . B
A immediately precedes B
A ,, B
A follows B
A , B
A immediately follows B
A <<, B
B is a leftmost descendent of A
A <<- B
B is a rightmost descendent of A
A >>, B
A is a leftmost descendent of B
A >>- B
A is a rightmost descendent of B
A <, B
B is the first child of A
A >, B
A is the first child of B
A <- B
B is the last child of A
A >- B
A is the last child of B
A < B B is the last child of A A > B
A is the last child of B
A 0)
A >i B
A is the ith child of B (i > 0)
A <-i B
B is the ith-to-last child of A (i > 0)
A >-i B
A is the ith-to-last child of B (i > 0)
A <: B
B is the only child of A
A >: B
A is the only child of B
A <<: B
A dominates B via an unbroken chain (length > 0) of unary local trees.
A >>: B
A is dominated by B via an unbroken chain (length > 0) of unary local trees.
A $++ B
A is a left sister of B (same as $.. for context-free trees)
A $-- B
A is a right sister of B (same as $,, for context-free trees)
A $+ B
A is the immediate left sister of B (same as $. for context-free trees)
A $- B
A is the immediate right sister of B (same as $, for context-free trees)
A $.. B
A is a sister of B and precedes B
A $,, B
A is a sister of B and follows B
A $. B
A is a sister of B and immediately precedes B
A $, B
A is a sister of B and immediately follows B
A <+(C) B
A dominates B via an unbroken chain of (zero or more) nodes matching description C
A >+(C) B
A is dominated by B via an unbroken chain of (zero or more) nodes matching description C
A .+(C) B
A precedes B via an unbroken chain of (zero or more) nodes matching description C
A ,+(C) B
A follows B via an unbroken chain of (zero or more) nodes matching description C
A <<# B
B is a head of phrase A
A >># B
A is a head of phrase B
A <# B
B is the immediate head of phrase A
A ># B
A is the immediate head of phrase B
A == B
A and B are the same node
A : B
[this is a pattern-segmenting operator that places no constraints on the relationship between A and B]

Label descriptions can be literal strings, which much match labels exactly, or regular
expressions in regular expression bars: /regex/. Literal string matching proceeds as
String equality. In order to prevent ambiguity with other Tregex symbols, only standard
"identifiers" are allowed as literals, i.e., strings matching a-zA-Z* .
If you want to use other symbols, you can do so by using a regular expression instead of
a literal string. A disjunctive list of literal strings can be given separated by '|'.
The special string '' (two underscores) can be used to match any node. (WARNING!!
Use of the '' node description may seriously slow down search.) If a label description
is preceeded by '@', the label will match any node whose basicCategory matches the description.
NB: A single '@' thus scopes over a disjunction specified by '|': @NP|VP means things with basic category NP or VP.

Label description regular expressions are matched as find(), as in Perl/tgrep;
you need to specify ^ or $ to constrain matches.

In a chain of relations, all relations are relative to the first node in the chain.
For example, (S < VP < NP) means an S over a VP and also over an NP. If instead what
you want is an S above a VP above an NP, you should write S < (VP < NP).

Nodes can be grouped using parentheses '(' and ')' as in S < (NP $++ VP) to match an S
over an NP, where the NP has a VP as a right sister.

Boolean relational operators

Relations can be combined using the '&' and '|' operators, negated with the '!' operator,
and made optional with the '?' operator. Thus (NP < NN | < NNS) will match an NP node
dominating either an NN or an NNS. (NP > S & $++ VP) matches an NP that is both under
an S and has a VP as a right sister.

Relations can be grouped using brackets '[' and ']'. So the expression

NP [< NN | < NNS] & > S

matches an NP that (1) dominates either an NN or an NNS, and (2) is under an S. Without
brackets, & takes precedence over |, and equivalent operators are left-associative. Also
note that & is the default combining operator if the operator is omitted in a chain of
relations, so that the two patterns are equivalent:
(S < VP < NP)
(S < VP & < NP)

As another example, (VP < VV | < NP % NP) can be written explicitly as (VP [< VV | [< NP & % NP] ] ).

Relations can be negated with the '!' operator, in which case the expression will match
only if there is no node satisfying the relation. For example (NP !< NNP) matches only
NPs not dominating an NNP. Label descriptions can also be negated with '!': (NP < !NNP|NNS)
matches NPs dominating some node that is not an NNP or an NNS.

Relations can be made optional with the '?' operator. This way the expression will match even
if the optional relation is not satisfied. This is useful when used together with node naming
(see below).

Basic Categories

In order to consider only the "basic category" of a tree label, i.e. to ignore functional tags
or other annotations on the label, prefix that node's description with the @ symbol. For example
(@NP < @/NN.?/). This can only be used for individual nodes; if you want all nodes to use the
basic category, it would be more efficient to use a TreeNormalizer to remove functional tags
before passing the tree to the TregexPattern.

Segmenting patterns

The ":" operator allows you to segment a pattern into two pieces. This can simplify your pattern
writing. For example, the pattern S : NP matches only those S nodes in trees that also have an NP node.

Naming nodes

Nodes can be given names (a.k.a. handles) using '='. A named node will be stored in a map that
maps names to nodes so that if a match is found, the node corresponding to the named node can
be extracted from the map. For example (NP < NNP=name) will match an NP dominating an NNP
and after a match is found, the map can be queried with the name to retreived the matched node
using {@link TregexMatcher#getNode(Object o)} with (String) argument "name" (not "=name"). Note
that you are not allowed to name a node that is under the scope of a negation operator (the
semantics would be unclear, since you can't store a node that never gets matched to). Trying to
do so will cause a ParseException to be thrown. Named nodes can be put within the scope of an
optional operator.

Named nodes that refer back to previous named nodes need not have a node description -- this is
known as "backreferencing". In this case, the expression will match only when all instances of
the same name get matched to the same tree node. For example, the pattern:

(@NP <, (@NP $+ (/,/ $+ (@NP $+ /,/=comma))) <- =comma)

matches only an NP dominating exactly the sequence NP, NP; the mother NP cannot have any other
daughters. Multiple backreferences are allowed. If the node with no node description does not
refer to a previously named node, there will be no error, the expression simply will not match
anything.

Another way to refer to previously named nodes is with the "link" symbol: ''. A link is like a
backreference, except that instead of having to be equal to the referred node, the
current node only has to match the label of the referred to node. A link cannot have a node
description, i.e. the '' symbol must immediately follow a relation symbol.

Variable Groups

If you write a node description using a regular expression, you can assign its matching groups to
variable names. If more than one node has a group assigned to the same variable name, then matching
will only occur when all such groups capture the same string. This is useful for enforcing
coindexation constraints. The syntax is:

/ /#%

For example, the pattern (designed for Penn Treebank trees):

@SBAR < /^WH.*-([0-9]+)$/#1%index<<(__=empty < (/^-NONE-/< /^\T\-([0-9]+)$/#1%index))

will match only such that the WH- node under the SBAR is coindexed with the trace node that gets the name empty.

MISCELLANEOUS

Head Finders

To use the headship relations <# ># <<# >># correctly it is
important to specify a HeadFinder class appropriate to the trees
that you are searching. For information about how to specify a
HeadFinder class at the command line or through the API, please read
the javadocs for the class
edu.stanford.nlp.trees.tregex.TregexPattern. The following
HeadFinder classes are included with the Tregex distribution:

Penn Treebank of English (http://www.cis.upenn.edu/~treebank/):

text

edu.stanford.nlp.trees.CollinsHeadFinder (default)

Penn Treebank of Chinese (http://www.cis.upenn.edu/~chinese/):

text

edu.stanford.nlp.trees.international.pennchinese.ChineseHeadFinder

Penn Treebank of Arabic (http://www.ircs.upenn.edu/arabic/):

text

edu.stanford.nlp.trees.international.arabic.ArabicHeadFinder

NEGRA (http://www.coli.uni-saarland.de/projects/sfb378/negra-corpus/)

and

TIGER (http://www.ims.uni-stuttgart.de/projekte/TIGER/TIGERCorpus/)

treebanks of German (these can use the same headfinder):

text

edu.stanford.nlp.trees.international.negra.NegraHeadFinder

Tuebingen Treebank of Written German (http://www.sfs.uni-tuebingen.de/en/ascl/resources/corpora/tueba-dz.html):

text

edu.stanford.nlp.trees.international.tuebadz.TueBaDZHeadFinder

Tdiff

TregexGUI supports a constituent diff'ing method--similar to the UNIX diff command--for trees. To
enable Tdiff:

Clear the tree file list: File -> Clear tree file list
Enable Tdiff: Options -> Tdiff
Load two (2) files using the "File -> Load" dialog.
Select "Browse" on the main display

The GUI will display differences between each pair of trees in the two files. As such, the two files must
contain the same number of trees.

The first file in the tree file list is treated as the reference. Trees from the second file
will be displayed in the GUI, with bracketing differences highlighted in blue. Below the tree,
constituents in the reference tree that do not appear in the tree from the second file are shown
as lines below each respective span.

Tregex searches are supported and apply to the trees in the second file.

This feature was designed for debugging and analyzing parser output.

THANKS

Thanks to the members of the Stanford Natural Language Processing Lab
for great collaborative work on Java libraries for natural language
processing.

http://nlp.stanford.edu/javanlp/

LICENSE

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/ .

For more information, bug reports, fixes, contact:
Christopher Manning
Dept of Computer Science, Gates 2A
Stanford CA 94305-9020
USA
[email protected]
http://nlp.stanford.edu/software/tregex.html

CONTACT

For questions about this distribution, please contact Stanford's JavaNLP group at
[email protected]. We provide assistance on a best-effort basis.

Doc Tregex Tregex Manifest (`doc/tregex/tregex-manifest.txt`)

Manifest-Version: 1.0
Created-By: Stanford JavaNLP
Main-class: edu.stanford.nlp.trees.tregex.gui.TregexGUI
Class-Path: ../lib/AppleJavaExtensions.jar

Doc Tsurgeon README Tsurgeon (`doc/tsurgeon/README-tsurgeon.txt`)

Tsurgeon v4.2.0 - 2020-11-17

Original core Tregex code by Roger Levy and Galen Andrew.
Original core Tsurgeon code by Roger Levy.
GUI by Anna Rafferty.
Support code, additional features, etc. by Chris Manning.
This release prepared by John Bauer.

This package contains Tregex and Tsurgeon.

As of version 1.2, the Tsurgeon tree-transformation utility is bundled
together with Tregex. See the file README.tsurgeon for details.

Java version 1.8+ is required to use the current version of Tregex.

TSURGEON

Tsurgeon is a tool for modifying trees that match a particular Tregex
pattern. Further documentation for Tregex and Tregex GUI can be found in
README-tregex.txt and README-gui.txt, respectively.

Brief description:

Takes some trees, tries to match one or more tregex expressions to
each tree, and for each successful match applies some surgical
operations to the tree. Pretty-prints each resulting tree (after all
successful match/operation sets have applied) to standard output.

A simple example:

./tsurgeon.csh -treeFile atree exciseNP renameVerb

RUNNING TREGEX

Program Command Line Options:

-treeFile

specify the name of the file that has the trees you want to transform.

-po

Apply a single operation to every tree using the specified match
pattern and the specified operation.

-s

Prints the output trees one per line, instead of pretty-printed.

The arguments are then Tsurgeon scripts.
Each argument should be the name of a transformation file that contains a list of pattern
and transformation operation list pairs. That is, it is a sequence of pairs of a
TregexPattern pattern on one or more lines, then a
blank line (empty or whitespace), then a list of transformation operations one per line
(as specified by Tsurgeon syntax below to apply when the pattern is matched,
and then another blank line (empty or whitespace).
Note the need for blank lines: The code crashes if they are not present as separators
(although the blank line at the end of the file can be omitted).
The script file can include comment lines, either whole comment lines or
trailing comments introduced by %, which extend to the end of line. A needed percent
mark in patterns or operations can be escaped by a preceding backslash.

TSURGEON SYNTAX

Legal operation syntax and semantics (see Examples section for further detail):

delete ...

For each name_i, deletes the node it names and everything below it.

prune ...

For each name_i, prunes out the node it names. Pruning differs from
deletion in that if pruning a node causes its parent to have no
children, then the parent is in turn pruned too.

excise

The name1 node should either dominate or be the same as the name2
node. This excises out everything from name1 to name2. All the
children of name2 go into the parent of name1, where name1 was.

relabel

Relabels the node to have the new label. There are three possible forms
for the new-label:
relabel nodeX VP - for changing a node label to an alphanumeric
string, relabel nodeX /''/ - for relabeling a node to something that
isn't a valid identifier without quoting, and relabel nodeX
/^VB(.*)$/verb/$1/ - for regular expression based relabeling. In the
last case, all matches of the regular expression against the node
label are replaced with the replacement String. This has the semantics
of Java/Perl's replaceAll: you may use capturing groups and put them
in replacements with $n. Also, as in the example, you can escape a
slash in the middle of the second and third forms with / and \.
This last version lets you make a new label that is an arbitrary
String function of the original label and additional characters that
you supply.

insert
insert

inserts the named node, or a manually specified tree (see below for
syntax), into the position specified. Right now the only ways to
specify position are:

text

$+ <name>     the left sister of the named node
 $- <name>     the right sister of the named node
 >i <name>     the i_th daughter of the named node.
 >-i <name>    the i_th daughter, counting from the right, of the named node.

move

moves the named node into the specified position. To be precise, it
deletes (NOT prunes) the node from the tree, and re-inserts it
into the specified position. See above for how to specify position

replace

deletes name1 and inserts a copy of name2 in its place.

adjoin

adjoins the specified auxiliary tree (see below for syntax) into the
target node specified. The daughters of the target node will become
the daughters of the foot of the auxiliary tree.

adjoinH

similar to adjoin, but preserves the target node and makes it the root
of . (It is still accessible as name. The root of
the auxiliary tree is ignored.)

adjoinF

similar to adjoin, but preserves the target node and makes it the foot
of . (It is still accessible as name, and retains
its status as parent of its children. The foot of the auxiliary tree
is ignored.)

coindex ...

Puts a (Penn Treebank style) coindexation suffix of the form "-N" on
each of nodes name_1 through name_m. The value of N will be
automatically generated in reference to the existing coindexations
in the tree, so that there is never an accidental clash of
indices across things that are not meant to be coindexed.

Syntax for trees to be inserted or adjoined:

A tree to be adjoined in can be specified with LISP-like
parenthetical-bracketing tree syntax such as those used for the Penn
Treebank. For example, for the NP "the dog" to be inserted you might
use the syntax

(NP (Det the) (N dog))

That's all that there is for a tree to be inserted. Auxiliary trees
(a la Tree Adjoining Grammar) must also have exactly one frontier node
ending in the character "@", which marks it as the "foot" node for
adjunction. Final instances of the character "@" in terminal node labels
will be removed from the actual label of the tree.

For example, if you wanted to adjoin the adverb "breathlessly" into a
VP, you might specify the following auxiliary tree:

(VP (Adv breathlessly) VP@ )

All other instances of "@" in terminal nodes must be escaped (i.e.,
appear as @); this escaping will be removed by tsurgeon.

In addition, any node of a tree can be named (the same way as in
tregex), by appending = to the node label. That name can be
referred to by subsequent tsurgeon operations triggered by the same
match. All other instances of "=" in node labels must be escaped
(i.e., appear as =); this escaping will be removed by tsurgeon. For
example, if you want to insert an NP trace somewhere and coindex it
with a node named "antecedent" you might say

insert (NP (-NONE- T=trace))
coindex trace antecedent $

Examples of Tsurgeon operations:

Tree (used in all examples):
(ROOT
(S
(NP (NNP Maria_Eugenia_Ochoa_Garcia))
(VP (VBD was)
(VP (VBN arrested)
(PP (IN in)
(NP (NNP May)))))
(. .)))

Apply delete:
VP < PP=prep
delete prep
Result:
(ROOT
(S
(NP (NNP Maria_Eugenia_Ochoa_Garcia))
(VP (VBD was)
(VP (VBN arrested)
(. .)))
The PP node directly dominated by a VP is removed, as is
everything under it.

Apply prune:
S < (NP < NNP=noun)
prune noun
Result:
(ROOT
(S
(VP (VBD was)
(VP (VBN arrested)
(PP (IN in)
(NP (NNP May)))))
(. .)))
The NNP node is removed, and since this results in the NP above it
having no terminal children, the NP node is deleted as well.
Note: This is different from delete in which the NP above the NNP
would remain.

Apply excise:
VP < PP=prep
excise prep prep
Result:
(ROOT
(S
(NP (NNP Maria_Eugenia_Ochoa_Garcia))
(VP (VBD was)
(VP (VBN arrested)
(IN in)
(NP (NNP May)))))
(. .)))
The PP node is removed, and all of its children are added in the
place it was previously located. Excise removes all the nodes from
the first named node to the second named node, and the children of
the second node are added as children of the parent of the first node.
Thus, for another example:
VP=verb < PP=prep
excise verb prep
Result:
(ROOT
(S
(NP (NNP Maria_Eugenia_Ochoa_Garcia))
(VP (VBD was)
(IN in)
(NP (NNP May)))
(. .)))

Apply relabel:
VP=v < PP=prep
relabel prep verbPrep
Result:
(ROOT
(S
(NP (NNP Maria_Eugenia_Ochoa_Garcia))
(VP (VBD was)
(VP (VBN arrested)
(verbPrep (IN in)
(NP (NNP May)))))
(. .)))
The label for the node called prep (PP) is changed to verbPrep.
The other form of relabel uses regular expressions; consider the following
operation:
/^VB.+/=v
relabel v /^VB(.)$/ #1
Result:
(ROOT
(S
(NP (NNP Maria_Eugenia_Ochoa_Garcia))
(VP (D was)
(VP (N arrested)
(PP (IN in)
(NP (NNP May)))))
(. .)))
The Tregex pattern matches all nodes that begin "VB" and have at least one
more character. The Tsurgeon operation then matches the node label to the
regular expression "^VB(.)$" and selects the text matching the first part
that is not completely specified in the pattern. In this case, that is the
part matching the wildcard (.*), which matches all characters after the VB.
The node is then relabeled with that part of the text, causing, for example,
"VBD" to be relabeled "D". The "#1" specifies that the name of the node
should be the first group in the regex.

Apply insert (shown here with inserting a node, but could also be a tree):
S < (NP < (NNP=name !$- DET))
insert (DET Ms.) $+ name
Result:
(ROOT
(S
(NP (DET Ms.)
(NNP Maria_Eugenia_Ochoa_Garcia))
(VP (VBD was)
(VP (VBN arrested)
(PP (IN in)
(NP (NNP May)))))
(. .)))
The pattern matches the NNP node that is directly dominated by an NP
(which is directly dominated by an S) and is not a direct right sister
of a DET. Thus, the (DET Ms.) node is inserted immediately to the left
of that NNP node, as specified by "$+ name". "$+" is the location and
"name" describes what node the location is with respect to.
Note: Tsurgeon will re-search for matches after each run of the script;
thus, cycles may occur, causing the program to not terminate. The key
is to write patterns that match prior to the changes you would like to
make but that do not match afterwards. If the clause "!$- DET" had been
left out in this example, Tsurgeon would have matched the pattern after
every insert operation, causing an infinite number of DETs to be added.

Apply move:
VP=verb < PP=prep
move prep $- verb
Result:
(ROOT
(S
(NP (NNP Maria_Eugenia_Ochoa_Garcia))
(VP (VBD was)
(VP (VBN arrested)))
(PP (IN in)
(NP (NNP May)))
(. .)))
The PP is moved out of the VP that dominates it and added as a direct right
sister of the VP. As for insert, "$-" specifies the location for prep while
"verb" specifies what that location is relative to.
Note: "move" is a macro operation that deletes the given node and then inserts
it. "move" does not use prune, and thus any branches that now lack terminals will
remain rather than being removed.

Apply replace:
S < (NP=name < NNP)
replace name (NP (DET A) (NN woman))
Result:
(ROOT
(S
(NP (DET A)
(NN woman))
(VP (VBD was)
(VP (VBN arrested)
(PP (IN in)
(NP (NNP May)))))
(. .)))
"name" is matched to an NP that is dominated by an S and dominates an NNP, and
a new subtree ("(NP (DET A) (NN woman))") is added in the place where "name" was.
Note: This operation is vulnerable to falling into an infinite loop. See the note
concerning the "insert" operation and how patterns are matched.

Apply adjoin:
S < (NP=name < NNP)
adjoin (NP (DET A) (NN woman) NP@) name
Result:
(ROOT
(S
(NP (DET A)
(NN woman)
(NP (NNP Maria_Eugenia_Ochoa_Garcia)))
(VP (VBD was)
(VP (VBN arrested)
(PP (IN in)
(NP (NNP May)))))
(. .)))
First, the NP is matched to the NP dominating the NNP tag. Then, the specified
tree ("(NP (DET A) (NN woman) NP@)") is placed in that location. The "@" symbol
specifies that the children of the original NP node ("name") are to be placed
as children of a new NP node that is directly to the right of (NN woman). If
the specified tree were "(NP (DET A) (NN woman) VP@)" then the child
(NNP Maria_Eugenia_Ochoa_Garcia) would appear under a VP. Exactly one "@" node
must appear in the specified tree in order to indicate where to place the node
from the original tree.

Apply adjoinH:
S < (NP=name < NNP)
adjoinH ((NP (DET A) (NN woman) NP@)) name
Result:
(ROOT
(S
(NP (NP (DET A)
(NN woman)
(NP (NNP Maria_Eugenia_Ochoa_Garcia))))
(VP (VBD was)
(VP (VBN arrested)
(PP (IN in)
(NP (NNP May)))))
(. .)))
This operation differs from adjoin in that it retains the named node (in this
case, "name"). The named node is made the root of the specified tree, resulting
in two NP nodes dominating the DET in this example whereas only one was present
in the previous example. Note that the specified tree is wrapped in an extra
pair of parentheses in order to show the syntax for retaining the named node.
If the extra parentheses were not there and the specified tree was, for example,
(VP (DET A) (NN woman) NP@), the VP would be ignored in order to retain an NP as
the root. Thus, in this case, "adjoinH (VP (DET A) (NN woman) NP@) name" and
"adjoinH ((DET A) (NN woman) NP@) name" both produce the same tree:
(ROOT
(S
(NP (DET A)
(NN woman)
(NP (NNP Maria_Eugenia_Ochoa_Garcia)))
(VP (VBD was)
(VP (VBN arrested)
(PP (IN in)
(NP (NNP May)))))
(. .)))

Apply adjoinF:
S < (NP=name < NNP)
adjoinF (NP(DET A) (NN woman) @) name
Result:
(ROOT
(S
(NP (DET A)
(NN woman)
(NP (NNP Maria_Eugenia_Ochoa_Garcia)))
(VP (VBD was)
(VP (VBN arrested)
(PP (IN in)
(NP (NNP May)))))
(. .)))
This operation is very similar to adjoin and adjoinH, but this time the original
named node ("name" in this case) is maintained as the root of the subtree that
is adjoined. Thus, no node label needs to be given in front of the "@" and if
one is given, it will be ignored. For instance, "adjoinF (NP(DET A) (NN woman) VP@) name"
would still produce the same tree as above, despite the VP preceding the @.

Apply coindex:
NP=node < NNP=name
coindex node name
Result:
(ROOT
(S
(NP-1 (NNP-1 Maria_Eugenia_Ochoa_Garcia))
(VP (VBD was)
(VP (VBN arrested)
(PP (IN in)
(NP-2 (NNP-2 May)))))
(. .)))
This causes the named nodes to be numbered such that all nodes that are part
of the same match have the same number and all matches have distinct new names.
We had two instances of an NP dominating an NNP in this example, and they were
renamed such that NP-i < NNP-i for each match, with 1 <= i <= number of matches.

TSURGEON SCRIPTS

Script format:

Tsurgeon scripts are a combination of a Tregex pattern to match and a series
of Tsurgeon operations to perform on that match. The first line of a Tsurgeon
script should be the Tregex pattern. This should be followed by a blank line,
and then each subsequent line may contain one Tsurgeon operation. Tsurgeon
operations should not be separated by blank lines. The following is an example
of correctly formatted script:

S < NP=node < NNP=name

relabel node NP_NAME
coindex node name

Comments:

The character % introduces a comment that extends to the end of the
line. All other intended uses of % must be escaped as % .

CONTACT

For questions about this distribution, please contact Stanford's JavaNLP group at
[email protected]. We provide assistance on a best-effort basis.

LICENSE

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/ .

For more information, bug reports, fixes, contact:
Christopher Manning
Dept of Computer Science, Gates 2A
Stanford CA 94305-9020
USA
[email protected]
http://nlp.stanford.edu/software/tregex.html

Doc Tokensregex Examples Color.Rules (`doc/tokensregex/examples/color.rules.txt`)

Colors

Example to add "COLOR" as ner tag, and hex RGB code as the normalized tag for strings matching a color

Case insensitive pattern matching (see java.util.regex.Pattern flags)

ENV.defaultStringPatternFlags = 2

Map variable names to annotation keys

ner = { type: "CLASS", value: "edu.stanford.nlp.ling.CoreAnnotations$NamedEntityTagAnnotation" }
normalized = { type: "CLASS", value: "edu.stanford.nlp.ling.CoreAnnotations$NormalizedNamedEntityTagAnnotation" }
tokens = { type: "CLASS", value: "edu.stanford.nlp.ling.CoreAnnotations$TokensAnnotation" }

Create OR pattern of

regular expression over tokens to hex RGB code

for colors and save it in a variable

Define ruleType to be over tokens

ENV.defaults["ruleType"] = "tokens"

Define rule that

upon matching pattern defined by $Color

annotate matched tokens ($0) with ner="COLOR" and normalized=matched value ($$0.value)

{ pattern: ( $Colors ),
action: ( Annotate($0, ner, "COLOR"), Annotate($0, normalized, $$0.value ) ) }

Doc Tokensregex Examples Retokenize (`doc/tokensregex/examples/retokenize.txt`)

Uses TokensRegex library to split further split tokens with -

Map variable names to annotation keys

tokens = { type: "CLASS", value: "edu.stanford.nlp.ling.CoreAnnotations$TokensAnnotation" }

Indicates matched expressions should replace the tokens annotation

options.matchedExpressionsAnnotationKey = tokens;

Indicates that the matched expressions and tokens should be combined

options.extractWithTokens = TRUE;

Indicates that the matched expressions should be flattened to be just tokens

options.flatten = TRUE;

Mark units

Define ruleType to be over tokens

ENV.defaults["ruleType"] = "tokens"

Case insensitive pattern matching (see java.util.regex.Pattern flags)

ENV.defaultStringPatternFlags = 2

Indicates that after matching, and computing the result, the result should be placed in the tokens annotation

ENV.defaultResultAnnotationKey = tokens

Define rule where upon matching tokens with -, the matched token is split

{ pattern: ( /.+-.+/ ), result: Split($0[0], /-/, TRUE) }

--- METRICS ---

Files Extracted: 35
Estimated Token Budget: ~41666 tokens
Canonical Reference: https://codewiki.google/github.com/stanfordnlp/CoreNLP

Technical Documentation: stanfordnlp/CoreNLP

Overview (README.md) Copy Section

Stanford CoreNLP

Build Instructions Copy Section

Provided build

Build with Ant

Build with Maven

Models

Install by Gradle Copy Section

Useful resources Copy Section

Doc Classify README (`doc/classify/README.txt`) Copy Section

Stanford Classifier v4.2.0 - 2020-11-17 Copy Chapter

CHANGES Copy Chapter

Doc Corenlp README (`doc/corenlp/README.txt`) Copy Section

Stanford CoreNLP - Stanford's Suite of NLP Tools Copy Chapter

CHANGES Copy Chapter

Doc Corenlp Input (`doc/corenlp/input.txt`) Copy Section

Doc Lexparser README (`doc/lexparser/README.txt`) Copy Section

Stanford Lexicalized Parser v4.2.1 - 2021-05-05 Copy Chapter

CHANGES Copy Chapter

Doc Lexparser README Dependencies (`doc/lexparser/README_dependencies.txt`) Copy Section

UNIVERSAL/STANFORD DEPENDENCIES. Stanford Parser v3.7.0 Copy Chapter

Doc Lexparser Pos Sentences (`doc/lexparser/pos-sentences.txt`) Copy Section

Doc Lexparser Testsent (`doc/lexparser/testsent.txt`) Copy Section

Doc Loglinear ARCH (`doc/loglinear/ARCH.txt`) Copy Section

Doc Loglinear OPTIMIZATION (`doc/loglinear/OPTIMIZATION.txt`) Copy Section

Doc Loglinear QUICKSTART (`doc/loglinear/QUICKSTART.txt`) Copy Section

GraphicalModel model = new GraphicalModel(); Copy Chapter

}); Copy Chapter

}); Copy Chapter

model.getVariableMetadataByReference(7).put(LogLikelihoodFunction.VARIABLE_TRAINING_VALUE, "0"); Copy Chapter

ConcatVector weights = opt.optimize(trainingSet, new LogLikelihoodFunction(), new ConcatVector(0), 0.1); Copy Chapter

int[] mapAssignment = tree.calculateMAP(); double[][] marginalProbabilities = tree.calculateMarginals(); Copy Chapter

Doc Loglinear README (`doc/loglinear/README.txt`) Copy Section

Doc Loglinear TESTING (`doc/loglinear/TESTING.txt`) Copy Section

Doc Ner README (`doc/ner/README.txt`) Copy Section

Stanford NER - v4.2.0 - 2020-11-17 Copy Chapter

CHANGES Copy Chapter

Doc Ner TODO 2009 (`doc/ner/TODO-2009.txt`) Copy Section

Doc Ner Ner Manifest (`doc/ner/ner-manifest.txt`) Copy Section

Doc Ner Performance (`doc/ner/performance.txt`) Copy Section

Doc Ner Sample W Time (`doc/ner/sample-w-time.txt`) Copy Section

Doc Ner Sample (`doc/ner/sample.txt`) Copy Section

Doc Openie README (`doc/openie/README.txt`) Copy Section

Doc Openie Openie Manifest (`doc/openie/openie-manifest.txt`) Copy Section

Doc Segmenter README Arabic (`doc/segmenter/README-Arabic.txt`) Copy Section

Stanford Arabic Segmenter - v4.2.0 - 2020-11-17 Copy Chapter

Doc Segmenter README Chinese (`doc/segmenter/README-Chinese.txt`) Copy Section

Stanford Chinese Segmenter - v4.2.0 - 2020-11-17 Copy Chapter

Doc Segmenter README (`doc/segmenter/README.txt`) Copy Section

LICENSE Copy Chapter

CHANGES Copy Chapter

Doc Tagger README Models (`doc/tagger/README-Models.txt`) Copy Section

English taggers Copy Chapter

Chinese tagger Copy Chapter

Arabic tagger Copy Chapter

French tagger Copy Chapter

German tagger Copy Chapter

Spanish tagger Copy Chapter

Doc Tagger README (`doc/tagger/README.txt`) Copy Section

QUICKSTART Copy Chapter

To tag a file using the pre-trained bidirectional model

To train a simple model

To test a model

Using models for French, German, and Spanish

CONTENTS Copy Chapter

THANKS Copy Chapter

CHANGES Copy Chapter

LICENSE Copy Chapter

CONTACT Copy Chapter

Doc Tagger Postagger Manifest (`doc/tagger/postagger-manifest.txt`) Copy Section

Doc Tagger Sample Input (`doc/tagger/sample-input.txt`) Copy Section

Doc Tagger Sample Output (`doc/tagger/sample-output.txt`) Copy Section

Doc Tregex README Gui (`doc/tregex/README-gui.txt`) Copy Section

Tregex GUI v4.2.0 - 2020-11-17 Copy Chapter

TREGEX GRAPHICAL USER INTERFACE (GUI) README Copy Chapter

LOADING TREEBANKS/TREE FILES Copy Chapter

PERFORMING TREGEX SEARCHES Copy Chapter

USING TSURGEON Copy Chapter

SAVING RESULTS Copy Chapter

Overview (README.md)

Build Instructions

Install by Gradle

Useful resources

Doc Classify README (`doc/classify/README.txt`)

Stanford Classifier v4.2.0 - 2020-11-17

CHANGES

Doc Corenlp README (`doc/corenlp/README.txt`)

Stanford CoreNLP - Stanford's Suite of NLP Tools

CHANGES

Doc Corenlp Input (`doc/corenlp/input.txt`)

Doc Lexparser README (`doc/lexparser/README.txt`)

Stanford Lexicalized Parser v4.2.1 - 2021-05-05

CHANGES

Doc Lexparser README Dependencies (`doc/lexparser/README_dependencies.txt`)

UNIVERSAL/STANFORD DEPENDENCIES. Stanford Parser v3.7.0

Doc Lexparser Pos Sentences (`doc/lexparser/pos-sentences.txt`)

Doc Lexparser Testsent (`doc/lexparser/testsent.txt`)

Doc Loglinear ARCH (`doc/loglinear/ARCH.txt`)

Doc Loglinear OPTIMIZATION (`doc/loglinear/OPTIMIZATION.txt`)

Doc Loglinear QUICKSTART (`doc/loglinear/QUICKSTART.txt`)

GraphicalModel model = new GraphicalModel();

});

});

model.getVariableMetadataByReference(7).put(LogLikelihoodFunction.VARIABLE_TRAINING_VALUE, "0");

ConcatVector weights = opt.optimize(trainingSet, new LogLikelihoodFunction(), new ConcatVector(0), 0.1);

int[] mapAssignment = tree.calculateMAP(); double[][] marginalProbabilities = tree.calculateMarginals();

Doc Loglinear README (`doc/loglinear/README.txt`)

Doc Loglinear TESTING (`doc/loglinear/TESTING.txt`)

Doc Ner README (`doc/ner/README.txt`)

Stanford NER - v4.2.0 - 2020-11-17

CHANGES

Doc Ner TODO 2009 (`doc/ner/TODO-2009.txt`)

Doc Ner Ner Manifest (`doc/ner/ner-manifest.txt`)

Doc Ner Performance (`doc/ner/performance.txt`)

Doc Ner Sample W Time (`doc/ner/sample-w-time.txt`)

Doc Ner Sample (`doc/ner/sample.txt`)

Doc Openie README (`doc/openie/README.txt`)

Doc Openie Openie Manifest (`doc/openie/openie-manifest.txt`)

Doc Segmenter README Arabic (`doc/segmenter/README-Arabic.txt`)

Stanford Arabic Segmenter - v4.2.0 - 2020-11-17

Doc Segmenter README Chinese (`doc/segmenter/README-Chinese.txt`)

Stanford Chinese Segmenter - v4.2.0 - 2020-11-17

Doc Segmenter README (`doc/segmenter/README.txt`)

LICENSE

CHANGES

Doc Tagger README Models (`doc/tagger/README-Models.txt`)

English taggers

Chinese tagger

Arabic tagger

French tagger

German tagger

Spanish tagger

Doc Tagger README (`doc/tagger/README.txt`)

QUICKSTART

CONTENTS

THANKS

CHANGES

LICENSE

CONTACT

Doc Tagger Postagger Manifest (`doc/tagger/postagger-manifest.txt`)

Doc Tagger Sample Input (`doc/tagger/sample-input.txt`)

Doc Tagger Sample Output (`doc/tagger/sample-output.txt`)

Doc Tregex README Gui (`doc/tregex/README-gui.txt`)

Tregex GUI v4.2.0 - 2020-11-17

TREGEX GRAPHICAL USER INTERFACE (GUI) README

LOADING TREEBANKS/TREE FILES

PERFORMING TREGEX SEARCHES

USING TSURGEON

SAVING RESULTS

MULTILANGUAGE SUPPORT

QUESTIONS

LICENSE

CHANGES

Doc Tregex README Tregex (`doc/tregex/README-tregex.txt`)

Tregex v4.2.0 - 2020-11-17

QUICKSTART

CONTENTS

TREGEX

MISCELLANEOUS