# Technical Documentation: stanfordnlp/CoreNLP > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/stanfordnlp/CoreNLP) ### Overview (README.md) # Stanford CoreNLP [](https://github.com/stanfordnlp/CoreNLP/actions/workflows/run-tests.yaml) [](https://mvnrepository.com/artifact/edu.stanford.nlp/stanford-corenlp) [](https://twitter.com/stanfordnlp/) [Stanford CoreNLP](http://stanfordnlp.github.io/CoreNLP/) provides a set of natural language analysis tools written in Java. It can take raw human language text input and give the base forms of words, their parts of speech, whether they are names of companies, people, etc., normalize and interpret dates, times, and numeric quantities, mark up the structure of sentences in terms of syntactic phrases or dependencies, and indicate which noun phrases refer to the same entities. It was originally developed for English, but now also provides varying levels of support for (Modern Standard) Arabic, (mainland) Chinese, French, German, Hungarian, Italian, and Spanish. Stanford CoreNLP is an integrated framework, which makes it very easy to apply a bunch of language analysis tools to a piece of text. Starting from plain text, you can run all the tools with just two lines of code. Its analyses provide the foundational building blocks for higher-level and domain-specific text understanding applications. Stanford CoreNLP is a set of stable and well-tested natural language processing tools, widely used by various groups in academia, industry, and government. The tools variously use rule-based, probabilistic machine learning, and deep learning components. The Stanford CoreNLP code is written in Java and licensed under the GNU General Public License (v2 or later). Note that this is the full GPL, which allows many free uses, but not its use in proprietary software that you distribute to others. ### Build Instructions Several times a year we distribute a new version of the software, which corresponds to a stable commit. During the time between releases, one can always use the latest, under development version of our code. Here are some helpful instructions to use the latest code: #### Provided build Sometimes we will provide updated jars here which have the latest version of the code. At present, [the current released version of the code](https://stanfordnlp.github.io/CoreNLP/#download) is our most recent released jar, though you can always build the very latest from GitHub HEAD yourself. #### Build with Ant 1. Make sure you have Ant installed, details here: [http://ant.apache.org/](http://ant.apache.org/) 2. Compile the code with this command: `cd CoreNLP ; ant` 3. Then run this command to build a jar with the latest version of the code: `cd CoreNLP/classes ; jar -cf ../stanford-corenlp.jar edu` 4. This will create a new jar called stanford-corenlp.jar in the CoreNLP folder which contains the latest code 5. The dependencies that work with the latest code are in CoreNLP/lib and CoreNLP/liblocal, so make sure to include those in your CLASSPATH. 6. When using the latest version of the code make sure to download the latest versions of the [corenlp-models](http://nlp.stanford.edu/software/stanford-corenlp-models-current.jar), [english-models](http://nlp.stanford.edu/software/stanford-english-corenlp-models-current.jar), and [english-models-kbp](http://nlp.stanford.edu/software/stanford-english-kbp-corenlp-models-current.jar) and include them in your CLASSPATH. If you are processing languages other than English, make sure to download the latest version of the models jar for the language you are interested in. #### Build with Maven 1. Make sure you have Maven installed, details here: [https://maven.apache.org/](https://maven.apache.org/) 2. If you run this command in the CoreNLP directory: `mvn package` , it should run the tests and build this jar file: `CoreNLP/target/stanford-corenlp-4.6.0.jar` 3. When using the latest version of the code make sure to download the latest versions of the [corenlp-models](http://nlp.stanford.edu/software/stanford-corenlp-models-current.jar), [english-extra-models](http://nlp.stanford.edu/software/stanford-english-extra-corenlp-models-current.jar), and [english-kbp-models](http://nlp.stanford.edu/software/stanford-english-kbp-corenlp-models-current.jar) and include them in your CLASSPATH. If you are processing languages other than English, make sure to download the latest version of the models jar for the language you are interested in. 4. If you want to use Stanford CoreNLP as part of a Maven project you need to install the models jars into your Maven repository. Below is a sample command for installing the Spanish models jar. For other languages just change the language name in the command. To install `stanford-corenlp-models-current.jar` you will need to set `-Dclassifier=models`. Here is the sample command for Spanish: `mvn install:install-file -Dfile=/location/of/stanford-spanish-corenlp-models-current.jar -DgroupId=edu.stanford.nlp -DartifactId=stanford-corenlp -Dversion=4.6.0 -Dclassifier=models-spanish -Dpackaging=jar` #### Models The models jars that correspond to the latest code can be found in the table below. Some of the larger (English) models -- like the shift-reduce parser and WikiDict -- are not distributed with our default models jar. These require downloading the English (extra) and English (kbp) jars. Resources for other languages require usage of the corresponding models jar. The best way to get the models is to use git-lfs and clone them from Hugging Face Hub. For instance, to get the French models, run the following commands: ``` # Make sure you have git-lfs installed # (https://git-lfs.github.com/) git lfs install git clone https://huggingface.co/stanfordnlp/corenlp-french ``` The jars can be directly downloaded from the links below or the Hugging Face Hub page as well. | Language | Model Jar | Last Updated | | --- | --- | --- | | Arabic | [download](https://nlp.stanford.edu/software/stanford-arabic-corenlp-models-current.jar) [(HF Hub)](https://huggingface.co/stanfordnlp/corenlp-arabic/tree/main) | 4.6.0 | | Chinese | [download](https://nlp.stanford.edu/software/stanford-chinese-corenlp-models-current.jar) [(HF Hub)](https://huggingface.co/stanfordnlp/corenlp-chinese/tree/main)| 4.6.0 | | English (extra) | [download](https://nlp.stanford.edu/software/stanford-english-extra-corenlp-models-current.jar) [(HF Hub)](https://huggingface.co/stanfordnlp/corenlp-english-extra/tree/main) | 4.6.0 | | English (KBP) | [download](https://nlp.stanford.edu/software/stanford-english-kbp-corenlp-models-current.jar) [(HF Hub)](https://huggingface.co/stanfordnlp/corenlp-english-kbp/tree/main) | 4.6.0 | | French | [download](https://nlp.stanford.edu/software/stanford-french-corenlp-models-current.jar) [(HF Hub)](https://huggingface.co/stanfordnlp/corenlp-french/tree/main) | 4.6.0 | | German | [download](https://nlp.stanford.edu/software/stanford-german-corenlp-models-current.jar) [(HF Hub)](https://huggingface.co/stanfordnlp/corenlp-german/tree/main) | 4.6.0 | | Hungarian | [download](https://nlp.stanford.edu/software/stanford-hungarian-corenlp-models-current.jar) [(HF Hub)](https://huggingface.co/stanfordnlp/corenlp-hungarian/tree/main) | 4.6.0 | | Italian | [download](https://nlp.stanford.edu/software/stanford-italian-corenlp-models-current.jar) [(HF Hub)](https://huggingface.co/stanfordnlp/corenlp-italian/tree/main)| 4.6.0 | | Spanish | [download](https://nlp.stanford.edu/software/stanford-spanish-corenlp-models-current.jar) [(HF Hub)](https://huggingface.co/stanfordnlp/corenlp-spanish/tree/main)| 4.6.0 | Thank you to [Hugging Face](https://huggingface.co/) for helping with our hosting! ### Install by Gradle If you don't know Gradle itself, see official site: https://gradle.org Write the following in your build.gradle according to [Maven Central](https://search.maven.org/artifact/edu.stanford.nlp/stanford-corenlp/4.5.5/jar): ```Gradle dependencies { implementation 'edu.stanford.nlp:stanford-corenlp:4.5.5' } ``` If you want to analyse English, add following: ```Gradle implementation "edu.stanford.nlp:stanford-corenlp:4.5.5:models" implementation "edu.stanford.nlp:stanford-corenlp:4.5.5:models-english" implementation "edu.stanford.nlp:stanford-corenlp:4.5.5:models-english-kbp" ``` If you use another version, replace "4.5.5" to a version you use. ### Useful resources You can find releases of Stanford CoreNLP on [Maven Central](https://search.maven.org/artifact/edu.stanford.nlp/stanford-corenlp/4.5.4/jar). You can find more explanation and documentation on [the Stanford CoreNLP homepage](http://stanfordnlp.github.io/CoreNLP/). For information about making contributions to Stanford CoreNLP, see the file [CONTRIBUTING.md](CONTRIBUTING.md). Questions about CoreNLP can either be posted on StackOverflow with the tag [stanford-nlp](http://stackoverflow.com/questions/tagged/stanford-nlp), or on the [mailing lists](https://nlp.stanford.edu/software/#Mail). --- ### Doc Classify README (`doc/classify/README.txt`) Stanford Classifier v4.2.0 - 2020-11-17 ------------------------------------------------- Copyright (c) 2003-2012 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. Original core classifier code and command line interface by Dan Klein and Chris Manning. Support code, additional features, etc. by Kristina Toutanova, Jenny Finkel, Galen Andrew, Joseph Smarr, Chris Cox, Roger Levy, Rajat Raina, Pi-Chuan Chang, Marie-Catherine de Marneffe, Eric Yeh, Anna Rafferty, and John Bauer. This release prepared by John Bauer. This package contains a maximum entropy classifier. For more information about the classifier, point a web browser at the included javadoc directory, starting at the Package page for the edu.stanford.nlp.classify package, and looking also at the ColumnDataClassifier class documentation therein. This software requires Java 8 (JDK 1.8.0+). (You must have installed it separately. Check the command "java -version".) QUICKSTART COMMAND LINE INTERFACE To classify the included example dataset cheeseDisease (in the examples directory), type the following at the command line while in the main classifier directory: java -cp "*:." edu.stanford.nlp.classify.ColumnDataClassifier -prop examples/cheese2007.prop This will classify the included test data, cheeseDisease.test, based on the probability that each example is a cheese or a disease, as calculated by a linear classifier trained on cheeseDisease.train. The cheese2007.prop file demonstrates how features are specified. The first feature in the file, useClassFeature, indicates that a feature should be used based on class frequency in the training set. Most other features are calculated on specific columns of data in your tab-delimited text file. For example, "1.useNGrams=true" indicates that n-gram features should be created for the values in column 1 (numbering begins at 0!). Note that you must specify, for example, "true" in "1.useNGrams=true"; "1.useNGrams" alone will not cause n-gram features to be created. N-gram features are character subsequences of the string in the column, for example, "t", "h", "e", "th", "he", "the" from the word "the". You can also specify various other kinds of features such as just using the string value as a categorical feature (1.useString=true) or splitting up a longer string into bag-of-words features (1.splitWordsRegexp=[ ] 1.useSplitWords=true). The prop file also allows a choice of printing and optimization options, and allows you to specify training and test files (e.g., in cheese2007.prop under the "Training input" comment). See the javadoc for ColumnDataClassifier within the edu.stanford.nlp.classify package for more information on these and other options. Another included dataset is the iris dataset which uses numerical features to separate types of irises. To specify the use of a real-valued rather than categorical feature, you can use one or more of "realValued", "logTransform", or "logitTransform" for a given column. "realValued" adds the number in the given column as a feature value, while the transform options perform either a log or a logit transform on the value first. The format of these feature options is the same as for categorical features; for instance, iris2007.prop shows the use of real valued features such as "2.realValued=true". CLASSIFYING YOUR OWN DATA FILES To classify your own data files, they should be in tab-delimited text from which to make features as shown above, SVMLight format, or as tab-delimited text with the exact feature values you would like. Then specify the train and test files on the command line or in a .prop file with "trainFile=/myPath/myTrainFile.train" and "testFile==/myPath/myTestFile.test". You can also create a serialized classifier using the serializeTo option followed by a file path. CODE EXAMPLES You can also directly use the classes in this package to train classifiers within other programs. An example of this is shown in ClassifierExample, in the package edu.stanford.nlp.classify. This class demonstrates how to build a classifier factory, creating a classifier and setting various parameters in the classifier, training the classifier, and finally testing the classifier on a different data set. NO GUI This package does not provide a graphical user interface. The classifier is accessible only via the command line or programmatically. LICENSE // Stanford Classifier // Copyright (c) 2003-2007 The Board of Trustees of // The Leland Stanford Junior University. All Rights Reserved. // // This program is free software; you can redistribute it and/or // modify it under the terms of the GNU General Public License // as published by the Free Software Foundation; either version 2 // of the License, or (at your option) any later version. // // This program is distributed in the hope that it will be useful, // but WITHOUT ANY WARRANTY; without even the implied warranty of // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the // GNU General Public License for more details. // // You should have received a copy of the GNU General Public License // along with this program. If not, see http://www.gnu.org/licenses/ . // // For more information, bug reports, fixes, contact: // Christopher Manning // Dept of Computer Science, Gates 2A // Stanford CA 94305-9020 // USA // java-nlp-support@lists.stanford.edu // https://nlp.stanford.edu/software/classifier.html ------------------------- CHANGES ------------------------- 2020-11-17 4.2.0 Update for compatibility 2020-05-10 4.0.0 Update for compatibility 2018-10-16 3.9.2 Update for compatibility 2018-02-27 3.9.1 Updated for compatibility 2016-10-31 3.7.0 Update for compatibility 2015-12-09 3.6.0 Update for compatibility 2015-04-20 3.5.2 Update for compatibility 2015-01-29 3.5.1 New input/output options, support for GloVe word vectors 2014-10-26 3.5.0 Upgrade to Java 1.8 2014-08-27 3.4.1 Update for compatibility 2014-06-16 3.4 Update for compatibility 2014-01-04 3.3.1 Bugfix release 2013-11-12 3.3.0 Update for compatibility 2013-06-19 3.2.0 Update for compatibility 2013-04-04 2.1.8 Update to maintain compatibility 2012-11-11 2.1.7 new pair-of-words features 2012-07-09 2.1.6 Minor bug fixes 2012-05-22 2.1.5 Re-release to maintain compatibility with other releases 2012-03-09 2.1.4 Bugfix for svmlight format 2011-12-16 2.1.3 Re-release to maintain compatibility with other releases 2011-09-14 2.1.2 Change ColumnDataClassifier to be an object with API rather than static methods; ColumnDataClassifier thread safe 2011-06-15 2.1.1 Re-release to maintain compatibility with other releases 2011-05-15 2.1 Updated with more documentation 2007-08-15 2.0 New command line interface, substantial increase in options and features (updated on 2007-09-28 with a bug fix) 2003-05-26 1.0 Initial release --- ### Doc Corenlp README (`doc/corenlp/README.txt`) Stanford CoreNLP - Stanford's Suite of NLP Tools ------------------------------------------------ Copyright © 2009-2020 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. DOCUMENTATION Please look at the URL below for documentation for Stanford CoreNLP: https://nlp.stanford.edu/software/corenlp.html LICENSE // // StanfordCoreNLP -- a suite of NLP tools // Copyright © 2009-2020 The Board of Trustees of // The Leland Stanford Junior University. All Rights Reserved. // // This program is free software; you can redistribute it and/or // modify it under the terms of the GNU General Public License // as published by the Free Software Foundation; either version 2 // of the License, or (at your option) any later version. // // This program is distributed in the hope that it will be useful, // but WITHOUT ANY WARRANTY; without even the implied warranty of // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the // GNU General Public License for more details. // // You should have received a copy of the GNU General Public License // along with this program. If not, see http://www.gnu.org/licenses/ . // // For more information, bug reports, fixes, contact: // Christopher Manning // Dept of Computer Science, Gates 2A // Stanford CA 94305-9020 // USA // --------------------------------- CHANGES --------------------------------- 2026-09-24 4.6.0 Security fixes, enhanced-graph Semgrex queries, SemgrexStats, SUTime bugfixes, and dependency upgrades 2025-06-06 4.5.10 Additional Semgrex / SSurgeon features and remove patterns/lucene because of security warning 2025-03-23 4.5.9 Additional Semgrex / SSurgeon features and a security patch recommended for deserialization 2024-12-11 4.5.8 Update paths to reflect an internal file server dying. Add brat demo files to distrib to reduce dependencies on nlp.stanford.edu. Update the con->dep converter for UD. Update some external libraries for security vulnerabilities 2024-04-18 4.5.7 Add the f1 scores to the parser output, improve the dependency converter 2024-01-31 4.5.6 Minor lemmatizer and tokenizer fixes 2023-09-06 4.5.5 Fix up some SD and UD conversion errors. Add SceneGraph to the server. Fix Tregex optional bug. "fourty" and forty (40) days in SUTime. 2023-03-15 4.5.4 Minor updates to Ssurgeon, include the Morphology CLI 2023-03-10 4.5.3 Fix the collinizer. Add an Ssurgeon interface 2023-01-19 4.5.2 Update package dependencies to avoid security problems. Add protobuf CLI tools for use in Python or elsewhere 2022-08-29 4.5.1 Bugfixes to tokenizer and semgrex 2022-07-21 4.5.0 Tokenizer and lemmatizer upgrades, along with a new tsurgeon operation and some bugfixes 2022-01-20 4.4.0 Fix issue with Italian depparse, tsurgeon CLI, fix security issues, bug fixes 2021-10-14 4.3.1 Minor bugfixes to German umlaut tokenization, CDC tokenizer offset, Hungarian model package 2021-09-26 4.3.0 Add trained tokenizer from corenlp-it, add Italian and Hungarian pipelines using data from FBK, UD, Szeged, NYTK, and SPMRL. Better emoji support in the PTB tokenizer 2021-05-14 4.2.2 Fix issue with demo. 2021-05-05 4.2.1 Fix Turkish locale bug, QuoteAnnotator crash fixes, smaller srparser models, improvements to enhanced UD converter, Updated dependencies (istack, protobuf), batch processing of semgrex & enhancer requests when using stanza 2020-11-16 4.2.0 Bug fixes, Retrained English parser models with improved trees, Updated dependencies (ejml, junit, jflex), Speed up loading Wikidict annotator, New features for server handling of tokensregex and tregex requests, Release built directly from GitHub repo 2020-07-31 4.1.0 Improved server interface, improved memory usage of sutime, spanish tokenization upgrades 2020-04-19 4.0.0 Changed to UDv2 tokenization ("new" LDC Treebank, for English); handles multi-word-tokens; improved UDv2-based taggers and parsers for English, French, German, Spanish; new French NER; new Chinese segmenter; library updates, bug fixes 2018-10-05 3.9.2 improved NER pipeline and entity mention confidences; support for Java 11; new POS models for English; 4 methods for setting document dates; tokenizer improvements; CoreNLP runs as filter from stdin to stdout; bug fixes 2018-02-27 3.9.1 Bug fixes, minor enhancements 2018-01-31 3.9.0 Spanish KBP and new dependency parse model, wrapper API for data, quote attribution improvements, easier use of coref info, bug fixes 2017-06-09 3.8.0 Web service annotator, discussion forum handling, new French and Spanish models 2016-10-31 3.7.0 KBP Annotator, improved coreference, Arabic pipeline 2015-12-09 3.6.0 Improved coreference, OpenIE integration, Stanford CoreNLP server 2015-04-20 3.5.2 Switch to Universal dependencies, add Chinese coreference system to CoreNLP 2015-01-29 3.5.1 NER, dependency parser, SPIED improvements; general bugfixes 2014-10-26 3.5.0 Upgrade to Java 1.8; add annotators for dependency parsing and relation extraction 2014-08-27 3.4.1 Add Spanish models 2014-06-16 3.4 Add shift reduce parser 2014-01-04 3.3.1 Bugfix release 2013-11-12 3.3.0 Add sentiment model, minor sutime improvements 2013-06-19 3.2.0 New RNN parser model, more efficient tagger 2013-04-04 1.3.5 Speed improvements, coref improvements, Chinese version, -nthreads option 2012-11-12 1.3.4 Improved ner model and dependency code, now possible to change annotator pool for later StanfordCoreNLP objects 2012-07-09 1.3.3 Minor bug fixes 2012-05-22 1.3.2 Improvements to sutime 2012-03-09 1.3.1 Now supports caseless models (available as DLC) 2011-12-16 1.3.0 Threadsafe! Bugs in time annotation fixed 2011-09-14 1.2.0 Time expression recognizer added to ner annotator Output bugfixes Parser can now substitute for tagger 2011-06-19 1.1.0 Improved coref release 2011-05-15 1.0.4 More efficient dcoref data structure Supports already-tokenized input text 2011-04-17 1.0.3 Compatible with other releases Support loading arbitrary annotators Tagger bug fixes, such as "EOS" token 2010-11-11 1.0.2 Remove wn.jar 2010-11-11 1.0.1 Add xml removal 2010-10-07 1.0 Initial release --- ### Doc Corenlp Input (`doc/corenlp/input.txt`) Stanford University is located in California. It is a great university, founded in 1891. --- ### Doc Lexparser README (`doc/lexparser/README.txt`) Stanford Lexicalized Parser v4.2.1 - 2021-05-05 ----------------------------------------------- Copyright (c) 2002-2020 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. Original core parser code by Dan Klein. Support code, additional modules, languages, features, internationalization, compaction, typed dependencies, etc. by Christopher Manning, Roger Levy, Teg Grenager, Galen Andrew, Marie-Catherine de Marneffe, Jenny Finkel, Spence Green, Bill MacCartney, Anna Rafferty, Huihsin Tseng, Pi-Chuan Chang, Wolfgang Maier, Richard Eckart, Richard Socher, John Bauer, Sebastian Schuster, and Jon Gauthier. This release was prepared by Jason Bolton. This package contains 6 parsers: a high-accuracy unlexicalized PCFG; a lexicalized dependency parser; a factored model, where the estimates of dependencies and an unlexicalized PCFG are jointly optimized to give a lexicalized PCFG treebank parser; a TreeRNN parser, where recursive neural networks trained with semantic word vectors are used to score parse trees; a Shift-Reduce Constituency Parser; and a transition-based neural dependency parser. Also included are grammars for various languages for use with these parsers. For more information about the parser API, point a web browser at the included javadoc directory (use the browser's Open File command to open the index.html file inside the javadoc folder). Start by looking at the Package page for the edu.stanford.nlp.parser.lexparser package, and then look at the page for the LexicalizedParser class documentation therein, particularly documentation of the main method. Secondly, you should also look at the Parser FAQ on the web: https://nlp.stanford.edu/software/parser-faq.html This software requires Java 8 (JDK 1.8.0+). (You must have installed it separately. Check that the command "java -version" works and gives 1.8+.) QUICKSTART UNIX COMMAND-LINE USAGE On a Unix system you should be able to parse the English test file with the following command: ./lexparser.sh data/testsent.txt This uses the PCFG parser, which is quick to load and run, and quite accurate. [Notes: it takes a few seconds to load the parser data before parsing begins; continued parsing is quicker. To use the lexicalized parser, replace englishPCFG.ser.gz with englishFactored.ser.gz in the lexparser.sh script and use the flag -mx600m to give more memory to java.] WINDOWS GUI USAGE On a Windows system, assuming that java is on your PATH, you should be able to run a parsing GUI by double-clicking on the lexparser-gui.bat icon, or giving the command lexparser-gui in this directory from a command prompt. Click Load File, Browse, and navigate to and select testsent.txt in the top directory of the parser distribution. Click Load Parser, Browse, and select the models jar, also in the top directory of the parser distribution. From the models jar, select englishPCFG.ser.gz. Click Parse to parse the first sentence. NEURAL NETWORK DEPENDENCY PARSER USAGE To use the neural net dependency parser, issue the following command: java -Xmx2g -cp "*" edu.stanford.nlp.parser.nndep.DependencyParser \ -model edu/stanford/nlp/models/parser/nndep/english_UD.gz \ -textFile data/english-onesent.txt -outFile data/english-onesent.txt.out The output will be written to data/english-onesent.txt.out If you want to run on a language other than English, you will need to use a language specific POS tagger. Here is an example for Chinese: java -Xmx2g -cp "*" edu.stanford.nlp.parser.nndep.DependencyParser \ -model edu/stanford/nlp/models/parser/nndep/UD_Chinese.gz \ -tagger.model edu/stanford/nlp/models/pos-tagger/chinese-distsim.tagger \ -textFile data/chinese-onesent-utf8.txt -outFile data/chinese-onesent-utf8.txt.out OTHER USE CASES The GUI is also available under Unix: lexparser-gui.sh Under Mac OS X, you can double-click on lexparser-gui.command to invoke the GUI. The command-line version works on all platforms. Use lexparser.bat to run it under Windows. The GUI is only for exploring the parser. It does not allow you to save output. You need to use the command-line program or programmatic API to do serious work with the parser. ADDITIONAL GRAMMARS The parser is supplied with several trained grammars. There are English grammars based on the standard LDC Penn Treebank WSJ training sections 2-21 (wsj*), and ones based on an augmented data set, better for questions, commands, and recent English and biomedical text (english*). All grammars are located in the included models jar. (If you'd like to have grammar files like in older versions of the parser, you can get them by extracting them from the jar file with the 'jar -xf' command.) MULTILINGUAL PARSING In addition to the English grammars, the parser comes with trained grammars for Arabic, Chinese, French, and German. To parse with these grammars, run lexparser-lang.sh with no arguments to see usage instructions. You can change language-specific settings passed to the parser by modifying lexparser_lang.def. You can also train and evaluate new grammars using: lexparser-lang-train-test.sh To see how we trained the grammars supplied in this distribution, see bin/makeSerialized.csh You will not be able to run this script (since it uses Stanford-specific file paths), but you should be able to see what we did. Arabic Trained on parts 1-3 of the Penn Arabic Treebank (ATB) using the pre-processing described in (Green and Manning, 2010). The default input encoding is UTF-8 Arabic script. You can convert text in Buckwalter encoding to UTF-8 with the package edu.stanford.nlp.international.arabic.Buckwalter which is included in stanford-parser.jar. The parser *requires* segmentation and tokenization of raw text per the ATB standard prior to parsing. You can generate this segmentation and tokenization with the Stanford Word Segmenter, which is available separately at: https://nlp.stanford.edu/software/segmenter.html Chinese There are Chinese grammars trained just on mainland material from Xinhua and more mixed material from the LDC Chinese Treebank. The default input encoding is GB18030. French The standalone parser distribution comes with a neural dependency parser model trained on the French-GSD data set (version 2.2). The standalone parser distribution does not include functionality for producing UD 2.2 tokenization, so pre-tokenized text (text tokenized by whitespace) must be provided when running the neural dependency parser, and the "-tokenized" flag must be used. Example command: java -Xmx2g -cp "*" edu.stanford.nlp.parser.nndep.DependencyParser \ -model edu/stanford/nlp/models/parser/nndep/UD_French.gz \ -tagger.model edu/stanford/nlp/models/pos-tagger/french-ud.tagger \ -tokenized -textFile example.txt -outFile example.txt.out Note that "example.txt" should contain UD 2.2 tokens, separated by whitespace. The only provided French constituency parser is a shift-reduce parser. At this time running the shift-reduce parser on French text requires running a pipeline with the full Stanford CoreNLP package. To use the shift-reduce constituency parser on text and the UD 2.2 tokenization, upgrade to the full Stanford CoreNLP package and run a French pipeline. German The constituency parser was trained on the Negra corpus. Details are included in (Rafferty and Manning, 2008). This parser expects UD 2.2 tokenization. Input text files must be UD 2.2 tokens separated by whitespace. The "-tokenized" flag must be used. The neural dependency parser was trained on the German-GSD data set (version 2.2). The standalone parser distribution does not include functionality for producing UD 2.2 tokenization, so pre-tokenized text (text tokenized by whitespace) must be provided when running the neural dependency parser, and the "-tokenized" flag must be used. Example command: java -Xmx2g -cp "*" edu.stanford.nlp.parser.nndep.DependencyParser \ -model edu/stanford/nlp/models/parser/nndep/UD_German.gz \ -tagger.model edu/stanford/nlp/models/pos-tagger/german-ud.tagger \ -tokenized -textFile example.txt -outFile example.txt.out German shift reduce parsers are available, but running them on input German text requires the use of a full Stanford CoreNLP pipeline. The lower accuracy lexicalized parsers have options for running on input text. To use the shift-reduce constituency parser on text and the UD 2.2 tokenization, upgrade to the full Stanford CoreNLP package and run a German pipeline. Spanish The constituency parser was trained on the Spanish AnCora treebank and LDC provided discussion forum and newswire treebanks. This parser expects UD 2.0 tokenization. The neural dependency parser was trained on the Spanish AnCora data set (version 2.0). The standalone parser distribution does not include functionality for producing UD 2.0 tokenization, so pre-tokenized text (text tokenized by whitespace) must be provided when running the neural dependency parser, and the "-tokenized" flag must be used. Example command: java -Xmx2g -cp "*" edu.stanford.nlp.parser.nndep.DependencyParser \ -model edu/stanford/nlp/models/parser/nndep/UD_Spanish.gz \ -tagger.model edu/stanford/nlp/models/pos-tagger/spanish-ud.tagger \ -tokenized -textFile example.txt -outFile example.txt.out Spanish shift reduce parsers are available, but running them on input Spanish text requires the use of a full Stanford CoreNLP pipeline. The lower accuracy lexicalized parsers have options for running on input text. To use the shift-reduce constituency parser on text and the UD 2.2 tokenization, upgrade to the full Stanford CoreNLP package and run a Spanish pipeline. TREEBANK PREPROCESSING The pre-processed versions of the ATB described in (Green and Manning, 2010) and the FTB described in (Green et al., 2011) can be reproduced using the TreebankPreprocessor included in this release. The configuration files are located in /conf. For example, to create the ATB data, run: bin/run-tb-preproc -v conf/atb-latest.conf Note that you'll need to update the conf file paths to your local treebank distributions as the data is not distributed with the parser. You'll also need to set the classpath in the cmd_line variable of run-tb-preproc. The TreebankPreprocessor conf files support various options, which are documented in edu.stanford.nlp.international.process.ConfigParser EVALUATION METRICS The Stanford parser comes with Java implementations of the following evaluation metrics: Dependency Labeled Attachment Evalb (Collins, 1997) -Includes per-category evaluation with the -c option Leaf Ancestor (Sampson and Babarczy, 2003) -Both micro- and macro-averaged score Tagging Accuracy See the usage instructions and javadocs in the requisite classes located in edu.stanford.nlp.parser.metrics. UNIVERSAL DEPENDENCIES vs. STANFORD DEPENDENCIES Since v3.5.2 the default dependency representation is the new Universal Dependencies representation. Universal Dependencies were developed with the goal of being a cross-linguistically valid representation. Note that some constructs such as prepositional phrases are now analyzed differently and that the set of relations was updated. Please look at the Universal Dependencies documentation for more information: http://www.universaldependencies.org The parser also still supports the original Stanford Dependencies representation as described in the StanfordDependenciesManual.pdf. Use the flag -originalDependencies to obtain original Stanford Dependencies. LICENSE // StanfordLexicalizedParser -- a probabilistic lexicalized NL CFG parser // Copyright (c) 2002-2020 The Board of Trustees of // The Leland Stanford Junior University. All Rights Reserved. // // This program is free software; you can redistribute it and/or // modify it under the terms of the GNU General Public License // as published by the Free Software Foundation; either version 2 // of the License, or (at your option) any later version. // // This program is distributed in the hope that it will be useful, // but WITHOUT ANY WARRANTY; without even the implied warranty of // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the // GNU General Public License for more details. // // You should have received a copy of the GNU General Public License // along with this program. If not, see http://www.gnu.org/licenses/ . // // For more information, bug reports, fixes, contact: // Christopher Manning // Dept of Computer Science, Gates 2A // Stanford CA 94305-9020 // USA // parser-support@lists.stanford.edu // https://nlp.stanford.edu/downloads/lex-parser.html --------------------------------- CHANGES --------------------------------- 2021-05-05 4.2.1 Reduce size of srparser models 2020-11-17 4.2.0 Retrain English models with treebank fixes 2020-05-22 4.0.0 Model tokenization updated to UDv2.0 2018-10-16 3.9.2 Update for compatibility 2018-02-27 3.9.1 new French and Spanish UD models, misc. UD enhancements, bug fixes 2017-06-09 3.8.0 Updated for compatibility 2016-10-31 3.7.0 new UD models 2015-12-09 3.6.0 Updated for compatibility 2015-04-20 3.5.2 Switch to universal dependencies 2015-01-29 3.5.1 Dependency parser improvements; general bugfixes 2014-10-26 3.5.0 Upgrade to Java 1.8; add neural-network dependency parser 2014-08-27 3.4.1 Add Spanish models 2014-06-16 3.4 Shift-reduce parser 2014-01-04 3.3.1 Bugfix release, dependency improvements 2013-11-12 3.3.0 Remove the attr dependency, add imperatives to English training data 2013-06-19 3.2.0 New RNN model for WSJ and English with improved test set accuracy, rel dependency removed 2013-04-05 2.0.5 Dependency improvements, ctb7 model, -nthreads option 2012-11-12 2.0.4 Dependency speed improvements; other dependency changes 2012-07-09 2.0.3 Minor bug fixes 2012-05-22 2.0.2 Supports adding extra data in non-tree format 2012-03-09 2.0.1 Caseless English model added, ready for maven 2012-01-11 2.0.0 Threadsafe! 2011-09-14 1.6.9 Added some imperatives to the English training data; added root dependency. 2011-06-15 1.6.8 Added French parser and leaf ancestor evaluation metric; reorganized distribution; new data preparation scripts; rebuilt grammar models; other bug fixes 2011-05-15 1.6.7 Minor bug fixes 2011-04-17 1.6.6 Compatible with tagger, corenlp and tregex. 2010-10-30 1.6.5 Further improvements to English Stanford Dependencies and other minor changes 2010-08-16 1.6.4 More minor bug fixes and improvements to English Stanford Dependencies and question parsing 2010-07-09 1.6.3 Improvements to English Stanford Dependencies and question parsing, minor bug fixes 2010-02-25 1.6.2 Improvements to Arabic parser models, and to English and Chinese Stanford Dependencies 2008-10-19 1.6.1 Slightly improved Arabic, German and Stanford Dependencies 2007-08-18 1.6 Added Arabic, k-best PCCFG parsing; improved English grammatical relations 2006-05-30 1.5.1 Improved English and Chinese grammatical relations; fixed UTF-8 handling 2005-07-20 1.5 Added grammatical relations output; fixed bugs introduced in 1.4 2004-03-24 1.4 Made PCFG faster again (by FSA minimization); added German support 2003-09-06 1.3 Made parser over twice as fast; added tokenization options 2003-07-20 1.2 Halved PCFG memory usage; added support for Chinese 2003-03-25 1.1 Improved parsing speed; included GUI, improved PCFG grammar 2002-12-05 1.0 Initial release --- ### Doc Lexparser README Dependencies (`doc/lexparser/README_dependencies.txt`) UNIVERSAL/STANFORD DEPENDENCIES. Stanford Parser v3.7.0 ----------------------------------------------------------- IMPORTANT: Starting with version 3.5.2 the default dependencies representation output by the Stanford Parser is the new Universal Dependencies Representation. Universal Dependencies were developed with the goal of being a cross-linguistically valid representation. Note that some constructions such as prepositional phrases are now analyzed differently and that the set of relations was updated. The online documentation of English Universal Dependencies at http://www.universaldependencies.org should be consulted for the current set of dependency relations. The parser and converter also still support the original Stanford Dependencies as described in the Stanford Dependencies manual. Use the flag -originalDependencies to obtain the original Stanford Dependencies. Note, however, that we are no longer maintaining the SD converter or representation and we therefore recommend to use the Universal Dependencies representation for any new projects. The manual for the English version of the Stanford Dependencies representation: StanfordDependenciesManual.pdf should be consulted for the set of dependency relations in the original Stanford Dependencies representation and the correct commands for generating Stanford Dependencies together with any of the Stanford Parser, another parser, or a treebank. A typed dependencies representation is also available for Chinese. For the moment the documentation consists of the code, and a brief presentation in this paper: Pi-Chuan Chang, Huihsin Tseng, Dan Jurafsky, and Christopher D. Manning. 2009. Discriminative Reordering with Chinese Grammatical Relations Features. Third Workshop on Syntax and Structure in Statistical Translation. http://nlp.stanford.edu/pubs/ssst09-chang.pdf -------------------------------------- DEPENDENCIES SCHEMES For an overview of the original English Universal Dependencies schemes, please look at: Marie-Catherine de Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Universal Stanford dependencies: A cross-linguistic typology. 9th International Conference on Language Resources and Evaluation (LREC 2014). http://nlp.stanford.edu/~manning/papers/USD_LREC14_UD_revision.pdf and Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A Multilingual Treebank Collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). http://nlp.stanford.edu/pubs/nivre2016ud.pdf Please note, though, that some of the relations discussed in the first paper were subsequently updated and please refer to the online documentation at http://www.universaldependencies.org for an up to date documention of the set of relations. For an overview of the enhanced and enhanced++ dependency representations, please look at: Sebastian Schuster and Christopher D. Manning. 2016. Enhanced English Universal Dependencies: An Improved Representation for Natural Language Understanding Tasks. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). http://nlp.stanford.edu/~sebschu/pubs/schuster-manning-lrec2016.pdf For an overview of the original typed dependencies scheme, please look at: Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating Typed Dependency Parses from Phrase Structure Parses. 5th International Conference on Language Resources and Evaluation (LREC 2006). http://nlp.stanford.edu/~manning/papers/LREC_2.pdf For more discussion of the design principles, please see: Marie-Catherine de Marneffe and Christopher D. Manning. 2008. The Stanford typed dependencies representation. In Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation, pp. 1-8. http://nlp.stanford.edu/~manning/papers/dependencies-coling08.pdf These papers can be cited as references for the original English Stanford Dependencies and Enlgish Universal Dependencies. -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.7.0 Implementation of enhanced and enhanced++ dependency representations as described in Schuster and Manning (2016). Fixed concurrency issue. -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.5.2 Switch to Universal Dependencies as the default representation. Please see the Universal Dependencies documentation at http://www.universaldependencies.org for more information on the new relations. -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.5.1 A couple of small fixes were made, leading to ccomp and advcl being recognized in a couple of new environments. -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.4 One major change was made to the dependency taxonomy: - We decided to collapse together the two dependencies partmod and infmod, since they have similar function and mainly differ in the form of the verbal head, which is anyways recorded in the POS tag. Those two relations are removed from the taxonomy, and a new relation vmod covering the union of both was added. -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.3.1 A couple of fixes/improvements were made in the dependency conversion, and one change was made to the taxonomy of relations. - The partmod and infmod relations were deleted, and replaced with vmod for reduced, non-finite verbal modifiers. The distinction between these two relations can be recovered from the POS tag of the dependent. - A couple of improvements were made to the conversion, the largest one being recognizing pobj inside a PP not headed by something tagged as IN or TO. -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.3 Some fixes/improvements were made in the dependency conversion, and one change was made to the taxonomy of relations. - For currency amount expressions with a currency symbol like "$", it had previously been the case that "$" was the head, and then each number word modified it as a number. We realized that this was unnecessarily inconsistent. For the expression "two thousand dollars", "dollars" is the head, but "thousand" is a num modifier of it, and number is used for the parts of a number multi-word expression only. This analysis is now also used for cases with a currency symbol. E.g., "for $ 52.7 million": prep(for, $) num($, million) number(million, 52.7). Similarly, for "the $ 2.29 billion value", we changed the analysis from num(value, $) number($, billion) to amod(value, $) num($, billion). This corresponds to hwat you got for "a two dollar value". This is actually the most common change (at least on WSJ newswire!). - Remove the attr relation. Some cases disappear by making the question phrase of WHNP be NP questions the root. Others (predicative NP complements) become xcomp. - Less aggressive labeling of participial form VPs as xcomp. More of them are correctly labeled partmod (but occasionally a true xcomp is also mislabeled as partmod). - Small rule changes to recognize a few more ccomp and parataxis. -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v3.2, JUNE 2013 An improved dependency conversion means that our dependency trees are not always projective, one deletion was made from the taxonomy of relations, and various small converter fixes were made: - rel was removed. rel was originally used as the relation for an overt relativizer in a relative clause. But it was never a real grammatical relation, and we gradually started labeling easy cases as nsubj or dobj. In this release, rel is removed, pobj cases are also labeled, and the remaining hard cases are labeled as dep. - As a result of correctly labeling a pobj in questions and relative clauses, the converter now sometimes produces non-projective dependency trees (ones with crossing dependencies, if the words are laid out in their normal order in a line, and all dependency arcs are drawn above them). This is not a bug, it's an improvement in the generated dependencies, but you should be aware that Stanford Dependencies trees are now occasionally non-projective. (Some simple dependency parsing algorithms only produce projective dependency trees.) -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v2.0.5, MARCH 2013 We have begun a more major effort to improve the suitability and coverage of Stanford Dependencies on less formal text types, and to clean up a couple of the more quirky dependencies in the original set. These changes are still ongoing, but in this first installment, we have removed 3 dependencies and added 2: - abbrev was removed, and is now viewed as just a case of appos. - complm was removed, and is now viewed as just a case of mark. (This is consistent with an HPSG-like usage of mark.) - purpcl was removed, and is now viewed as just a case of advcl. - discourse was added. The lack of a dependency type for interjections was an omission even in the early versions, but it became essential as we expanded our consideration of informal text types. It is used for interjections, fillers, discourse markers and emoticons. - goeswith was added. In badly edited text, it is used to join the two parts of a word. A few other changes and improvements were also made, including improvements in the recognition of advcl. There has been a reduction of "dep" dependencies of about 14% on newswire (and higher on more informal text genres). -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v2.0.4, NOVEMBER 2012 A few minor changes and fixes were made: HYPH is now recognized, and treated as punctuation and clausal complements of adjectives (including comparatives) are recognized as ccomp. -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v1.6.9 This version adds an explicit root dependency in the set of dependencies returned. In the past, there had been no explicit representation of the root of the sentence in the set of dependencies returned, except in the CoNLL format output, which always showed the root. Now, there is always an explicit extra dependency that marks the sentence root, using a fake ROOT pseudoword with index 0. That is, the root is marked in this way: root(ROOT-0, depends-3) Otherwise there were only a couple of minute changes in the dependencies produced (appositions are now recognized in WHNPs!). -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- v1.6.8 This version includes only small fixes, principally addressing some gaps in the correct treatment of dependencies in inverted sentence (SQ and SINV) constructions, and some errors in the treatment of copulas in the presence of temporal NPs. -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- NOVEMBER 2010 - JANUARY 2011 Two changes were made to the taxonomy of dependencies. - measure (phrase modifier) was generalized and replaced by npadvmod (noun phrase adverbial modifier) which includes measure phrases and other adverbial uses of noun phrases. Temporal NPs (tmod) are now a subtype of npadvmod in the dependency hierarchy. - mwe (multi-word expression) is introduced for certain common function word dependencies for which another good analysis isn't easy to come by (and which were frequently dep before) such as "instead of" or "rather than". A new option has ben added to allow the copula to be treated as the head when it has an adjective or noun complement. The conversion software will now work fairly well with the David Vadas version of the treebank with extra noun phrase structure. (A few rare cases that are handled with the standard treebank aren't yet handled, but you will get better dependencies for compound nouns and multiword adjectival modifiers, etc.) Considerable improvements were made in the coverage of named dependencies. You should expect to see only about half as many generic "dep" dependencies as in version 1.6.4. -------------------------------------- CHANGES IN ENGLISH TYPED DEPENDENCIES CODE -- JUNE-AUGUST 2010 No new dependency relations have been introduced. There have been some significant improvements in the generated dependencies, principally covering: - Better resolution of nsubj and dobj long distance dependencies (but v1.6.4 fixes the overpercolation of dobj in v1.6.3) - Better handling of conjunction distribution in CCprocessed option - Correction of bug in v1.6.2 that made certain verb dependents noun dependents. - Better dependencies are generated for question structures (v1.6.4) - Other minor improvements in recognizing passives, adverbial modifiers, etc. --- ### Doc Lexparser Pos Sentences (`doc/lexparser/pos-sentences.txt`) People can butter their bread with a knife . People can butter/VB their bread with a knife . People can butter/NN their bread with a knife . People/NNS can/MD butter/VB their/PRP$ bread/NN with/IN a/DT knife/NN ./. People/NNS can/VB butter/NN their/PRP$ bread/NN with/IN a/DT knife/NN ./. People/NNS can/NN butter/NN their/PRP$ bread/NN with/IN a/DT knife/NN ./. People/NN can/NN butter/NN their/NN bread/NN with/NN a/NN knife/NN ./NN --- ### Doc Lexparser Testsent (`doc/lexparser/testsent.txt`) Scores of properties are under extreme fire threat as a huge blaze continues to advance through Sydney's north-western suburbs. Fires have also shut down the major road and rail links between Sydney and Gosford. The promotional stop in Sydney was everything to be expected for a Hollywood blockbuster - phalanxes of photographers, a stretch limo to a hotel across the Quay - but with one difference. A line-up of masseurs was waiting to take the media in hand. Never has the term "massaging the media" seemed so accurate. --- ### Doc Loglinear ARCH (`doc/loglinear/ARCH.txt`) Architectural Overview of edu.stanford.nlp.loglinear: The goal of this package is to provide fast, general structure log-linear modelling that's easy to use and to extend. The package is broken into three parts: model, inference, and learning Model contains all of the basic storage elements, as well as means to serialize and deserialize for both storage and network transit. Inference depends on model, and provides an implementation of the clique tree message passing algorithm for efficient exact inference in tree-structured graphs. Learning depends on inference and model, and provides a simple interface to efficient multithreaded batch learning, with an implementation of AdaGrad guarded by backtracking. We will go over model, then inference, then learning. ##################################################### Model module overview: ##################################################### *** ConcatVector: The key to the speed of loglinear is the ConcatVector class. ConcatVector provides a useful abstraction for NLP machine learning: a concatenation of vectors, treated as a single vector. The basic idea is to have each feature output a vector, which are then stored in a ConcatVector (or 'concatenated vector'). When the dot-product is taken of two ConcatVectors, the result is the sum of the dot product of each of the concatenated components of each vector in sequence. To write that out explicitly, if a feature ConcatVector f is composed of a number of vector f_i's, and a weight ConcatVector w is composed of a number of vector w_i's, then dot(f,w) is: \sum_i dot(f_i, w_i) This leaves us with two key advantages over a regular vector: each component can be individually tuned for sparsity, and each component has an isolated namespace and so can an individual feature vector can grow after training begins (say discovering a new word in a one-hot feature vector), and the weight vector will behave appropriately without hassle. *** NDArray We have a basic NDArray, which allows a standard iterator over possible assignments that creates a lot of int[] arrays on the heap, and a more elaborate iterator that saves GC by mutating a single array passed over. You'll see this used throughout the code in hot loops marked by an "//OPTIMIZATION" comment *** ConcatVectorTable ConcatVectorTable is a subclass of NDArray that we use to store factor tables for the log-linear graphical model, where each element of the table represents the features for one joint assignment to the variables the factor is associated with. In order to get a factor like you learned about in CS 228, each element of the table is dot-producted with weights. We don't do this at construction to allow a single set of GraphicalModel objects to be used throughout training. *** GraphicalModel GraphicalModel is a super stripped down implementation of a graphical model. It holds factors, represented by lists of neighbor indices and a ConcatVectorTable for features. It was deliberate to make all downstream annotations on the model (like observations for inference or observations for training) go into a HashMap. This is to maintain easy backwards compatibility with previous serialized versions as features change, and to make life more convenient for downstream algorithms that may be passing GraphicalModel objects across module or network boundaries, and don't want to create tons of little 'ride-along' objects that add annotations to the GraphicalModel. ##################################################### Inference module overview: ##################################################### *** TableFactor This is the traditional 'factor' datatype that you're used to hearing about from Daphne in 228 and "Probabilistic Graphical Models". It's a subclass of NDArray, and has fast operations for product and marginalize dataflows. It's the key building block for inference. *** CliqueTree This object takes a GraphicalModel at creation and provides high speed tree-shaped message passing inference for both exact marginals and exact MAP estimates. It exists as a new object for each GraphicalModel, rather than a static call for each model, to allow for cacheing some messages when repeated marginals are needed on only slightly changing models. ##################################################### Learning module overview: ##################################################### *** AbstractDifferentiableFunction This follows the Optimize.jl package convention of providing both gradient and function value in a single return value. *** LogLikelihoodFunction An implementation of AbstractDifferentiableFunction for calculating the log-likelihood of a log-linear model as given by a GraphicalModel. *** AbstractOnlineOptimizer This is the basic interface for online optimizers to follow. It is sketched out right now, but no implementations have been made yet. *** AbstractBatchOptimizer There is a fair amount of redundant complexity involved in writing an optimizer that needs to calculate the gradient on the entire batch of examples every update step. The work between threads must be carefully balanced so that the time between the first thread finishing, and the last, during which the CPU utilization is far less than 100%, is minimized. This is managed through rough estimating of the amount of work each item represents, and a perceptron style updating once the system is running, based on CPU time used for each thread. We also implement a convenience function here to allow the user to interrupt training early if they are happy with convergence to this point, since that involves some tricky Java threading to make work. *** BacktrackingAdaGradOptimizer This subclasses AbstractBatchOptimizer, and implements a simple AdaGrad gradient descent guarded by backtracking line search to maximize an AbstractDifferentiableFunction. --- ### Doc Loglinear OPTIMIZATION (`doc/loglinear/OPTIMIZATION.txt`) Optimization for loglinear was done last, and driven by realistic benchmarks wherever possible. There are 2 major benchmarks for loglinear so far: - Training CoNLL linear chain model with a mixture of dense and sparse features: org.stanford.nlp.loglinear.learning.CoNLLBenchmark - A bunch of microbenchmarks for ConcatVector: org.stanford.nlp.loglinear.model.ConcatVectorBenchmark The general findings so far have been: - The JNI doesn't speed up ConcatVector operations, even with AVX assembly, since the JIT seems to vectorize automatically, and occasionally will give the C code a copy of the arrays to mutate, which is absurdly slow. - The vast majority of training time is spent on feature vector related operations. Message passing takes less than 5% of total time. The rest of the time is evenly split between the initial dot product with weights to get factor values and the final summation of feature vectors to get the derivative of the log-likelihood. - Huge heap wastage occurs if you allow the featurizing code to run once and keep around ConcatVector's uselessly, which results in general slowdowns and long GC waits. Instead keeping featurizing code as thunks is much faster. - Making ConcatVector copy-on-write is a very valuable way to keep the GC from working too hard. A very common case is a clone vector that is only ever read from, so optimizing for that yielded a 50% drop in GC load. - A non-trivial fraction of time is wasted on poorly balanced work queues for different threads during batch gradient descent. Balancing more carefully yielded a 10% speedup. --- ### Doc Loglinear QUICKSTART (`doc/loglinear/QUICKSTART.txt`) loglinear package quickstart: First, read the ConcatVector section in ARCH.txt. To jump straight into working code, go read generateSentenceModel() in edu.stanford.nlp.loglinear.CoNLLBenchmark. ##################################################### Creating and featurizing a GraphicalModel ##################################################### To construct a GraphicalModel, which you'll need for training and inference, do the following: ------------- GraphicalModel model = new GraphicalModel(); ------------- Now, to add a factor to the model, you'll need to know two things: Who are the neighbor variables of this factor, and how many states do each of those variables possess? As an example, let's add a factor between variable 2 and variable 7, where variable 2 has 3 states, and variable 7 has 2 states. ------------- int[] neighbors = new int[]{2,7}; int[] neighborSizes = new int[]{3,2}; // This must appear in the same order as neighbors model.addFactor(neighbors, neighborSizes, (int[] assignment) -> { // TODO: In the next paragraph we'll discuss featurizing each possible assignment to neighbors return new ConcatVector(0); }); ------------- You'll also need to know how to featurize your new factors. That happens inside the closure you pass into the factor. The closure takes as an argument an assignment to the factor (in the same order as neighbors and neighborSizes) and expects in response a ConcatVector of features. Make sure your closures are idempotent! The system actually stores these closures, rather than their resulting ConcatVector's. This is an optimization. The GC in modern JVMs is tuned to collect large numbers of young objects, so creating new ConcatVectors every gradient calculation and then immediately disposing of them turns out to increase speed and dramatically decrease heap footprint and GC load. Here's how to create a feature closure: ------------- int[] neighbors = new int[]{2,7}; int[] neighborSizes = new int[]{3,2}; // This must appear in the same order as neighbors model.addFactor(neighbors, neighborSizes, (int[] assignment) -> { // This is how assignment[] is structured int variable2Assignment = assignment[0]; int variable7Assignment = assignment[1]; // Create a new ConcatVector with 2 segments: ConcatVector features = new ConcatVector(); // Add a dense feature as feature 0, of length 2 // (Dense features in ConcatVector's are mostly used for embeddings) features.setDenseComponent(0, new double[]{ variable2Assignment * 2 + variable7Assignment, variable2Assignment + variable7Assignment * 2, }); // Add a sparse (one-hot) feature as feature 1 int sparseIndex = variable2Assignment; double sparseValue = 1.0; features.setSparseComponent(1, sparseIndex, sparseValue); // Return our feature set to complete the closure return features; }); ------------- And that's all there is to it. Just repeat this several times to populate a GraphicalModel for whatever problem you've got. ##################################################### Training a set of weights ##################################################### Assuming you've got a bunch of GraphicalModel objects, there's not much you need to do to train a system. First, you need to provide labels to all your variables that the training system understands. To do this, you need to get the HashMap object that represents metadata, and put in labels that the LogLikelihood system understands. You must label every variable that is mentioned in any factor in your model. ------------- GraphicalModel model = new GraphicalModel(); // Omitted model construction, see previous section // Tell LogLikelihood to treat variable 2 as having the assignment "1" in training labels model.getVariableMetadataByReference(2).put(LogLikelihoodFunction.VARIABLE_TRAINING_VALUE, "1"); // Tell LogLikelihood to treat variable 7 as having the assignment "0" in training labels model.getVariableMetadataByReference(7).put(LogLikelihoodFunction.VARIABLE_TRAINING_VALUE, "0"); ------------- Once you've got an array of labeled models, training is pretty straightforward. We create an optimizer, pass it a LogLikelihoodFunction as the function to optimize, and the array of models as the data to optimize over. That returns us the optimal set of weights. ------------- GraphicalModel[] trainingSet = //omitted dataset construction; // Create the optimizer we will use AbstractBatchOptimizer opt = new BacktrackingAdaGradOptimizer(); // Call the optimizer, with a dataset, a function to optimize, initial weights, and l2 regularization constant ConcatVector weights = opt.optimize(trainingSet, new LogLikelihoodFunction(), new ConcatVector(0), 0.1); ------------- We can then use these weights for inference. ##################################################### Inference ##################################################### Inference is easy once we have a set of weights we want to use. We simply create a CliqueTree for the model we're trying to optimize and the weights we want, and then ask it for inference results. ------------- GraphicalModel model = // see previous section; ConcatVector weights = // see first section; CliqueTree tree = new CliqueTree(model, weights); int[] mapAssignment = tree.calculateMAP(); double[][] marginalProbabilities = tree.calculateMarginals(); ------------- The MAP assignment comes back as an array of assignments, where the assignment for variable 0 is at index 0, variable 1 is at index 1, and so forth. The marginalProbilities array is organized in the same way, except instead of an int for an assignment, there is an array of doubles, one for each possible assignment for the variable it represents, which represent global marginals. --- ### Doc Loglinear README (`doc/loglinear/README.txt`) For an explanation of how everything fits together, see ARCH.txt For a quick runnable object, go run edu.stanford.nlp.loglinear.CoNLLBenchmark in core's test package. For a tutorial, see QUICKSTART.txt For a brief overview of testing for this package (which was quite thorough), see TESTING.txt Look at OPTIMIZATION.txt for an overview of what exists, what's been done, and what we know about further optimizations. --- ### Doc Loglinear TESTING (`doc/loglinear/TESTING.txt`) The testing for the loglinear package uses functional invariants over randomly generated inputs, which in general yields much more durable software. This is aided by the JUnit port of Quickcheck. The general tactic for testing is to randomly generate inputs, then use the slow definitional approach on tractably small inputs and test that the output of our algorithms always matches exactly. The GitHub for the Quickcheck port is https://github.com/pholser/junit-quickcheck. The dependencies for that are in the lib/ folder in test1. Some of the general testing approaches are listed below: message passing -> tested against brute force factor multiplication and marginalization partition function -> tested against brute force multiplication and summation log likelihood gradient -> tested against definition of derivative optimization -> tested by making thousands of random perturbations around the function to check if any values are better concatVector -> tested against a non-sparse version table factor -> tested against functional invariants of results, and re-implementations using different algorithms --- ### Doc Ner README (`doc/ner/README.txt`) Stanford NER - v4.2.0 - 2020-11-17 ---------------------------------------------- This package provides a high-performance machine learning based named entity recognition system, including facilities to train models from supervised training data and pre-trained models for English. (c) 2002-2020. The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. Original CRF code by Jenny Finkel. Additional modules, features, internationalization, compaction, and support code by Christopher Manning, Dan Klein, Christopher Cox, Huy Nguyen Shipra Dingare, Anna Rafferty, and John Bauer. This release prepared by Jason Bolton. LICENSE The software is licensed under the full GPL v2+. Please see the file LICENCE.txt For more information, bug reports, and fixes, contact: Christopher Manning Dept of Computer Science, Gates 2A Stanford CA 94305-9020 USA java-nlp-support@lists.stanford.edu https://nlp.stanford.edu/software/CRF-NER.html CONTACT For questions about this distribution, please contact Stanford's JavaNLP group at java-nlp-user@lists.stanford.edu. We provide assistance on a best-effort basis. TUTORIAL Quickstart guidelines, primarily for end users who wish to use the included NER models, are below. For further instructions on training your own NER model, go to https://nlp.stanford.edu/software/crf-faq.html. INCLUDED SERIALIZED MODELS / TRAINING DATA The basic included serialized model is a 3 class NER tagger that can label: PERSON, ORGANIZATION, and LOCATION entities. It is included as english.all.3class.distsim.crf.ser.gz. It is trained on data from CoNLL, MUC6, MUC7, ACE, OntoNotes, and Wikipedia. Because this model is trained on both US and UK newswire, it is fairly robust across the two domains. We have also included a 4 class NER tagger trained on the CoNLL 2003 Shared Task training data that labels for PERSON, ORGANIZATION, LOCATION, and MISC. It is named english.conll.4class.distsim.crf.ser.gz . A third model is trained only on data from MUC and distinguishes between 7 different classes: english.muc.7class.distsim.crf.ser.gz. All of the serialized classifiers come in two versions, one trained to basically expected standard written English capitalization, and the other to ignore capitalization information. The case-insensitive versions of the three models available on the Stanford NER webpage. These models use a distributional similarity lexicon to improve performance (by between 1.5%-3% F-measure). The distributional similarity features make the models perform substantially better, but they require rather more memory. The distsim models are included in the release package. The nodistsim versions of the same models may be available on the Stanford NER webpage. Finally, we have models for other languages, including two German models, a Chinese model, and a Spanish model. The files for these models can be found at: http://nlp.stanford.edu/software/CRF-NER.html QUICKSTART INSTRUCTIONS This NER system requires Java 1.8 or later. Providing java is on your PATH, you should be able to run an NER GUI demonstration by just clicking. It might work to double-click on the stanford-ner.jar archive but this may well fail as the operating system does not give Java enough memory for our NER system, so it is safer to instead double click on the ner-gui.bat icon (Windows) or ner-gui.sh (Linux/Unix/MacOSX). Then, using the top option from the Classifier menu, load a CRF classifier from the classifiers directory of the distribution. You can then `either load a text file or web page from the File menu, or decide to use the default text in the window. Finally, you can now named entity tag the text by pressing the Run NER button. From a command line, you need to have java on your PATH and the stanford-ner.jar file and the lib directory in your CLASSPATH. (The way of doing this depends on your OS/shell.) The supplied ner.bat and ner.sh should work to allow you to tag a single file. For example, for Windows: ner file Or on Unix/Linux you should be able to parse the test file in the distribution directory with the command: java -mx600m -cp stanford-ner.jar:lib/* edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier classifiers/english.all.3class.distsim.crf.ser.gz -textFile sample.txt Here's an output option that will print out entities and their class to the first two columns of a tab-separated columns output file: java -mx600m -cp stanford-ner.jar:lib/* edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier classifiers/english.all.3class.distsim.crf.ser.gz -outputFormat tabbedEntities -textFile sample.txt > sample.tsv When run from a jar file, you also have the option of using a serialized classifier contained in the jar file. USING FULL STANFORD CORENLP NER FUNCTIONALITY This standalone distribution also allows access to the full NER capabilities of the Stanford CoreNLP pipeline. These capabilities can be accessed via the NERClassifierCombiner class. NERClassifierCombiner allows for multiple CRFs to be used together, and has options for recognizing numeric sequence patterns and time patterns with the rule-based NER of SUTime. Suppose one combines three CRF's CRF-1,CRF-2, and CRF-3 with the NERClassifierCombiner. When the NERClassiferCombiner runs, it will first apply the NER tags of CRF-1 to the text, then it will apply CRF-2's NER tags to any tokens not tagged by CRF-1 and so on. If the option ner.combinationMode is set to NORMAL (default), any label applied by CRF-1 cannot be applied by subsequent CRF's. For instance if CRF-1 applies the LOCATION tag, no other CRF's LOCATION tag will be used. If ner.combinationMode is set to HIGH_RECALL, this limitation will be deactivated. To use NERClassifierCombiner at the command-line, the jars in lib and stanford-ner.jar must be in the CLASSPATH. Here is an example command: java -mx2g edu.stanford.nlp.ie.NERClassifierCombiner -ner.model \ classifiers/english.conll.4class.distsim.crf.ser.gz,classifiers/english.muc.7class.distsim.crf.ser.gz \ -ner.useSUTime false -textFile sample-w-time.txt Let's break this down a bit. The flag "-ner.model" should be followed by a list of CRF's to be combined by the NERClassifierCombiner. Some serialized CRF's are provided in the classifiers directory. In this example the CRF's trained on the CONLL 4 class data and the MUC 7 class data are being combined. When the flag "-ner.useSUTime" is followed by "false", SUTime is shut off. You should note that when the "false" is omitted, the text "4 days ago" suddenly is tagged with DATE. These are the kinds of phrases SUTime can identify. NERClassifierCombiner can be run on different types of input as well. Here is an example which is run on CONLL style input: java -mx2g edu.stanford.nlp.ie.NERClassifierCombiner -ner.model \ classifiers/english.conll.4class.distsim.crf.ser.gz,classifiers/english.muc.7class.distsim.crf.ser.gz \ -map word=0,answer=1 -testFile sample-conll-file.txt It is crucial to include the "-map word=0,answer=1" , which is specifying that the input test file has the words in the first column and the answer labels in the second column. It is also possible to serialize and load an NERClassifierCombiner. This command loads the three sample crfs with combinationMode=HIGH_RECALL and SUTime=false, and dumps them to a file named test_serialized_ncc.ncc.ser.gz java -mx2g edu.stanford.nlp.ie.NERClassifierCombiner -ner.model \ classifiers/english.conll.4class.distsim.crf.ser.gz,classifiers/english.muc.7class.distsim.crf.ser.gz,\ classifiers/english.all.3class.distsim.crf.ser.gz -ner.useSUTime false \ -ner.combinationMode HIGH_RECALL -serializeTo test.serialized.ncc.ncc.ser.gz An example serialized NERClassifierCombiner with these settings is supplied in the classifiers directory. Here is an example of loading that classifier and running it on the sample CONLL data: java -mx2g edu.stanford.nlp.ie.NERClassifierCombiner -loadClassifier \ classifiers/example.serialized.ncc.ncc.ser.gz -map word=0,answer=1 \ -testFile sample-conll-file.txt For a more exhaustive description of NERClassifierCombiner go to http://nlp.stanford.edu/software/ncc-faq.html PROGRAMMATIC USE The NERDemo file illustrates a couple of ways of calling the system programatically. You should get the same results from java -cp stanford-ner.jar:lib/*:. -mx300m NERDemo classifiers/english.all.3class.distsim.crf.ser.gz sample.txt as from using CRFClassifier. For more information on API calls, look in the enclosed javadoc directory: load index.html in a browser and look first at the edu.stanford.nlp.ie.crf package and CRFClassifier class. If you wish to train your own NER systems, look also at the edu.stanford.nlp.ie package NERFeatureFactory class. SERVER VERSION The NER code may also be run as a server listening on a socket: java -mx1000m -cp stanford-ner.jar:lib/* edu.stanford.nlp.ie.NERServer 1234 You can specify which model to load with flags, either one on disk: java -mx1000m -cp stanford-ner.jar:lib/* edu.stanford.nlp.ie.NERServer -loadClassifier classifiers/all.3class.crf.ser.gz 1234 Or if you have put a model inside the jar file, as a resource under, say, models: java -mx1000m -cp stanford-ner.jar:lib/* edu.stanford.nlp.ie.NERServer -loadClassifier models/all.3class.crf.ser.gz 1234 RUNNING CLASSIFIERS FROM INSIDE A JAR FILE The software can run any serialized classifier from within a jar file by following the -loadClassifier flag by some resource available within a jar file on the CLASSPATH. An end user can make their own jar files with the desired NER models contained inside. This allows single jar file deployment. PERFORMANCE GUIDELINES Performance depends on many factors. Speed and memory use depend on hardware, operating system, and JVM. Accuracy depends on the data tested on. Nevertheless, in the belief that something is better than nothing, here are some statistics from one machine on one test set, in semi-realistic conditions (where the test data is somewhat varied). ner-eng-ie.crf-3-all2006-distsim.ser.gz (older version of ner-eng-ie.crf-3-all2008-distsim.ser.gz) Memory: 320MB (on a 32 bit machine) PERSON ORGANIZATION LOCATION 91.88 82.91 88.21 -------------------- CHANGES -------------------- 2020-11-17 4.2.0 Update for compatibility 2020-05-10 4.0.0 Update to UDv2.0 tokenization 2018-10-16 3.9.2 Update for compatibility 2018-02-27 3.9.1 KBP ner models for Chinese and Spanish 2017-06-09 3.8.0 Updated for compatibility 2016-10-31 3.7.0 Improved Chinese NER 2015-12-09 3.6.0 Updated for compatibility 2015-04-20 3.5.2 synch standalone and CoreNLP functionality 2015-01-29 3.5.1 Substantial accuracy improvements 2014-10-26 3.5.0 Upgrade to Java 1.8 2014-08-27 3.4.1 Add Spanish models 2014-06-16 3.4 Fix serialization bug 2014-01-04 3.3.1 Bugfix release 2013-11-12 3.3.0 Update for compatibility 2013-11-12 3.3.0 Update for compatibility 2013-06-19 3.2.0 Improve handling of line-by-line input 2013-04-04 1.2.8 nthreads option 2012-11-11 1.2.7 Improved English 3 class model by including data from Wikipedia, release Chinese model 2012-07-09 1.2.6 Minor bug fixes 2012-05-22 1.2.5 Fix encoding issue 2012-04-07 1.2.4 Caseless version of English models supported 2012-01-06 1.2.3 Minor bug fixes 2011-09-14 1.2.2 Improved thread safety 2011-06-19 1.2.1 Models reduced in size but on average improved in accuracy (improved distsim clusters) 2011-05-16 1.2 Normal download includes 3, 4, and 7 class models. Updated for compatibility with other software releases. 2009-01-16 1.1.1 Minor bug and usability fixes, changed API 2008-05-07 1.1 Additional feature flags, various code updates 2006-09-18 1.0 Initial release --- ### Doc Ner TODO 2009 (`doc/ner/TODO-2009.txt`) - Add German NER - Add BionNER - Add all uppercase/uncased NER model - Make it possible to have multiple NER matching threads sharing one data resource of features, etc. - --- ### Doc Ner Ner Manifest (`doc/ner/ner-manifest.txt`) Manifest-Version: 1.0 Created-By: Stanford JavaNLP Main-Class: edu.stanford.nlp.ie.crf.NERGUI --- ### Doc Ner Performance (`doc/ner/performance.txt`) PERFORMANCE NOTES: Note that these results are underestimates for DATE, TIME, etc. because not all the test data (mcdm.test) was marked for them. But they're not bad undercounts, because the classifiers have as much trouble based on the "all" data on recall as precision. For example: Sept 2006 F1 scores on mcdm.test ner-eng-ie.crf-3-all2006.ser.gz Memory: 100 MB PERSON ORGANIZATION LOCATION 89.19 80.15 85.48 ner-eng-ie.crf-3-all2006-distsim.ser.gz Memory: 320MB PERSON ORGANIZATION LOCATION 91.88 82.91 88.21 ner-eng-ie.crf-7-muc.ser.gz Memory: 120MB PERSON ORGANIZATION LOCATION DATE TIME MONEY PERCENT 74.45 59.93 76.27 55.59 73.12 64.96 71.05 ner-eng-ie.crf-7-muc-distsim.ser.gz Memory: 350MB PERSON ORGANIZATION LOCATION DATE TIME MONEY PERCENT 84.09 65.20 83.13 56.68 72.19 66.01 70.82 ner-eng-ie.crf-8-all2006.ser.gz Memory: 120MB PERSON ORGANIZATION LOCATION DATE TIME MONEY PERCENT 89.15 79.93 85.07 47.43 69.79 59.10 29.87 ner-eng-ie.crf-8-all2006-distsim.ser.gz Memory: 350MB PERSON ORGANIZATION LOCATION DATE TIME MONEY PERCENT 92.11 82.55 87.65 47.95 69.59 58.82 31.17 train_all.8.distsim_test_mcdm.test.results processed 125957 tokens with 11947 phrases; found: 11465 phrases; correct: 9437. accuracy: 95.16%; precision: 82.31%; recall: 78.99%; FB1: 80.62 DATE: precision: 85.23%; recall: 33.36%; FB1: 47.95 535 LOCATION: precision: 85.06%; recall: 90.41%; FB1: 87.65 3293 MISC: precision: 0.00%; recall: 0.00%; FB1: 0.00 518 MONEY: precision: 71.09%; recall: 50.17%; FB1: 58.82 211 ORGANIZATION: precision: 84.04%; recall: 81.12%; FB1: 82.55 3846 PERCENT: precision: 64.86%; recall: 20.51%; FB1: 31.17 37 PERSON: precision: 91.39%; recall: 92.83%; FB1: 92.11 2905 TIME: precision: 99.17%; recall: 53.60%; FB1: 69.59 120 --- ### Doc Ner Sample W Time (`doc/ner/sample-w-time.txt`) Barack Obama was born on August 4, 1961 in Honolulu, Hawaii which was 4 days ago. --- ### Doc Ner Sample (`doc/ner/sample.txt`) The fate of Lehman Brothers, the beleaguered investment bank, hung in the balance on Sunday as Federal Reserve officials and the leaders of major financial institutions continued to gather in emergency meetings trying to complete a plan to rescue the stricken bank. Several possible plans emerged from the talks, held at the Federal Reserve Bank of New York and led by Timothy R. Geithner, the president of the New York Fed, and Treasury Secretary Henry M. Paulson Jr. --- ### Doc Openie README (`doc/openie/README.txt`) Stanford Open Domain Relation Extractor, v3.5.2 - 2015-04-20 Copyright (c) 2002-2012 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. Original author: Gabor Angeli Code contributions: This package contains an open domain relation extractor, producing triples where each element can be an arbitrary plain-text element. --- ### Doc Openie Openie Manifest (`doc/openie/openie-manifest.txt`) Manifest-Version: 1.0 Created-By: Stanford JavaNLP Main-class: edu.stanford.nlp.naturalli.OpenIE --- ### Doc Segmenter README Arabic (`doc/segmenter/README-Arabic.txt`) Stanford Arabic Segmenter - v4.2.0 - 2020-11-17 -------------------------------------- (c) 2003-2020 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. Arabic segmenter by Spence Green CRF code by Jenny Finkel Support code by Stanford JavaNLP members The system requires Java 6 (JDK1.6) or higher. The Arabic word segmenter is based on a conditional random field (CRF) sequence classifier. The included TreebankPreprocessor package can be used to generate training data for the model, assuming a copy of Penn Arabic Treebank (ATB) is available. INSTALLATION The segmenter does not require any platform-specific installation. Unpack the gzip'd tar file in any convenient location. To invoke the segmenter, you'll need to add the ".jar" files in the unpacked directory to your Java classpath. If you don't know how to modify your classpath, read this tutorial: http://docs.oracle.com/javase/tutorial/essential/environment/paths.html The commands in the remainder of this README assume that you've added the jars to your classpath. USAGE The segmenter assumes that you have a newline delimited text file with UTF-8 encoding. If you're working with Arabic script, chances are that your file is already encoded in UTF-8. If it is in some other encoding, then you should check Google for a tutorial on "iconv," a handy tool that allows you to automatically change file encodings. Suppose that your raw Arabic file is called "my_arabic_file.txt". You can segment the tokens in this file with the following command: java -mx1g edu.stanford.nlp.international.arabic.process.ArabicSegmenter -loadClassifier data/arabic-segmenter-atb+bn+arztrain.ser.gz -textFile my_arabic_file.txt > my_arabic_file.txt.segmented Additional command line options are available to mark proclitics and enclitics that were split by the segmenter. Suppose that the raw token "AABBBCC" was split into three segments "AA BBB CC", where "AA" is a proclitic and "CC" is an enclitic. You can direct the segmenter to mark these clitics with these command-line options: -prefixMarker # -suffixMarker # In this case, the segmenter would produce "AA# BBB #CC". Here '#' is the segment marker, but the options accept any character as an argument. You can use different markers for proclitics and enclitics. ORTHOGRAPHIC NORMALIZATION The segmenter contains a deterministic orthographic normalization package: edu.stanford.nlp.international.arabic.process.ArabicTokenizer ArabicTokenizer supports various orthographic normalization options that can be configured in ArabicSegmenter using the -orthoOptions flag. The argument to -orthoOptions is a comma-separated list of normalization options. The following options are supported: useUTF8Ellipsis : Replaces sequences of three or more full stops with \u2026 normArDigits : Convert Arabic digits to ASCII equivalents normArPunc : Convert Arabic punctuation to ASCII equivalents normAlif : Change all alif forms to bare alif normYa : Map ya to alif maqsura removeDiacritics : Strip all diacritics removeTatweel : Strip tatweel elongation character removeQuranChars : Remove diacritics that appear in the Quran removeProMarker : Remove the ATB null pronoun marker removeSegMarker : Remove the ATB clitic segmentation marker removeMorphMarker : Remove the ATB morpheme boundary markers removeLengthening : Replace sequences of three identical characters with one atbEscaping : Replace left/right parentheses with ATB escape characters The orthographic normalization options must match at both training and test time! Consequently, if you want to apply an orthographic normalization that differs from the default, then you'll need to retrain ArabicSegmenter. SEGMENTING DIALECTAL TEXT The segmenter supports segmenting dialectal Arabic using domain adaptation. [Hal Daumé III, Frustratingly Easy Domain Adaptation, ACL 2007] The model that comes with this distribution is trained to support Egyptian dialect. To indicate that the provided text is in Egyptian dialect, add the command-line option: -domain arz You can also construct a file that specifies a dialect for each newline-separated sentence, by adding "atb" [MSA] or "arz" [Egyptian] at the beginning of each line followed by a space character. This feature is enabled with the flag: -withDomains See the bottom of the next section for information about training the segmenter on your own dialectal data. TRAINING THE SEGMENTER The current model is trained on parts 1-3 of the ATB, parts 1-8 of the ARZ treebank, and the Broadcast News treebank. To train a new model, you need to create a data file from the unpacked LDC distributions. You can create this data file with the script tb-preproc, which is included in the segmenter release. You'll need: - Python 2.7 (for running the preprocessing scripts) - an unpacked LDC distribution with files in *integrated* format - a directory with four text files called dev, train, test, and all, each of which lists filenames of integrated files (these usually end in .su.txt), one per line. dev, train, and test can be left empty if you have no need for a train/test split. The splits that we use are included in the distribution. You can also find them at http://nlp.stanford.edu/software/parser-arabic-data-splits.shtml Once you have these, run the tb-preproc script, providing the necessary arguments: atb_base - the most specific directory that is a parent of all integrated files you wish to include. Files will be located recursively by name in this directory; it is not recommended to have several copies of the same distribution within this directory (though all that will happen is that you will train on redundant data). splits_dir - the directory containing dev, train, and test listings output_prefix - the location and filename prefix that will identify the output files. The preprocessor appends "-all.utf8.txt" to this argument to give the name of the output file for the "all" split (and similarly for dev, train, and test). domain - [optional] a label for the Arabic dialect/genre that this data is in. Our model uses "atb" for ATB1-3, "bn" for Broadcast News, and "arz" for Egyptian. If a domain is given, additional files will be generated (named e.g. "output_prefix-withDomains-all.utf8.txt") for training the domain adaptation model. Suppose your output_prefix is "./atb". You should see files in the current working directory named atb-dev.utf8.txt atb-train.utf8.txt atb-test.utf8.txt You can use the train file to retrain the segmenter with this command: java -Xmx12g -Xms12g edu.stanford.nlp.international.arabic.process.ArabicSegmenter -trainFile atb-train.utf8.txt -serializeTo my_trained_segmenter.ser.gz This command will produce the serialized model "my_trained_segmenter.ser.gz" that you can use for raw text processing as described in the "USAGE" section above. The command above train the model with L2 regularization. For the model included in the distribution has been trained using L1 regularization, which decreases the model file size dramatically in exchange for a usually negligible drop in accuracy. To use L1 regularization, add the options -useOWLQN -priorLambda 0.05 TRAINING FOR DIALECT To train a model with domain adaptation, first make sure you have generated a training file with domain labels. You can create this using the preprocessing script with the optional domain argument, or do it yourself with a simple sed script (the -withDomains files differ from the simple training files only in the presence of a domain identifier prepended to each line followed by a space). These domain labels can be arbitrary strings, as long as they don't contain whitespace characters; thus, if you have data available for other dialects in ATB format, it is possible to train your own system that can support these dialects. For best results, include MSA data as well as your dialect data in your training. You can do this by simply concatenating the ATB1-3 -withDomains file and the dialect -withDomains file. (Adding data from dialects other than your target dialect should not hurt performance, as long as they are marked as different domains--it may even help!) The training command for domain-labeled data is: java -Xmx64g -Xms64g edu.stanford.nlp.international.arabic.process.ArabicSegmenter -withDomains -trainFile atb+arz-withDomains-train.utf8.txt -serializeTo my_trained_segmenter.ser.gz Warning: training with lots of data from several domains requires a lot of memory and processor time. If you have enough memory to fit all of the weights for the entire dataset in RAM (this is a bit less than 64G for ATB1-3 + BN + ARZ), training will take about ten days of single-threaded processor time. This can be parallelized by adding the option -multiThreadGrad If you are not running on a machine with 64G of RAM, the training is likely to take much longer. You have been warned. --- ### Doc Segmenter README Chinese (`doc/segmenter/README-Chinese.txt`) Stanford Chinese Segmenter - v4.2.0 - 2020-11-17 -------------------------------------------- (c) 2003-2020 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. Chinese segmenter by Pi-Chuan Chang, Huihsin Tseng, and Galen Andrew CRF code by Jenny Finkel Support code by Stanford JavaNLP members The system requires Java 6 (JDK1.6) or higher. USAGE Unix: > segment.sh [-k] [ctb|pku] ctb : Chinese Treebank pku : Beijing Univ. filename: The file you want to segment. Each line is a sentence. encoding: UTF-8, GB18030, etc. (This must be a character encoding name known by Java) size: size of the n-best list (just put '0' to print the best hypothesis without probabilities). -k: keep all white spaces in the input * Sample usage: segment.sh ctb test.simp.utf8 UTF-8 * Note: Large test file requires large memory usage. For processing large data files, you may want to change memory allocation in Java (e.g., to be able to use 8Gb of memory, you need to change "-mx2g" to "-mx8g" inside segment.sh). Another solution is to split the test file to smaller ones to reduce memory usage. * In addition to the command line scripts, there is a Java class "SegDemo" which shows how to call the segmenter in Java code. Usage: java -mx2g -cp "*:." SegDemo test.simp.utf8 SegDemo as supplied assumes that it is running in the home directory of the installation, and to run anywhere else, you need to set the path to the dictionaries. SEGMENTATION MODELS Two segmentation models are provided. The "ctb" model was trained with Chinese treebank (CTB) segmentation, and the "pku" model was trained with Beijing University's (PKU) segmentation. PKU models provide smaller vocabulary sizes and OOV rates on test data than CTB models. For both CTB and PKU, we provide two models representing slightly different feature sets: Models "ctb" and "pku" incorporate lexicon features to increase consistency in segmentation. DATA [Segmentation standard] (Chinese Penn Treebank) The supplied segmenter segments according to Chinese Penn Treebank segmentation conventions. For more information, see: http://www.cis.upenn.edu/~chinese/segguide.3rd.ch.pdf (Beijing University) This segmenter segments according to the Peking University standard. For more information, see: http://sighan.cs.uchicago.edu/bakeoff2005/data/pku_spec.pdf [Training data] (Chinese Penn Treebank) "data/ctb.gz" is trained with the training data in the LDC Chinese Treebank 7 (Beijing University) "data/pku.gz" is trained with the data provided by Peking University for the Second International Chinese Word Segmentation Bakeoff. See: http://sighan.cs.uchicago.edu/bakeoff2005/ MORE INFORMATION The details of the segmenter can be found in this paper: Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky and Christopher Manning. "A Conditional Random Field Word Segmenter." In Fourth SIGHAN Workshop on Chinese Language Processing. 2005. http://nlp.stanford.edu/pubs/sighan2005.pdf (Notice that the training data, features and normalizations used in this distribution are not exactly the same as the systems described in the paper.) The description of the lexicon features can be found in: Pi-Chuan Chang, Michel Gally and Christopher Manning. "Optimizing Chinese Word Segmentation for Machine Translation Performance" In ACL 2008 Third Workshop on Statistical Machine Translation. http://nlp.stanford.edu/pubs/acl-wmt08-cws.pdf For more information, look in the included Javadoc, starting with the edu.stanford.nlp.ie.crf.CRFClasifier class documentation. Send any questions or feedback to java-nlp-support@lists.stanford.edu. --- ### Doc Segmenter README (`doc/segmenter/README.txt`) The Stanford segmenter distribution includes tools for segmenting Chinese and Arabic text. See README-Chinese.txt and README-Arabic.txt for more details. ------------------------------------ LICENSE ------------------------------------ This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/ . For more information, bug reports, fixes, contact: Christopher Manning Dept of Computer Science, Gates 2A Stanford CA 94305-9020 USA manning@cs.stanford.edu ------------------------------------ CHANGES ------------------------------------ 2020-11-17 4.2.0 Update for compatibility 2020-05-10 4.0.0 New Chinese segmenter trained off of CTB 9.0 2018-10-16 3.9.2 Update for compatibility 2018-02-27 3.9.1 Updated for compatibility 2016-10-31 3.7.0 Update for compatibility 2015-12-09 3.6.0 Update for compatibility 2015-04-20 3.5.2 Update for compatibility 2015-01-29 3.5.1 Update for compatibility 2014-10-26 3.5.0 Upgrade to Java 1.8 2014-08-27 3.4.1 Update for compatibility 2014-06-16 3.4 Update Arabic segmenter 2014-01-04 3.3.1 Bugfix release 2013-11-12 3.3.0 Update for compatibility 2013-06-19 3.2.0 Improve handling of line-by-line input 2013-04-04 1.6.8 ctb7 model, -nthreads option 2012-11-11 1.6.7 Bugfixes to both Arabic and Chinese segmenters; Chinese segmenter can now load files from jar 2012-07-09 1.6.6 Improved Arabic model 2012-05-22 1.6.5 Supports stdin. 2012-03-09 1.6.4 Arabic segmenter now included. 2011-12-16 1.6.3 Updated code to maintain compatibility 2011-09-14 1.6.2 Updated code to maintain compatibility 2011-06-15 1.6.1 Updated code to maintain compatibility 2011-05-15 1.6 Updated models, code to be compatible with other current releases 2008-05-21 1.5 The models in distribution incorporate training lexicon features. In addition, the segmenter now supports k-best output. 2006-05-12 1.0 This distribution includes models of two segmentation standards -- CTB and PK (Beijing Univ.) 2006-04-10 0.9 Add normalization for punctuation/symbol characters from the "ASCII" (U+0021-U+0075) code point range. --- ### Doc Tagger README Models (`doc/tagger/README-Models.txt`) Stanford POS Tagger, v4.2.0 - 2020-11-17 Copyright (c) 2002-2020 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. This document contains (some) information about the models included in this release and that may be downloaded for the POS tagger website at http://nlp.stanford.edu/software/tagger.html . All of the models mentioned in this document are in the downloaded package in the same directory as this readme. All taggers are accompanied by the props files used to create them; please examine these files for more detailed information about the creation of the taggers. For English, the bidirectional taggers are slightly more accurate, but tag much more slowly; choose the appropriate tagger based on your speed/performance needs. English taggers --------------------------- english-left3words-distsim.tagger Trained on WSJ sections 0-18 and extra parser training data using the left3words architecture and includes word shape and distributional similarity features. Penn tagset. UDv2.0 tokenization standard. english-bidirectional-distsim.tagger Trained on WSJ sections 0-18 using a bidirectional architecture and including word shape and distributional similarity features. Penn Treebank tagset. UDv2.0 tokenization standard. english-caseless-left3words-distsim.tagger Trained on WSJ sections 0-18 and extra parser training data using the left3words architecture and includes word shape and distributional similarity features. Penn tagset. Ignores case. UDv2.0 tokenization standard. Chinese tagger --------------------------- chinese-nodistsim.tagger Trained on a combination of CTB7 texts from Chinese and Hong Kong sources. LDC Chinese Treebank POS tag set. chinese-distsim.tagger Trained on a combination of CTB7 texts from Chinese and Hong Kong sources with distributional similarity clusters. LDC Chinese Treebank POS tag set. Arabic tagger --------------------------- arabic.tagger Trained on the *entire* ATB p1-3. When trained on the train part of the ATB p1-3 split done for the 2005 JHU Summer Workshop (Diab split), using (augmented) Bies tags, it gets French tagger --------------------------- french-ud.tagger Trained on the French GSD (UDv2.2) data set German tagger --------------------------- german-ud.tagger Trained on the German GSD (UDv2.2) data set Spanish tagger -------------------------- spanish-ud.tagger Trained on the Spanish AnCora (UDv2.0) data set --- ### Doc Tagger README (`doc/tagger/README.txt`) Stanford POS Tagger, v4.2.0 - 2020-11-17 Copyright (c) 2002-2020 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. Original tagger author: Kristina Toutanova Code contributions: Christopher Manning, Dan Klein, William Morgan, Huihsin Tseng, Anna Rafferty, John Bauer Major rewrite for version 2.0 by Michel Galley. Current release prepared by: Jason Bolton This package contains a Maximum Entropy part of speech tagger. A Part-Of-Speech Tagger (POS Tagger) is a piece of software that reads text in some language and assigns parts of speech to each word (and other tokens), such as noun, verb, adjective, etc. Generally computational applications use more fine-grained POS tags like 'noun-plural'. This software is a Java implementation of the log-linear part-of-speech (POS) taggers described in: Kristina Toutanova and Christopher D. Manning. 2000. Enriching the Knowledge Sources Used in a Maximum Entropy Part-of-Speech Tagger. Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC-2000), Hong Kong. Kristina Toutanova, Dan Klein, Christopher Manning, and Yoram Singer. 2003. Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network. In Proceedings of HLT-NAACL 2003 pages 252-259. The system requires Java 1.8+ to be installed. About 60 MB of memory is required to run a trained tagger, depending on the OS, tagging model chosen, etc. (i.e., you may need to give to java an option like java -mx120m). Plenty of memory is needed to train a tagger. It depends on the complexity of the model but at least 1GB is recommended (java -mx1g). Two trained tagger models for English are included with the tagger, along with some caseless versions, and we provide models for some other languages. The tagger can be retrained on other languages based on POS-annotated training text. QUICKSTART ----------------------------------------------- The Stanford POS Tagger is designed to be used from the command line or programmatically via its API. There is a GUI interface, but it is for demonstration purposes only; most features of the tagger can only be accessed via the command line. To run the demonstration GUI you should be able to use any of the following 2 methods: 1) java -mx200m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTaggerGUI models/wsj-0-18-left3words-distsim.tagger 2) Running the appropriate script for your operating system: stanford-postagger-gui.bat ./stanford-postagger-gui.sh To run the tagger from the command line, you can start with the provided script appropriate for you operating system: ./stanford-postagger.sh models/wsj-0-18-left3words-distsim.tagger sample-input.txt stanford-postagger models\wsj-0-18-left3words-distsim.tagger sample-input.txt The output should match what is found in sample-output.txt The tagger has three modes: tagging, training, and testing. Tagging allows you to use a pretrained model (two English models are included) to assign part of speech tags to unlabeled text. Training allows you to save a new model based on a set of tagged data that you provide. Testing allows you to see how well a tagger performs by tagging labeled data and evaluating the results against the correct tags. Many options are available for training, tagging, and testing. These options can be set using a properties file. To start, you can generate a default properties file by: java -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -genprops > myPropsFile.prop This will create the file myPropsFile.prop with descriptions of each option for the tagger and the default values for these options specified. Any properties you can specify in a properties file can be specified on the command line or vice versa. For further information, please consult the Javadocs (start with the entry for MaxentTagger, which includes a table of all options which may be set to configure the tagger and descriptions of those options). To tag a file using the pre-trained bidirectional model ======================================================= java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/wsj-0-18-bidirectional-distsim.tagger -textFile sample-input.txt > sample-tagged.txt Tagged output will be printed to standard out, which you can redirect as above. Note that the bidirectional model is slightly more accurate but significantly slower than the left3words model. To train a simple model ======================= java -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -prop propertiesFile -model modelFile -trainFile trainingFile To test a model =============== java -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -prop propertiesFile -model modelFile -testFile testFile Using models for French, German, and Spanish =========================================== Starting with version 4.0.0, French, German, and Spanish are tokenized according to the UD 2.0 standard. This includes creating multiword tokens. This functionality requires the pipeline functionality only available in the full Stanford CoreNLP distribution. To tag French, German, or Spanish, one must provide UD 2.0 tokenized text, or upgrade to the full Stanford CoreNLP package to get UD 2.0 tokenization for these languages. To run on pretokenized text, add "-tokenize false" to your command. Example: java -mx300m -classpath stanford-postagger.jar edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/french-ud.tagger -tokenize false -textFile sample-input.txt > sample-tagged.txt CONTENTS ----------------------------------------------- README.txt This file. LICENSE.txt Stanford POS Tagger is licensed under the GNU General Public License (v2+). stanford-postagger.jar stanford-postagger-YYYY-MM-DD.jar This is a JAR file containing all the Stanford classes necessary to run the Stanford POS Tagger. The two jar files are identical. You can use either the one with a version (date) indication or without, as you prefer. src A directory containing the Java 1.8 source code for the Stanford POS Tagger distribution. build.xml, Makefile Files for building the distribution (with ant and make, respectively) models A directory containing trained POS taggers; the taggers end in ".tagger" and the props file used to make the taggers end in ".props". The ".props" files cannot be directly used on your own machine as they use paths on the Stanford NLP machines, but they may serve as examples for your own properties files. Included in the full version are other English taggers, a German tagger, an Arabic tagger, and a Chinese tagger. If you chose to download the smaller version of the tagger, you have only two English taggers (left3words is faster but slightly less accurate than bidirectional-distsim) - feel free to download any other taggers you need from the POS tagger website. More information about the models can be found in the README-Models.txt file in this directory. sample-input.txt A sample text file that you can tag to demonstrate the tagger. sample-output.txt Tagged output of the tagger (using the left3words model) stanford-postagger-gui.sh stanford-postagger-gui.bat Scripts for invoking the GUI demonstration version of the tagger. stanford-postagger.sh stanford-postagger.bat Scripts for running the command-line version of the tagger. javadoc Javadocs for the distribution. In particular, look at the javadocs for the class edu.stanford.nlp.tagger.maxent.MaxentTagger. TaggerDemo.java A sample file for how to call the tagger in your own program. You should be able to compile and run it with: javac -cp stanford-postagger.jar TaggerDemo.java java -cp ".:stanford-postagger.jar" TaggerDemo models/wsj-0-18-left3words-distsim.tagger sample-input.txt (If you are on Windows, you need to replace the ":" with a ";" in the -cp argument, and should use a "\" in place of the "/" in the filename....) THANKS ----------------------------------------------- Thanks to the members of the Stanford Natural Language Processing Lab for great collaborative work on Java libraries for natural language processing. http://nlp.stanford.edu/javanlp/ CHANGES ----------------------------------------------- 2020-11-17 4.2.0 Add currency data for English models. 2020-08-06 4.1.0 Add missing extractor, spanish tokenization upgrades 2020-05-22 4.0.0 Model tokenization updated to UDv2.0 2018-10-16 3.9.2 New English models, better currency symbol handling 2018-02-27 3.9.1 new French UD model 2017-06-09 3.8.0 new Spanish and French UD models 2016-10-31 3.7.0 Update for compatibility, German UD model 2015-12-09 3.6.0 Updated for compatibility 2015-04-20 3.5.2 Update for compatibility 2015-01-29 3.5.1 General bugfixes 2014-10-26 3.5.0 Upgrade to Java 1.8 2014-08-27 3.4.1 Add Spanish models 2014-06-16 3.4 Using CC tagset for French 2014-01-04 3.3.1 Bugfix release 2013-11-12 3.3.0 Add imperatives to English training data 2013-06-19 3.2.0 Decrease size and improve speed of tagger models for all languages 2013-04-04 3.1.5 Speed improvements, ctb7 model, -nthreads option 2012-11-11 3.1.4 Updated Chinese model 2012-07-09 3.1.3 Minor bug fixes 2012-05-22 3.1.2 Updated for compatibility with other releases 2012-03-09 3.1.1 Caseless models added 2012-01-06 3.1.0 French tagger added, tagging speed improved 2011-09-14 3.0.4 Updated for compatibility with other releases 2011-06-15 3.0.3 Updated for compatibility with other releases 2011-05-15 3.0.2 Can read training files in TSV format 2011-04-17 3.0.1 Improved German and Arabic models Compatible with other Stanford releases 2010-05-21 3.0.0 Re-entrant LICENSE ----------------------------------------------- Stanford POS Tagger Copyright (c) 2002-2010 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/ . For more information, bug reports, fixes, contact: Christopher Manning Dept of Computer Science, Gates 2A Stanford CA 94305-9020 USA Support/Questions: java-nlp-user@lists.stanford.edu Licensing: java-nlp-support@lists.stanford.edu http://nlp.stanford.edu/software/tagger.html CONTACT ----------------------------------------------- For questions about the Stanford POS tagger, please feel free to contact the Stanford JavaNLP user community at the mailing list java-nlp-user@lists.stanford.edu. You need to be a member of this mailing list to be able to post to it. Join the list either by emailing java-nlp-user-join@lists.stanford.edu (leave the subject and message body empty) or by using the web interface at: https://mailman.stanford.edu/mailman/listinfo/java-nlp-user This is the best list to post to in order to ask questions, make announcements, or for discussion among Stanford JavaNLP tool users. We provide assistance on a best-effort basis. You can also look at the list archives via https://mailman.stanford.edu/pipermail/java-nlp-user/. For licensing questions, please see the tagger webpage or contact Stanford JavaNLP at java-nlp-support@lists.stanford.edu. --- ### Doc Tagger Postagger Manifest (`doc/tagger/postagger-manifest.txt`) Manifest-Version: 1.0 Created-By: Stanford JavaNLP Main-class: edu.stanford.nlp.tagger.maxent.MaxentTaggerGUI --- ### Doc Tagger Sample Input (`doc/tagger/sample-input.txt`) A passenger plane has crashed shortly after take-off from Kyrgyzstan's capital, Bishkek, killing a large number of those on board. The head of Kyrgyzstan's civil aviation authority said that out of about 90 passengers and crew, only about 20 people have survived. The Itek Air Boeing 737 took off bound for Mashhad, in north-eastern Iran, but turned round some 10 minutes later. --- ### Doc Tagger Sample Output (`doc/tagger/sample-output.txt`) A_DT passenger_NN plane_NN has_VBZ crashed_VBN shortly_RB after_IN take-off_NN from_IN Kyrgyzstan_NNP 's_POS capital_NN ,_, Bishkek_NNP ,_, killing_VBG a_DT large_JJ number_NN of_IN those_DT on_IN board_NN ._. The_DT head_NN of_IN Kyrgyzstan_NNP 's_POS civil_JJ aviation_NN authority_NN said_VBD that_IN out_IN of_IN about_IN 90_CD passengers_NNS and_CC crew_NN ,_, only_RB about_IN 20_CD people_NNS have_VBP survived_VBN ._. The_DT Itek_NNP Air_NNP Boeing_NNP 737_CD took_VBD off_RP bound_VBN for_IN Mashhad_NNP ,_, in_IN north-eastern_JJ Iran_NNP ,_, but_CC turned_VBD round_NN some_DT 10_CD minutes_NNS later_RB ._. --- ### Doc Tregex README Gui (`doc/tregex/README-gui.txt`) Tregex GUI v4.2.0 - 2020-11-17 ---------------------------------------------- Copyright (c) 2003-2020 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. Original core Tregex code by Roger Levy and Galen Andrew. Original core Tsurgeon code by Roger Levy. TregexGUI by Anna Rafferty Support code, additional features, etc. by Chris Manning This release prepared by John Bauer. ---------------------------- TREGEX GRAPHICAL USER INTERFACE (GUI) README ---------------------------- The Tregex GUI is a graphical user interface for Tregex and Tsurgeon. You can access it by double- clicking on the jar file tregex.jar. For searching large treebanks, you may need to use more memory; the script run-tregex-gui.command includes this allocation of memory and can be run from the command line or double-click to run on a Mac. If you still have memory problems, you can allot more memory by opening the script in a text editor and changing "-mx300m" to include a bigger number (e.g., "-mx512m"). Tregex requires Java 1.5+. Further documentation for Tregex and Tsurgeon can be found in README-tregex.txt and README-tsurgeon.txt, respectively. ---------------------------- LOADING TREEBANKS/TREE FILES ---------------------------- To load a file containing Penn Treebank formatted trees, choose "Load trees..." from the file menu. Multiple tree files and/or directories may be selected. After selecting the tree files you wish to load, press "Load with file filters" to choose what filters you would like to apply to the files. All filters are run based on the name of the file. Possible filtering options are: - Prefix: Load only files that start with the given character sequence - Extension: Load only files that end with the given character sequence - Has number in range: Loads only numbered files such that the number falls in the given range, inclusive. Ranges can be disjoint as long as multiple ranges are comma-separated (e.g., "100-500,550-700") File filters are combined such that all loaded files must obey all of the filters; only one filter of any given type should be specified. Once the tree files are loaded, their names appear in the upper left hand panel "Tree files:". Unchecking the check boxes next to the files causes the unchecked files not to be included in searches/tsurgeon operations. To remove all files from the tree panel, choose "Clear all files" from the Edit menu. ---------------------------- PERFORMING TREGEX SEARCHES ---------------------------- To perform a Tregex search, load the files you would like to search and type a Tregex pattern in the "Pattern:" box in the top middle of the window. Press "Help" beneath the Pattern box for information about Tregex syntax. After you have typed the pattern, press "Search" to find all matches to the given pattern. By default, trees that contain at least one match are displayed in the "Matches:" panel in the top right of the window, and the first matching tree is graphically displayed in the bottom portion of the window. Click on a match in the Match panel to display it graphically. In the graphical display, matched nodes in the tree are displayed in a different color than other nodes. To display only the matched subtrees, choose "Preferences..." (Mac, from the Application menu) or "Options..." (other OS, under Tools), and check "Show only matched portions of the tree". You must rerun the search to switch between showing only matched portions and showing full trees. In preferences, other display options can also be set, such as the colors, size, and font used by the graphical display. ---------------------------- USING TSURGEON ---------------------------- Tsurgeon modifications can also be performed using Interactive Tregex. To enable Tsurgeon, choose "Preferences..." from the File menu and check "Enable Tsurgeon". You can now run Tsurgeon scripts. Tsurgeon commands must be paired with a Tregex pattern that names the nodes on which modifications will be performed. Type the Tregex pattern in the Pattern box, and type the modifications you would like to make in the "Tsurgeon script:" box. Then click "Run script" to perform the modifications. Each Tsurgeon operation must appear on a separate line in the Tsurgeon script box. Press "Help" for some information about Tsurgeon operation syntax. ---------------------------- SAVING RESULTS ---------------------------- You can save the results of a Tregex search or Tsurgeon operation by choosing "Save matches..." from the File menu. This saves all trees in the Matches panel in Penn Treebank form. "Save matched sentences..." saves the matches in sentence String form, just as they show up in the matches panel. You can also save a log of the number of matches found for each pattern you have searched. By clicking the "Statistics" button in the middle of the screen, below the Tsurgeon buttons, you can see a table of the patterns for which you have searched, the number of trees that each matched, and the number of overall matches that were found. To save this information in a tab delimited text file, choose "Save statistics..." from the File menu. All three save options save files in the encoding specified in the Preferences panel for loading tree files. ---------------------------- MULTILANGUAGE SUPPORT ---------------------------- Some multilanguage support is built into Tregex GUI, and most languages can be read by the GUI. To enable this support, choose go to Preferences (Mac, under the application menu) or Options (other OS, under the Tools menu). Several options may need to be changed: tree reader factory, head finder, font, and encoding. Several possible tree reader factories and head finders are provided; you may also specify your own. Two common languages you may be trying to use are Chinese or Arabic; any head finder or tree reader factory beginning with "Chinese" or "Arabic" will work for these languages, and additionally, CTBTreeReaderFactory is compatible with many Chinese treebanks. Based on your choice of head finder and tree reader factory, the Tregex GUI will guess if you may need a different font and/or text encoding. If a different text encoding is usually used for your selections, you will be prompted as to what text encoding you would like to use. This may also be specified directly in the Preferences panel. ---------------------------- QUESTIONS ---------------------------- For more information on Tregex or Tsurgeon, read README-tregex.txt and README-tsurgeon.txt, and also look at the javadocs suggested in those files. For questions about this distribution, please contact Stanford's JavaNLP group at parser-support@lists.stanford.edu. We provide assistance on a best-effort basis. ---------------------------- LICENSE ---------------------------- Tregex GUI Copyright (c) 2007-2011 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/ . For more information, bug reports, fixes, contact: Christopher Manning Dept of Computer Science, Gates 2A Stanford CA 94305-9020 USA parser-support@lists.stanford.edu http://nlp.stanford.edu/software/tregex.html ---------------------------- CHANGES ---------------------------- 2020-11-17 4.2.0 Update for compatibility 2020-05-10 4.0.0 Update for compatibility 2018-10-16 3.9.2 Update for compatibility 2018-02-27 3.9.1 Updated for compatibility 2017-06-09 3.8.0 Updated for compatibility 2016-10-31 3.7.0 Update for compatibility 2015-12-09 3.6.0 Updated for compatibility 2015-04-20 3.5.2 Update for compatibility 2015-01-29 3.5.1 General bugfixes 2014-10-26 3.5.0 Upgrade to Java 1.8 2014-08-27 3.4.1 Fix bug by adding matcher object for tsurgeon 2014-06-16 3.4 Improved efficiency for some operations, new tregex and tsurgeon patterns added 2014-01-04 3.3.1 Bugfix release, new createSubtree tsurgeon operation 2013-11-12 3.3.0 Allow a TregexMatcher to have its own HeadFinder, useful for the dependencies 2013-06-19 3.2.0 Fix for tsurgeon number reading bug 2013-04-04 2.0.6 Update to maintain compatibility 2012-11-11 2.0.5 Efficiency improvements 2012-07-09 2.0.4 Minor bug fixes 2012-05-22 2.0.3 Rebuilt to be compatible with everything. 2012-03-09 2.0.2 Efficiency improvements 2011-12-16 2.0.1 Fix bug in matchesAt, fix bug in category function, add macros 2011-09-14 2.0.0 Efficiency improvements, include semgrex. 2011-05-15 1.4.4 Rebuilt to be compatible with everything. 2011-05-15 1.4.3 Rebuilt to be compatible with everything. 2011-04-17 1.4.2 Rebuilt to be compatible with tagger, parser, and corenlp. 2010-11-18 1.4.1 Small fixes and improvements (improved help screens, multipattern Tsurgeon scripts with comments introduced by % supported, unclosed regex no longer crashes GUI, support character encodings in script files, fix bug in tregex matching immediate domination path, TregexGUI now shows filename and line number of each match in matches panel) 2009-09-30 1.4 GUI slider for tree size, generalized relabel command (incompatibly), __ and @ now supported in path constraints; bugfixes. 2008-05-06 1.1 Several bug fixes; addition of browse trees function, improved copy/paste and drag and drop support; misc. feature additions 2007-09-20 1.0 Initial release --- ### Doc Tregex README Tregex (`doc/tregex/README-tregex.txt`) Tregex v4.2.0 - 2020-11-17 ---------------------------------------------- Copyright (c) 2003-2020 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. Original core Tregex code by Roger Levy and Galen Andrew. Original core Tsurgeon code by Roger Levy. GUI by Anna Rafferty Support code, additional features, etc. by Chris Manning This release prepared by John Bauer. This package contains Tregex and Tsurgeon. Tregex is a Tgrep2-style utility for matching patterns in trees. It can be run in a graphical user interface, from the command line using the TregexPattern main method, or used programmatically in java code via the TregexPattern, TregexMatcher and TregexPatternCompiler classes. As of version 1.2, the Tsurgeon tree-transformation utility is bundled together with Tregex. See the file README.tsurgeon for details. Java version 1.8 is required to use Tregex. If you really want to use Tregex under an earlier version of Java, look into RetroWeaver: http://retroweaver.sourceforge.net/ QUICKSTART ----------------------------------------------- Programmatic use, command-line use, and GUI-use are supported. To access the graphical interface for Tsurgeon and Tregex, double-click the tregex.jar file. Some help (particularly with syntax) is provided within the program; for further assistance, see README-gui.txt and the documentation mentioned below. A full explanation of pattern syntax and usage is given in the javadocs (particularly TregexPattern), and some of this information is also presented in the TREGEX SYNTAX section below. As a quick example of usage, the following line will scan an English PennTreebank annotated corpus and print all nodes representing a verb phrase dominating a past-tense verb and a noun phrase. ./tregex.sh 'VP < VBD < NP' corpus_dir CONTENTS ----------------------------------------------- README-tregex.txt This file. README-tsurgeon.txt Documentation for Tsurgeon, a tool for modifying trees. README-gui.txt Documentation for the graphical interface for Tregex and Tsurgeon tools. LICENSE.txt Tregex is licensed under the GNU General Public License. stanford-tregex.jar This is a JAR file containing all the Stanford classes necessary to run tregex. src A directory containing the Java 1.8 source code for the Tregex distribution. javadoc Javadocs for the distribution. In particular, look at the javadocs for the class edu.stanford.nlp.trees.tregex.TregexPattern. The first part of that class's javadoc describes syntax and semantics for relations, node labels, node names, and variable groups. The docs for the main method describe command-line options. tregex.sh a shell script for invoking the Tregex tree search tool. tsurgeon.sh a shell script for invoking the Tsurgeon tree transformation tool. run-tregex-gui.command A command file that can be double-clicked on a Mac to start the gui. run-tregex-gui.bat A bat file that can be double-clicked on a PC to start the gui. examples a directory containing several sample files to show Tsurgeon operation: - atree a sample natural-language tree in Penn Treebank annotation style. - exciseNP - renameVerb - relabelWithGroupName Sample tree-transformation operation files for Tsurgeon. See README-tsurgeon.txt for more information about the contents of these files. TREGEX ----------------------------------------------- Tregex Pattern Syntax and Uses Using a Tregex pattern, you can find only those trees that match the pattern you're looking for. The following table shows the symbols that are allowed in the pattern, and below there is more information about using these patterns. Table of Symbols and Meanings: A << B A dominates B A >> B A is dominated by B A < B A immediately dominates B A > B A is immediately dominated by B A $ B A is a sister of B (and not equal to B) A .. B A precedes B A . B A immediately precedes B A ,, B A follows B A , B A immediately follows B A <<, B B is a leftmost descendent of A A <<- B B is a rightmost descendent of A A >>, B A is a leftmost descendent of B A >>- B A is a rightmost descendent of B A <, B B is the first child of A A >, B A is the first child of B A <- B B is the last child of A A >- B A is the last child of B A <` B B is the last child of A A >` B A is the last child of B A 0) A >i B A is the ith child of B (i > 0) A <-i B B is the ith-to-last child of A (i > 0) A >-i B A is the ith-to-last child of B (i > 0) A <: B B is the only child of A A >: B A is the only child of B A <<: B A dominates B via an unbroken chain (length > 0) of unary local trees. A >>: B A is dominated by B via an unbroken chain (length > 0) of unary local trees. A $++ B A is a left sister of B (same as $.. for context-free trees) A $-- B A is a right sister of B (same as $,, for context-free trees) A $+ B A is the immediate left sister of B (same as $. for context-free trees) A $- B A is the immediate right sister of B (same as $, for context-free trees) A $.. B A is a sister of B and precedes B A $,, B A is a sister of B and follows B A $. B A is a sister of B and immediately precedes B A $, B A is a sister of B and immediately follows B A <+(C) B A dominates B via an unbroken chain of (zero or more) nodes matching description C A >+(C) B A is dominated by B via an unbroken chain of (zero or more) nodes matching description C A .+(C) B A precedes B via an unbroken chain of (zero or more) nodes matching description C A ,+(C) B A follows B via an unbroken chain of (zero or more) nodes matching description C A <<# B B is a head of phrase A A >># B A is a head of phrase B A <# B B is the immediate head of phrase A A ># B A is the immediate head of phrase B A == B A and B are the same node A : B [this is a pattern-segmenting operator that places no constraints on the relationship between A and B] Label descriptions can be literal strings, which much match labels exactly, or regular expressions in regular expression bars: /regex/. Literal string matching proceeds as String equality. In order to prevent ambiguity with other Tregex symbols, only standard "identifiers" are allowed as literals, i.e., strings matching [a-zA-Z]([a-zA-Z0-9_])* . If you want to use other symbols, you can do so by using a regular expression instead of a literal string. A disjunctive list of literal strings can be given separated by '|'. The special string '__' (two underscores) can be used to match any node. (WARNING!! Use of the '__' node description may seriously slow down search.) If a label description is preceeded by '@', the label will match any node whose basicCategory matches the description. NB: A single '@' thus scopes over a disjunction specified by '|': @NP|VP means things with basic category NP or VP. Label description regular expressions are matched as find(), as in Perl/tgrep; you need to specify ^ or $ to constrain matches. In a chain of relations, all relations are relative to the first node in the chain. For example, (S < VP < NP) means an S over a VP and also over an NP. If instead what you want is an S above a VP above an NP, you should write S < (VP < NP). Nodes can be grouped using parentheses '(' and ')' as in S < (NP $++ VP) to match an S over an NP, where the NP has a VP as a right sister. Boolean relational operators Relations can be combined using the '&' and '|' operators, negated with the '!' operator, and made optional with the '?' operator. Thus (NP < NN | < NNS) will match an NP node dominating either an NN or an NNS. (NP > S & $++ VP) matches an NP that is both under an S and has a VP as a right sister. Relations can be grouped using brackets '[' and ']'. So the expression NP [< NN | < NNS] & > S matches an NP that (1) dominates either an NN or an NNS, and (2) is under an S. Without brackets, & takes precedence over |, and equivalent operators are left-associative. Also note that & is the default combining operator if the operator is omitted in a chain of relations, so that the two patterns are equivalent: (S < VP < NP) (S < VP & < NP) As another example, (VP < VV | < NP % NP) can be written explicitly as (VP [< VV | [< NP & % NP] ] ). Relations can be negated with the '!' operator, in which case the expression will match only if there is no node satisfying the relation. For example (NP !< NNP) matches only NPs not dominating an NNP. Label descriptions can also be negated with '!': (NP < !NNP|NNS) matches NPs dominating some node that is not an NNP or an NNS. Relations can be made optional with the '?' operator. This way the expression will match even if the optional relation is not satisfied. This is useful when used together with node naming (see below). Basic Categories In order to consider only the "basic category" of a tree label, i.e. to ignore functional tags or other annotations on the label, prefix that node's description with the @ symbol. For example (@NP < @/NN.?/). This can only be used for individual nodes; if you want all nodes to use the basic category, it would be more efficient to use a TreeNormalizer to remove functional tags before passing the tree to the TregexPattern. Segmenting patterns The ":" operator allows you to segment a pattern into two pieces. This can simplify your pattern writing. For example, the pattern S : NP matches only those S nodes in trees that also have an NP node. Naming nodes Nodes can be given names (a.k.a. handles) using '='. A named node will be stored in a map that maps names to nodes so that if a match is found, the node corresponding to the named node can be extracted from the map. For example (NP < NNP=name) will match an NP dominating an NNP and after a match is found, the map can be queried with the name to retreived the matched node using {@link TregexMatcher#getNode(Object o)} with (String) argument "name" (not "=name"). Note that you are not allowed to name a node that is under the scope of a negation operator (the semantics would be unclear, since you can't store a node that never gets matched to). Trying to do so will cause a ParseException to be thrown. Named nodes can be put within the scope of an optional operator. Named nodes that refer back to previous named nodes need not have a node description -- this is known as "backreferencing". In this case, the expression will match only when all instances of the same name get matched to the same tree node. For example, the pattern: (@NP <, (@NP $+ (/,/ $+ (@NP $+ /,/=comma))) <- =comma) matches only an NP dominating exactly the sequence NP, NP; the mother NP cannot have any other daughters. Multiple backreferences are allowed. If the node with no node description does not refer to a previously named node, there will be no error, the expression simply will not match anything. Another way to refer to previously named nodes is with the "link" symbol: '~'. A link is like a backreference, except that instead of having to be equal to the referred node, the current node only has to match the label of the referred to node. A link cannot have a node description, i.e. the '~' symbol must immediately follow a relation symbol. Variable Groups If you write a node description using a regular expression, you can assign its matching groups to variable names. If more than one node has a group assigned to the same variable name, then matching will only occur when all such groups capture the same string. This is useful for enforcing coindexation constraints. The syntax is: / /#% For example, the pattern (designed for Penn Treebank trees): @SBAR < /^WH.*-([0-9]+)$/#1%index<<(__=empty < (/^-NONE-/< /^\\*T\\*-([0-9]+)$/#1%index)) will match only such that the WH- node under the SBAR is coindexed with the trace node that gets the name empty. MISCELLANEOUS ----------------------------------------------- Head Finders To use the headship relations <# ># <<# >># correctly it is important to specify a HeadFinder class appropriate to the trees that you are searching. For information about how to specify a HeadFinder class at the command line or through the API, please read the javadocs for the class edu.stanford.nlp.trees.tregex.TregexPattern. The following HeadFinder classes are included with the Tregex distribution: Penn Treebank of English (http://www.cis.upenn.edu/~treebank/): edu.stanford.nlp.trees.CollinsHeadFinder (default) Penn Treebank of Chinese (http://www.cis.upenn.edu/~chinese/): edu.stanford.nlp.trees.international.pennchinese.ChineseHeadFinder Penn Treebank of Arabic (http://www.ircs.upenn.edu/arabic/): edu.stanford.nlp.trees.international.arabic.ArabicHeadFinder NEGRA (http://www.coli.uni-saarland.de/projects/sfb378/negra-corpus/) and TIGER (http://www.ims.uni-stuttgart.de/projekte/TIGER/TIGERCorpus/) treebanks of German (these can use the same headfinder): edu.stanford.nlp.trees.international.negra.NegraHeadFinder Tuebingen Treebank of Written German (http://www.sfs.uni-tuebingen.de/en/ascl/resources/corpora/tueba-dz.html): edu.stanford.nlp.trees.international.tuebadz.TueBaDZHeadFinder Tdiff TregexGUI supports a constituent diff'ing method--similar to the UNIX diff command--for trees. To enable Tdiff: 1) Clear the tree file list: File -> Clear tree file list 2) Enable Tdiff: Options -> Tdiff 3) Load two (2) files using the "File -> Load" dialog. 4) Select "Browse" on the main display The GUI will display differences between each pair of trees in the two files. As such, the two files must contain the same number of trees. The first file in the tree file list is treated as the reference. Trees from the second file will be displayed in the GUI, with bracketing differences highlighted in blue. Below the tree, constituents in the reference tree that do not appear in the tree from the second file are shown as lines below each respective span. Tregex searches are supported and apply to the trees in the second file. This feature was designed for debugging and analyzing parser output. THANKS ----------------------------------------------- Thanks to the members of the Stanford Natural Language Processing Lab for great collaborative work on Java libraries for natural language processing. http://nlp.stanford.edu/javanlp/ LICENSE ----------------------------------------------- Tregex, Tsurgeon, and Interactive Tregex Copyright (c) 2003-2012 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/ . For more information, bug reports, fixes, contact: Christopher Manning Dept of Computer Science, Gates 2A Stanford CA 94305-9020 USA parser-support@lists.stanford.edu http://nlp.stanford.edu/software/tregex.html CONTACT ----------------------------------------------- For questions about this distribution, please contact Stanford's JavaNLP group at parser-support@lists.stanford.edu. We provide assistance on a best-effort basis. --- ### Doc Tregex Tregex Manifest (`doc/tregex/tregex-manifest.txt`) Manifest-Version: 1.0 Created-By: Stanford JavaNLP Main-class: edu.stanford.nlp.trees.tregex.gui.TregexGUI Class-Path: ../lib/AppleJavaExtensions.jar --- ### Doc Tsurgeon README Tsurgeon (`doc/tsurgeon/README-tsurgeon.txt`) Tsurgeon v4.2.0 - 2020-11-17 ---------------------------------------------- Copyright (c) 2003-2020 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. Original core Tregex code by Roger Levy and Galen Andrew. Original core Tsurgeon code by Roger Levy. GUI by Anna Rafferty. Support code, additional features, etc. by Chris Manning. This release prepared by John Bauer. This package contains Tregex and Tsurgeon. Tregex is a Tgrep2-style utility for matching patterns in trees. It can be run in a graphical user interface, from the command line using the TregexPattern main method, or used programmatically in java code via the TregexPattern, TregexMatcher and TregexPatternCompiler classes. As of version 1.2, the Tsurgeon tree-transformation utility is bundled together with Tregex. See the file README.tsurgeon for details. Java version 1.8+ is required to use the current version of Tregex. TSURGEON ---------------------------------------------- Tsurgeon is a tool for modifying trees that match a particular Tregex pattern. Further documentation for Tregex and Tregex GUI can be found in README-tregex.txt and README-gui.txt, respectively. ---------------------------------------------- Brief description: Takes some trees, tries to match one or more tregex expressions to each tree, and for each successful match applies some surgical operations to the tree. Pretty-prints each resulting tree (after all successful match/operation sets have applied) to standard output. A simple example: ./tsurgeon.csh -treeFile atree exciseNP renameVerb ----------------------------------------- RUNNING TREGEX ----------------------------------------- Program Command Line Options: -treeFile specify the name of the file that has the trees you want to transform. -po Apply a single operation to every tree using the specified match pattern and the specified operation. -s Prints the output trees one per line, instead of pretty-printed. The arguments are then Tsurgeon scripts. Each argument should be the name of a transformation file that contains a list of pattern and transformation operation list pairs. That is, it is a sequence of pairs of a TregexPattern pattern on one or more lines, then a blank line (empty or whitespace), then a list of transformation operations one per line (as specified by Tsurgeon syntax below to apply when the pattern is matched, and then another blank line (empty or whitespace). Note the need for blank lines: The code crashes if they are not present as separators (although the blank line at the end of the file can be omitted). The script file can include comment lines, either whole comment lines or trailing comments introduced by %, which extend to the end of line. A needed percent mark in patterns or operations can be escaped by a preceding backslash. ----------------------------------------- TSURGEON SYNTAX ----------------------------------------- Legal operation syntax and semantics (see Examples section for further detail): delete ... For each name_i, deletes the node it names and everything below it. prune ... For each name_i, prunes out the node it names. Pruning differs from deletion in that if pruning a node causes its parent to have no children, then the parent is in turn pruned too. excise The name1 node should either dominate or be the same as the name2 node. This excises out everything from name1 to name2. All the children of name2 go into the parent of name1, where name1 was. relabel Relabels the node to have the new label. There are three possible forms for the new-label: relabel nodeX VP - for changing a node label to an alphanumeric string, relabel nodeX /''/ - for relabeling a node to something that isn't a valid identifier without quoting, and relabel nodeX /^VB(.*)$/verb\/$1/ - for regular expression based relabeling. In the last case, all matches of the regular expression against the node label are replaced with the replacement String. This has the semantics of Java/Perl's replaceAll: you may use capturing groups and put them in replacements with $n. Also, as in the example, you can escape a slash in the middle of the second and third forms with \/ and \\. This last version lets you make a new label that is an arbitrary String function of the original label and additional characters that you supply. insert insert inserts the named node, or a manually specified tree (see below for syntax), into the position specified. Right now the only ways to specify position are: $+ the left sister of the named node $- the right sister of the named node >i the i_th daughter of the named node. >-i the i_th daughter, counting from the right, of the named node. move moves the named node into the specified position. To be precise, it deletes (*NOT* prunes) the node from the tree, and re-inserts it into the specified position. See above for how to specify position replace deletes name1 and inserts a copy of name2 in its place. adjoin adjoins the specified auxiliary tree (see below for syntax) into the target node specified. The daughters of the target node will become the daughters of the foot of the auxiliary tree. adjoinH similar to adjoin, but preserves the target node and makes it the root of . (It is still accessible as name. The root of the auxiliary tree is ignored.) adjoinF similar to adjoin, but preserves the target node and makes it the foot of . (It is still accessible as name, and retains its status as parent of its children. The foot of the auxiliary tree is ignored.) coindex ... Puts a (Penn Treebank style) coindexation suffix of the form "-N" on each of nodes name_1 through name_m. The value of N will be automatically generated in reference to the existing coindexations in the tree, so that there is never an accidental clash of indices across things that are not meant to be coindexed. ----------------------------------------- Syntax for trees to be inserted or adjoined: A tree to be adjoined in can be specified with LISP-like parenthetical-bracketing tree syntax such as those used for the Penn Treebank. For example, for the NP "the dog" to be inserted you might use the syntax (NP (Det the) (N dog)) That's all that there is for a tree to be inserted. Auxiliary trees (a la Tree Adjoining Grammar) must also have exactly one frontier node ending in the character "@", which marks it as the "foot" node for adjunction. Final instances of the character "@" in terminal node labels will be removed from the actual label of the tree. For example, if you wanted to adjoin the adverb "breathlessly" into a VP, you might specify the following auxiliary tree: (VP (Adv breathlessly) VP@ ) All other instances of "@" in terminal nodes must be escaped (i.e., appear as \@); this escaping will be removed by tsurgeon. In addition, any node of a tree can be named (the same way as in tregex), by appending = to the node label. That name can be referred to by subsequent tsurgeon operations triggered by the same match. All other instances of "=" in node labels must be escaped (i.e., appear as \=); this escaping will be removed by tsurgeon. For example, if you want to insert an NP trace somewhere and coindex it with a node named "antecedent" you might say insert (NP (-NONE- *T*=trace)) coindex trace antecedent $ ----------------------------------------- Examples of Tsurgeon operations: Tree (used in all examples): (ROOT (S (NP (NNP Maria_Eugenia_Ochoa_Garcia)) (VP (VBD was) (VP (VBN arrested) (PP (IN in) (NP (NNP May))))) (. .))) Apply delete: VP < PP=prep delete prep Result: (ROOT (S (NP (NNP Maria_Eugenia_Ochoa_Garcia)) (VP (VBD was) (VP (VBN arrested) (. .))) The PP node directly dominated by a VP is removed, as is everything under it. Apply prune: S < (NP < NNP=noun) prune noun Result: (ROOT (S (VP (VBD was) (VP (VBN arrested) (PP (IN in) (NP (NNP May))))) (. .))) The NNP node is removed, and since this results in the NP above it having no terminal children, the NP node is deleted as well. Note: This is different from delete in which the NP above the NNP would remain. Apply excise: VP < PP=prep excise prep prep Result: (ROOT (S (NP (NNP Maria_Eugenia_Ochoa_Garcia)) (VP (VBD was) (VP (VBN arrested) (IN in) (NP (NNP May))))) (. .))) The PP node is removed, and all of its children are added in the place it was previously located. Excise removes all the nodes from the first named node to the second named node, and the children of the second node are added as children of the parent of the first node. Thus, for another example: VP=verb < PP=prep excise verb prep Result: (ROOT (S (NP (NNP Maria_Eugenia_Ochoa_Garcia)) (VP (VBD was) (IN in) (NP (NNP May))) (. .))) Apply relabel: VP=v < PP=prep relabel prep verbPrep Result: (ROOT (S (NP (NNP Maria_Eugenia_Ochoa_Garcia)) (VP (VBD was) (VP (VBN arrested) (verbPrep (IN in) (NP (NNP May))))) (. .))) The label for the node called prep (PP) is changed to verbPrep. The other form of relabel uses regular expressions; consider the following operation: /^VB.+/=v relabel v /^VB(.*)$/ #1 Result: (ROOT (S (NP (NNP Maria_Eugenia_Ochoa_Garcia)) (VP (D was) (VP (N arrested) (PP (IN in) (NP (NNP May))))) (. .))) The Tregex pattern matches all nodes that begin "VB" and have at least one more character. The Tsurgeon operation then matches the node label to the regular expression "^VB(.*)$" and selects the text matching the first part that is not completely specified in the pattern. In this case, that is the part matching the wildcard (.*), which matches all characters after the VB. The node is then relabeled with that part of the text, causing, for example, "VBD" to be relabeled "D". The "#1" specifies that the name of the node should be the first group in the regex. Apply insert (shown here with inserting a node, but could also be a tree): S < (NP < (NNP=name !$- DET)) insert (DET Ms.) $+ name Result: (ROOT (S (NP (DET Ms.) (NNP Maria_Eugenia_Ochoa_Garcia)) (VP (VBD was) (VP (VBN arrested) (PP (IN in) (NP (NNP May))))) (. .))) The pattern matches the NNP node that is directly dominated by an NP (which is directly dominated by an S) and is not a direct right sister of a DET. Thus, the (DET Ms.) node is inserted immediately to the left of that NNP node, as specified by "$+ name". "$+" is the location and "name" describes what node the location is with respect to. Note: Tsurgeon will re-search for matches after each run of the script; thus, cycles may occur, causing the program to not terminate. The key is to write patterns that match prior to the changes you would like to make but that do not match afterwards. If the clause "!$- DET" had been left out in this example, Tsurgeon would have matched the pattern after every insert operation, causing an infinite number of DETs to be added. Apply move: VP=verb < PP=prep move prep $- verb Result: (ROOT (S (NP (NNP Maria_Eugenia_Ochoa_Garcia)) (VP (VBD was) (VP (VBN arrested))) (PP (IN in) (NP (NNP May))) (. .))) The PP is moved out of the VP that dominates it and added as a direct right sister of the VP. As for insert, "$-" specifies the location for prep while "verb" specifies what that location is relative to. Note: "move" is a macro operation that deletes the given node and then inserts it. "move" does not use prune, and thus any branches that now lack terminals will remain rather than being removed. Apply replace: S < (NP=name < NNP) replace name (NP (DET A) (NN woman)) Result: (ROOT (S (NP (DET A) (NN woman)) (VP (VBD was) (VP (VBN arrested) (PP (IN in) (NP (NNP May))))) (. .))) "name" is matched to an NP that is dominated by an S and dominates an NNP, and a new subtree ("(NP (DET A) (NN woman))") is added in the place where "name" was. Note: This operation is vulnerable to falling into an infinite loop. See the note concerning the "insert" operation and how patterns are matched. Apply adjoin: S < (NP=name < NNP) adjoin (NP (DET A) (NN woman) NP@) name Result: (ROOT (S (NP (DET A) (NN woman) (NP (NNP Maria_Eugenia_Ochoa_Garcia))) (VP (VBD was) (VP (VBN arrested) (PP (IN in) (NP (NNP May))))) (. .))) First, the NP is matched to the NP dominating the NNP tag. Then, the specified tree ("(NP (DET A) (NN woman) NP@)") is placed in that location. The "@" symbol specifies that the children of the original NP node ("name") are to be placed as children of a new NP node that is directly to the right of (NN woman). If the specified tree were "(NP (DET A) (NN woman) VP@)" then the child (NNP Maria_Eugenia_Ochoa_Garcia) would appear under a VP. Exactly one "@" node must appear in the specified tree in order to indicate where to place the node from the original tree. Apply adjoinH: S < (NP=name < NNP) adjoinH ((NP (DET A) (NN woman) NP@)) name Result: (ROOT (S (NP (NP (DET A) (NN woman) (NP (NNP Maria_Eugenia_Ochoa_Garcia)))) (VP (VBD was) (VP (VBN arrested) (PP (IN in) (NP (NNP May))))) (. .))) This operation differs from adjoin in that it retains the named node (in this case, "name"). The named node is made the root of the specified tree, resulting in two NP nodes dominating the DET in this example whereas only one was present in the previous example. Note that the specified tree is wrapped in an extra pair of parentheses in order to show the syntax for retaining the named node. If the extra parentheses were not there and the specified tree was, for example, (VP (DET A) (NN woman) NP@), the VP would be ignored in order to retain an NP as the root. Thus, in this case, "adjoinH (VP (DET A) (NN woman) NP@) name" and "adjoinH ((DET A) (NN woman) NP@) name" both produce the same tree: (ROOT (S (NP (DET A) (NN woman) (NP (NNP Maria_Eugenia_Ochoa_Garcia))) (VP (VBD was) (VP (VBN arrested) (PP (IN in) (NP (NNP May))))) (. .))) Apply adjoinF: S < (NP=name < NNP) adjoinF (NP(DET A) (NN woman) @) name Result: (ROOT (S (NP (DET A) (NN woman) (NP (NNP Maria_Eugenia_Ochoa_Garcia))) (VP (VBD was) (VP (VBN arrested) (PP (IN in) (NP (NNP May))))) (. .))) This operation is very similar to adjoin and adjoinH, but this time the original named node ("name" in this case) is maintained as the root of the subtree that is adjoined. Thus, no node label needs to be given in front of the "@" and if one is given, it will be ignored. For instance, "adjoinF (NP(DET A) (NN woman) VP@) name" would still produce the same tree as above, despite the VP preceding the @. Apply coindex: NP=node < NNP=name coindex node name Result: (ROOT (S (NP-1 (NNP-1 Maria_Eugenia_Ochoa_Garcia)) (VP (VBD was) (VP (VBN arrested) (PP (IN in) (NP-2 (NNP-2 May))))) (. .))) This causes the named nodes to be numbered such that all nodes that are part of the same match have the same number and all matches have distinct new names. We had two instances of an NP dominating an NNP in this example, and they were renamed such that NP-i < NNP-i for each match, with 1 <= i <= number of matches. ----------------------------------------- TSURGEON SCRIPTS ----------------------------------------- Script format: Tsurgeon scripts are a combination of a Tregex pattern to match and a series of Tsurgeon operations to perform on that match. The first line of a Tsurgeon script should be the Tregex pattern. This should be followed by a blank line, and then each subsequent line may contain one Tsurgeon operation. Tsurgeon operations should not be separated by blank lines. The following is an example of correctly formatted script: S < NP=node < NNP=name relabel node NP_NAME coindex node name Comments: The character % introduces a comment that extends to the end of the line. All other intended uses of % must be escaped as \% . ----------------------------------------- CONTACT ----------------------------------------- For questions about this distribution, please contact Stanford's JavaNLP group at parser-support@lists.stanford.edu. We provide assistance on a best-effort basis. ----------------------------------------- LICENSE ----------------------------------------- Tregex, Tsurgeon, and Interactive Tregex Copyright (c) 2003-2011 The Board of Trustees of The Leland Stanford Junior University. All Rights Reserved. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/ . For more information, bug reports, fixes, contact: Christopher Manning Dept of Computer Science, Gates 2A Stanford CA 94305-9020 USA parser-support@lists.stanford.edu http://nlp.stanford.edu/software/tregex.html --- ### Doc Tokensregex Examples Color.Rules (`doc/tokensregex/examples/color.rules.txt`) # Colors # Example to add "COLOR" as ner tag, and hex RGB code as the normalized tag for strings matching a color # Case insensitive pattern matching (see java.util.regex.Pattern flags) ENV.defaultStringPatternFlags = 2 # Map variable names to annotation keys ner = { type: "CLASS", value: "edu.stanford.nlp.ling.CoreAnnotations$NamedEntityTagAnnotation" } normalized = { type: "CLASS", value: "edu.stanford.nlp.ling.CoreAnnotations$NormalizedNamedEntityTagAnnotation" } tokens = { type: "CLASS", value: "edu.stanford.nlp.ling.CoreAnnotations$TokensAnnotation" } # Create OR pattern of # regular expression over tokens to hex RGB code # for colors and save it in a variable $Colors = ( /red/ => "#FF0000" | /green/ => "#00FF00" | /blue/ => "#0000FF" | /magenta/ => "#FF00FF" | /cyan/ => "#00FFFF" | /orange/ => "#FF7F00" | /brown/ => "#964B00" | /purple/ => "#800080" | /gray/ => "#777777" | /black/ => "#000000" | /white/ => "#FFFFFF" | (/pale|light/) /blue/ => "#ADD8E6" ) # Define ruleType to be over tokens ENV.defaults["ruleType"] = "tokens" # Define rule that # upon matching pattern defined by $Color # annotate matched tokens ($0) with ner="COLOR" and normalized=matched value ($$0.value) { pattern: ( $Colors ), action: ( Annotate($0, ner, "COLOR"), Annotate($0, normalized, $$0.value ) ) } --- ### Doc Tokensregex Examples Retokenize (`doc/tokensregex/examples/retokenize.txt`) # Uses TokensRegex library to split further split tokens with - # Map variable names to annotation keys tokens = { type: "CLASS", value: "edu.stanford.nlp.ling.CoreAnnotations$TokensAnnotation" } # Indicates matched expressions should replace the tokens annotation options.matchedExpressionsAnnotationKey = tokens; # Indicates that the matched expressions and tokens should be combined options.extractWithTokens = TRUE; # Indicates that the matched expressions should be flattened to be just tokens options.flatten = TRUE; # Mark units # Define ruleType to be over tokens ENV.defaults["ruleType"] = "tokens" # Case insensitive pattern matching (see java.util.regex.Pattern flags) ENV.defaultStringPatternFlags = 2 # Indicates that after matching, and computing the result, the result should be placed in the tokens annotation ENV.defaultResultAnnotationKey = tokens # Define rule where upon matching tokens with -, the matched token is split { pattern: ( /.+-.+/ ), result: Split($0[0], /-/, TRUE) } --- --- METRICS --- - Files Extracted: 35 - Estimated Token Budget: ~41666 tokens - Canonical Reference: https://codewiki.google/github.com/stanfordnlp/CoreNLP