awesome-nlp Documentation

Book: A curated list of resources dedicated to Natural Language Processing (NLP)

18,805 stars Unknown 1 file · ~32,216 tokens #awesome#awesome-list#deep-learning#language#machine-learning#natural-language-processing#nlp#text-mining
RAW Doc

1. Project Overview & Quickstart (keon/awesome-nlp)

File: README.md

awesome-nlp

Sponsored by Atlas Cloud

AI API aggregation platform with an OpenAI-compatible LLM endpoint for NLP tasks such as translation, summarization, multilingual generation, and structured extraction.

A curated list of resources dedicated to Natural Language Processing

Please read the contribution guidelines before contributing. Please add your favourite NLP resource by raising a pull request

Scope

This list covers natural language processing — linguistic analysis, multilingual tooling, classical and neural methods, datasets, and evaluation. Large language models are included only where they advance or evaluate a core NLP task or capability (tokenization, multilinguality, MT, summarization, NER, QA, factuality, probing, distillation). General-purpose chatbots, agent frameworks, prompt-template repositories, code-generation tools, and RAG application starter kits live in other lists — see See Also.

Contents

Where to follow current NLP research:

Historical highlights

Prominent NLP Research Labs

Back to Top

Tutorials

Back to Top

Reading Content

General Machine Learning

Introductions and Guides to NLP

Blogs and Newsletters

Videos and Online Courses

Back to Top

Books

Libraries

Back to Top

  • Node.js and Javascript - Node.js Libaries for NLP | Back to Top

    • Twitter-text - A JavaScript implementation of Twitter's text processing library
    • Knwl.js - A Natural Language Processor in JS
    • Retext - Extensible system for analyzing and manipulating natural language
    • NLP Compromise - Natural Language processing in the browser
    • Natural - general natural language facilities for node
    • Poplar - A web-based annotation tool for natural language processing (NLP)
    • NLP.js - An NLP library for building bots
    • node-question-answering - Fast and production-ready question answering w/ DistilBERT in Node.js

  • Python - Python NLP Libraries | Back to Top

    • sentimental-onix Sentiment models for spacy using onnx
    • TextAttack - Adversarial attacks, adversarial training, and data augmentation in NLP
    • TextBlob - Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of Natural Language Toolkit (NLTK) and Pattern, and plays nicely with both :+1:
    • spaCy - Industrial strength NLP with Python and Cython :+1:
      • textacy - Higher level NLP built on spaCy
    • gensim - Python library to conduct unsupervised semantic modelling from plain text :+1:
    • scattertext - Python library to produce d3 visualizations of how language differs between corpora
    • GluonNLP (archived) - A deep learning toolkit for NLP, built on MXNet/Gluon.
    • AllenNLP (archived) - An NLP research library, built on PyTorch, for developing state-of-the-art deep learning models on a wide variety of linguistic tasks.
    • PyTorch-NLP - NLP research toolkit designed to support rapid prototyping with better data loaders, word vector loaders, neural network layer representations, common NLP metrics such as BLEU
    • Rosetta - Text processing tools and wrappers (e.g. Vowpal Wabbit)
    • PyNLPl - Python Natural Language Processing Library. General purpose NLP library for Python, handles some specific formats like ARPA language models, Moses phrasetables, GIZA++ alignments.
    • foliapy - Python library for working with FoLiA, an XML format for linguistic annotation.
    • PySS3 - Python package implementing the SS3 white-box text classifier; ships with interactive visualization tools that explain predictions.
    • jPTDP - A toolkit for joint part-of-speech (POS) tagging and dependency parsing. jPTDP provides pre-trained models for 40+ languages.
    • BigARTM - a fast library for topic modelling
    • Snips NLU - A production ready library for intent parsing
    • Chazutsu - A library for downloading&parsing standard NLP research datasets
    • Word Forms - Word forms can accurately generate all possible forms of an English word
    • Multilingual Latent Dirichlet Allocation (LDA) - A multilingual and extensible document clustering pipeline
    • Natural Language Toolkit (NLTK) - A library containing a wide variety of NLP functionality, supporting over 50 corpora.
    • NLP Architect - A library for exploring the state-of-the-art deep learning topologies and techniques for NLP and NLU
    • Flair - A very simple framework for state-of-the-art multilingual NLP built on PyTorch. Includes BERT, ELMo and Flair embeddings.
    • Kashgari - Simple, Keras-powered multilingual NLP framework, allows you to build your models in 5 minutes for named entity recognition (NER), part-of-speech tagging (PoS) and text classification tasks. Includes BERT and word2vec embedding.
    • FARM - Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
    • Haystack - End-to-end Python framework for building natural language search interfaces to data. Leverages Transformers and the State-of-the-Art of NLP. Supports DPR, Elasticsearch, HuggingFace’s Modelhub, and much more!
    • Rita DSL - a DSL, loosely based on RUTA on Apache UIMA. Allows to define language patterns (rule-based NLP) which are then translated into spaCy, or if you prefer less features and lightweight - regex patterns.
    • Transformers - Natural Language Processing for TensorFlow 2.0 and PyTorch.
    • Tokenizers - Tokenizers optimized for Research and Production.
    • fairSeq Facebook AI Research implementations of SOTA seq2seq models in Pytorch.
    • corex_topic - Hierarchical Topic Modeling with Minimal Domain Knowledge
    • Sockeye - Neural Machine Translation (NMT) toolkit that powers Amazon Translate.
    • DL Translate - A deep learning-based translation library for 50 languages, built on transformers and Facebook's mBART Large.
    • Jury - Evaluation of NLP model outputs offering various automated metrics.
    • python-ucto - Unicode-aware regular-expression based tokenizer for various languages. Python binding to C++ library, supports FoLiA format.
    • Pearmut - Human annotation tool for multilingual NLP tasks, such as machine translation.
    • Stanza - Stanford NLP's Python toolkit for tokenization, POS, lemma, dependency parsing, and NER across 70+ languages.
    • Sentence-Transformers - sentence/document embeddings, semantic search, and re-ranking; current standard for retrieval-style NLP.
    • Argilla - open-source data annotation and feedback collection platform for LLM and NLP datasets.
    • HuggingFace Datasets - standardized loaders and processing for thousands of NLP datasets.
    • HuggingFace Evaluate - reference implementations for NLP metrics.
    • sacrebleu - reproducible BLEU/chrF/TER scoring for machine translation.
    • COMET - learned MT metrics, current de-facto standard.
    • LangTest - 60+ test types for NLP model robustness, bias, and fairness.
    • yasbd-lib - High-accuracy, rule-based sentence boundary detector (SBD). Drop-in pysbd adapter, streaming APIs, CLI, and a spaCy component across 39+ languages.

  • C++ - C++ Libraries | Back to Top

    • InsNet - A neural network library for building instance-dependent NLP models with padding-free dynamic batching.
    • MIT Information Extraction Toolkit - C, C++, and Python tools for named entity recognition and relation extraction
    • CRF++ - Open source implementation of Conditional Random Fields (CRFs) for segmenting/labeling sequential data & other Natural Language Processing tasks.
    • CRFsuite - CRFsuite is an implementation of Conditional Random Fields (CRFs) for labeling sequential data.
    • BLLIP Parser - BLLIP Natural Language Parser (also known as the Charniak-Johnson parser)
    • colibri-core - C++ library, command line tools, and Python binding for extracting and working with basic linguistic constructions such as n-grams and skipgrams in a quick and memory-efficient way.
    • ucto - Unicode-aware regular-expression based tokenizer for various languages. Tool and C++ library. Supports FoLiA format.
    • libfolia - C++ library for the FoLiA format
    • frog - Memory-based NLP suite developed for Dutch: PoS tagger, lemmatiser, dependency parser, NER, shallow parser, morphological analyzer.
    • MeTA - ModErn Text Analysis: a C++ data sciences toolkit for mining big text data.
    • Mecab (Japanese)
    • Moses
    • StarSpace - a library from Facebook for creating embeddings of word-level, paragraph-level, document-level and for text classification
    • QSMM - adaptive probabilistic top-down and bottom-up parsers

  • Java - Java NLP Libraries | Back to Top

    • Stanford NLP
    • OpenNLP
    • NLP4J
    • Word2vec in Java
    • ReVerb Web-Scale Open Information Extraction
    • OpenRegex An efficient and flexible token-based regular expression language and engine.
    • CogcompNLP - Core libraries developed in the U of Illinois' Cognitive Computation Group.
    • MALLET - MAchine Learning for LanguagE Toolkit - package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text.
    • RDRPOSTagger - A robust POS tagging toolkit available (in both Java & Python) together with pre-trained models for 40+ languages.

  • Kotlin - Kotlin NLP Libraries | Back to Top

    • Lingua A language detection library for Kotlin and Java, suitable for long and short text alike
    • Kotidgy — an index-based text data generator written in Kotlin

  • Scala - Scala NLP Libraries | Back to Top

    • Saul - Library for developing NLP systems, including built in modules like SRL, POS, etc.
    • ATR4S - Toolkit with state-of-the-art automatic term recognition methods.
    • tm - Implementation of topic modeling based on regularized multilingual PLSA.
    • word2vec-scala - Scala interface to word2vec model; includes operations on vectors like word-distance and word-analogy.
    • Epic - Epic is a high performance statistical parser written in Scala, along with a framework for building complex structured prediction models.
    • Spark NLP - Spark NLP is a natural language processing library built on top of Apache Spark ML that provides simple, performant & accurate NLP annotations for machine learning pipelines that scale easily in a distributed environment.

  • R - R NLP Libraries | Back to Top

    • text2vec - Fast vectorization, topic modeling, distances and GloVe word embeddings in R.
    • wordVectors - An R package for creating and exploring word2vec and other word embedding models
    • RMallet - R package to interface with the Java machine learning tool MALLET
    • dfr-browser - Creates d3 visualizations for browsing topic models of text in a web browser.
    • dfrtopics - R package for exploring topic models of text.
    • sentiment_classifier - Sentiment Classification using Word Sense Disambiguation and WordNet Reader
    • jProcessing - Japanese Natural Langauge Processing Libraries, with Japanese sentiment classification
    • corporaexplorer - An R package for dynamic exploration of text collections
    • tidytext - Text mining using tidy tools
    • spacyr - R wrapper to spaCy NLP
    • CRAN Task View: Natural Language Processing

  • Clojure | Back to Top

    • Clojure-openNLP - Natural Language Processing in Clojure (opennlp)
    • Infections-clj - Rails-like inflection library for Clojure and ClojureScript
    • postagga - A library to parse natural language in Clojure and ClojureScript

  • Go | Back to Top

    • prose - Text processing library supporting tokenization, part-of-speech tagging, and named-entity extraction.
    • gojieba - Go implementation of the jieba Chinese word segmentation algorithm.
    • kagome - Japanese morphological analyzer written in pure Go.
    • go-propisyu - Converts numbers to Russian words with correct grammatical gender and noun declension.

  • Ruby | Back to Top

  • Rust | Back to Top

    • whatlang — Natural language recognition library based on trigrams
    • rust-bert - Ready-to-use NLP pipelines and Transformer-based models
    • snips-nlu-rs (archived — Snips was discontinued) - A production ready library for intent parsing

  • NLP++ - NLP++ Language | Back to Top

  • Julia | Back to Top

    • CorpusLoaders - A variety of loaders for various NLP corpora
    • Languages - A package for working with human languages
    • TextAnalysis - Julia package for text analysis
    • TextModels - Neural Network based models for Natural Language Processing
    • WordTokenizers - High performance tokenizers for natural language processing and other related tasks
    • Word2Vec - Julia interface to word2vec

Services

NLP as API with higher level functionality such as NER, Topic tagging and so on | Back to Top

  • Wit-ai - Natural Language Interface for apps and devices
  • IBM Watson's Natural Language Understanding - API and Github demo
  • Amazon Comprehend - NLP and ML suite covers most common tasks like NER, tagging, and sentiment analysis
  • Google Cloud Natural Language API - Syntax Analysis, NER, Sentiment Analysis, and Content tagging in atleast 9 languages include English and Chinese (Simplified and Traditional).
  • ParallelDots - High level Text Analysis API Service ranging from Sentiment Analysis to Intent Analysis
  • Microsoft Cognitive Service
  • TextRazor
  • Rosette
  • Textalytic - Natural Language Processing in the Browser with sentiment analysis, named entity extraction, POS tagging, word frequencies, topic modeling, word clouds, and more
  • NLP Cloud - SpaCy NLP models (custom and pre-trained ones) served through a RESTful API for named entity recognition (NER), POS tagging, and more.
  • Cloudmersive - Unified and free NLP APIs that perform actions such as speech tagging, text rephrasing, language translation/detection, and sentence parsing

Annotation Tools

  • GATE - General Architecture and Text Engineering is 15+ years old, free and open source
  • Anafora is free and open source, web-based raw text annotation tool
  • brat - brat rapid annotation tool is an online environment for collaborative text annotation
  • doccano - doccano is free, open-source, and provides annotation features for text classification, sequence labeling and sequence to sequence
  • INCEpTION - A semantic annotation platform offering intelligent assistance and knowledge management
  • prodigy is an annotation tool powered by active learning, costs $
  • LightTag - Hosted and managed text annotation tool for teams, costs $
  • rstWeb - open source local or online tool for discourse tree annotations
  • GitDox - open source server annotation tool with GitHub version control and validation for XML data and collaborative spreadsheet grids
  • Datasaur support various NLP tasks for individual or teams, freemium based
  • Konfuzio - team-first hosted and on-prem text, image and PDF annotation tool powered by active learning, freemium based, costs $
  • UBIAI - Easy-to-use text annotation tool for teams with most comprehensive auto-annotation features. Supports NER, relations and document classification as well as OCR annotation for invoice labeling, costs $
  • Shoonya - Shoonya is free and open source data annotation platform with wide varities of organization and workspace level management system. Shoonya is data agnostic, can be used by teams to annotate data with various level of verification stages at scale.
  • Annotation Lab - Free End-to-End No-Code platform for text annotation and DL model training/tuning. Out-of-the-box support for Named Entity Recognition, Classification, Relation extraction and Assertion Status Spark NLP models. Unlimited support for users, teams, projects, documents. Not FOSS.
  • FLAT - FLAT is a web-based linguistic annotation environment based around the FoLiA format, a rich XML-based format for linguistic annotation. Free and open source.
  • Argilla - open-source platform for collecting human feedback, building NLP and LLM datasets, and curating preference data.
  • Label Studio - open-core multi-modal labeling platform; widely used for NLP labeling.
  • Potato - Free, open-source annotation tool covering 21+ task types (classification, span, coreference, entity linking, agent trace evaluation) with built-in MACE quality control, attention checks, AI-assisted labeling, and 300+ example tasks.

Tasks and Methods

NLP tasks organized by linguistic problem. Each subsection lists foundational/classical work first, then neural approaches, then LLM-based methods where relevant. For modern LM-specific research (pretraining, evaluation, retrieval, reasoning, etc.) see Language Models for NLP.

Text Embeddings

Back to Top

Static word embeddings (foundational):

Contextual embeddings:

  • ELMo - deep contextualized word representations.
  • CoVe - contextualized vectors learned from MT.
  • ULMFiT - language-model fine-tuning for text classification.
  • InferSent - sentence representations from NLI.

Modern sentence and document embeddings: see Retrieval for NLP (Sentence-Transformers, E5, BGE-M3, Nomic, GritLM) and MTEB for current leaderboards.

Tokenization, Morphology, and Segmentation

Back to Top

  • SentencePiece - language-agnostic subword tokenization.

  • BPE and Unigram LM - the two dominant subword schemes.

  • Stanza - tokenization, lemma, and morphology for 70+ languages.

  • UDPipe - tokenization, tagging, lemmatization, parsing for Universal Dependencies.

  • Morfessor - unsupervised morphological segmentation.
    Tokenizer research and architecture (also see Language Models):

  • Byte-Pair Encoding (Sennrich et al.) - subword units for neural MT; foundation of modern tokenizers.

  • SentencePiece - language-agnostic subword tokenization (BPE and Unigram).

  • Tokenizers - fast Rust implementations of BPE, WordPiece, Unigram.

  • ByT5 - tokenizer-free byte-level model.

  • CANINE - tokenization-free encoder operating on Unicode characters.

  • How Good is Your Tokenizer? - tokenizer fairness across languages.

  • Byte Latent Transformer (BLT) (Meta, 2024) - dynamic byte-level patching that matches BPE-tokenized models at scale; revives the tokenizer-free direction.

  • SuperBPE (2025) - superword tokenization that improves on BPE for downstream tasks.

  • Over-Tokenized Transformer (ICML 2025) - decouples input and output vocabularies; shows a log-linear relationship between input vocabulary size and training loss, scaling vocabulary independently of model size.

  • Foundations of Tokenization (ICLR 2025) - first formal unified framework for tokenizer models using stochastic-map category theory; establishes conditions for statistical consistency.

  • The Token Tax: Systematic Bias in Multilingual Tokenization (2025) - quantifies how tokenization fertility predicts model accuracy across languages, exposing structural cost penalties for morphologically complex and low-resource languages.

  • Reducing Tokenization Premiums for Low-Resource Languages (2026) - post-hoc vocabulary additions that coalesce multi-token character sequences for low-resource languages, reducing inference cost without retraining.

POS Tagging and Dependency Parsing

Back to Top

Named Entity Recognition and Information Extraction

Back to Top

Foundational and neural:

Open and instruction-following IE:

  • Universal NER - instruction-tuned LM for open-set NER across languages.
  • GLiNER (2023) - small, generalist NER model that handles arbitrary entity types at inference.
  • GoLLIE - guideline-following information extraction with LMs.
  • REBEL - end-to-end relation extraction as seq2seq.

LLM-based:

Coreference Resolution

Back to Top

Text Classification and Sentiment Analysis

Back to Top

Topic Modeling

Back to Top

Summarization

Back to Top

Machine Translation

Back to Top

Statistical and foundational neural:

Massively multilingual:

Evaluation:

  • COMET - learned MT metric; current de-facto standard alongside chrF.
  • sacrebleu - reproducible BLEU/chrF/TER scoring.
  • BERTScore - similarity-based generation metric.

LLM-based:

Question Answering and Reading Comprehension

Back to Top

Datasets and foundational systems:

Modern open-domain QA:

  • DPR and FiD - retrieve-then-read; the standard pre-LLM open-domain QA pipeline.
  • Atlas - retrieval-augmented LM for few-shot QA.
  • See also Retrieval for NLP.

LLM-era:

Information Extraction Beyond NER

Back to Top

Retrieval and Embeddings

Back to Top

Dense and late-interaction retrieval, increasingly the substrate for QA and IR:

  • DPR (Dense Passage Retrieval) - dual-encoder retrieval baseline.

  • ColBERT and ColBERTv2 - late-interaction retrieval; strong on out-of-domain.

  • E5 and E5-Mistral - widely used dense embedding families.

  • BGE and BGE-M3 (2024) - multilingual, multi-functionality embeddings; top of MTEB across languages.

  • Nomic Embed (2024) - fully open, reproducible embedding model.

  • Matryoshka Representation Learning - nested embeddings supporting variable dimensionality at inference.

  • GritLM (2024) - unified generation and embedding from one model.

  • RAG (Retrieval-Augmented Generation) - the original retrieval-augmented framework; foundation for modern QA pipelines.

  • Gemini Embedding (2025) - Gemini-derived dense embeddings; SOTA on MMTEB across 250+ languages and on cross-lingual retrieval (XOR-Retrieve, XTREME-UP).

  • Qwen3-Embedding (2025) - decoder-based embedding series (0.6B-8B) built on Qwen3; #1 on MTEB Multilingual and MTEB Code, surpassing prior proprietary models.

  • Rank1 (2025) - first reranking model trained with test-time compute via DeepSeek-R1 reasoning-trace distillation; SOTA on instruction-following and OOD retrieval.

  • ReasonEmbed (2025) - embedding model for reasoning-intensive retrieval with ReMixer data synthesis and Redapter adaptive training; record nDCG@10 of 38.1 on BRIGHT.

  • ColBERT-Att (2026) - extends late-interaction retrieval by integrating query and document attention weights into ColBERT scoring; improves recall on MS-MARCO, BEIR, and LoTTE.
    Embedding and retrieval benchmarks:

  • MMTEB (2025) - community expansion of MTEB to 500+ tasks across 250+ languages.

Speech and Text

Back to Top

A short pointer set, since this borders adjacent fields:

  • Whisper - multilingual ASR; the modern open default.
  • SeamlessM4T - unified speech and text translation.
  • Canary (NVIDIA, 2024) - top open multilingual ASR model.
  • FunASR - industrial-grade ASR toolkit; 170× realtime on GPU, 50+ languages, built-in VAD, punctuation, speaker diarization, and emotion detection. Includes non-autoregressive SenseVoice and LLM-based Fun-ASR-Nano models.
  • Wav2Vec 2.0 - foundational self-supervised speech pretraining.
  • Coqui TTS and VieNeu-TTS - open TTS.

Datasets

Back to Top

Dataset hubs and lists:

Pretraining-scale corpora (open):

  • The Pile - 825 GiB diverse text corpus.
  • RedPajama / RedPajama-V2 (2023-2024) - reproductions of LLaMA pretraining data; V2 is 30T tokens with quality signals.
  • Dolma (AI2, 2023-2024) - 3T-token open pretraining corpus with documented filtering pipeline.
  • FineWeb / FineWeb-Edu (2024) - 15T-token cleaned web corpus; FineWeb-Edu filters for educational quality.
  • CulturaX - 6.3T tokens across 167 languages.
  • Common Corpus (2024) - 2T-token open-license multilingual corpus.

Task and instruction datasets:

Multilingual NLP Frameworks

Back to Top

  • UDPipe is a trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files. Primarily written in C++, offers a fast and reliable solution for multilingual NLP processing.
  • NLP-Cube : Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing. New platform, written in Python with Dynet 2.0. Offers standalone (CLI/Python bindings) and server functionality (REST API).
  • UralicNLP is an NLP library mostly for many endangered Uralic languages such as Sami languages, Mordvin languages, Mari languages, Komi languages and so on. Also some non-endangered languages are supported such as Finnish together with non-Uralic languages such as Swedish and Arabic. UralicNLP can do morphological analysis, generation, lemmatization and disambiguation.

Language Models for NLP

Back to Top

Pretrained language models and the research around them, scoped to NLP tasks and linguistic phenomena. For general-purpose LLM tooling, agents, or RAG application kits, see See Also.

Pretraining and Adaptation

Encoders (still the workhorse for classical NLP tasks):

  • BERT - bidirectional transformer pretraining; foundation for most encoder-based NLP work since 2018.
  • RoBERTa - robustly optimized BERT pretraining; common encoder baseline.
  • DeBERTa / DeBERTa-v3 - disentangled attention; strong on classification, NER, NLI.
  • ELECTRA - replaced-token-detection pretraining, sample-efficient.
  • ModernBERT (2024) - modernized encoder with rotary embeddings, FlashAttention, 8K context; current go-to encoder for classification, NER, retrieval.
  • NeoBERT (2025) - 250M-parameter encoder integrating modern architecture improvements (RoPE, 4K context, optimized depth-to-width); state of the art on MTEB, surpasses ModernBERT and RoBERTa-large under identical fine-tuning.

Encoder-decoder and seq2seq:

  • T5 and FLAN-T5 - text-to-text framing for NLP tasks; strong instruction-tuned encoder-decoder baselines.
  • BART - denoising seq2seq pretraining; widely used for summarization and generation.

Open decoder-only LMs (used as substrate for NLP tasks):

Multilingual and Cross-Lingual Models

  • XLM-R - cross-lingual masked LM trained on CommonCrawl, 100 languages.
  • mT5 - multilingual T5 covering 101 languages.
  • BLOOM - 176B-parameter open multilingual LM, 46 natural languages.
  • Aya 23 / Aya Expanse (Cohere For AI, 2024) - massively multilingual instruction-tuned models covering 23-101 languages.
  • Glot500 - encoder for 500+ languages, focus on low-resource.
  • NLLB-200 - No Language Left Behind: MT for 200 languages.
  • MADLAD-400 - 400+ language MT model and 3T-token multilingual corpus.
  • SeamlessM4T / Seamless (Meta, 2023-2024) - multilingual and multimodal speech-text translation, 100+ languages.
  • SEA-LION / SeaLLM (2024-2025) - LMs targeting Southeast Asian languages.
  • Babel (2025) - open multilingual LLMs (9B and 83B) covering the top 25 languages by speaker population (~90% of global speakers); surpasses comparably-sized open multilingual models on XCOPA, XNLI, MGSM, FLORES-200.
  • Lugha-Llama (Princeton/Mila, 2025) - Llama-3.1-8B adapted for low-resource African languages via the curated WURA corpus; SOTA open-source results on IrokoBench and AfriQA.
  • AfriqueLLM (McGill, 2026) - suite of open LLMs (4B-14B) continued-pretrained on 26B tokens across 20 African languages with a comprehensive empirical study of data mixing.
  • TranslateGemma (Google, 2026) - open translation-specialized models built on Gemma 3, covering 55 language pairs via SFT and RL with quality-reward models.
  • MiLMMT-46 (Xiaomi, 2026) - open multilingual MT scaled across 46 languages, matching commercial systems like Google Translate and Gemini 3 Pro.

Evaluation and Benchmarks

NLU and cross-lingual:

  • GLUE and SuperGLUE - English NLU benchmarks.
  • XTREME and XGLUE - cross-lingual NLU.
  • XNLI - cross-lingual natural language inference, 15 languages.
  • FLORES-200 - MT evaluation across 200 languages.
  • MTEB - Massive Text Embedding Benchmark; standard for sentence/document encoders.
  • BEIR - heterogeneous IR benchmark for retrieval models.

Modern LM evaluation (2023-2026):

  • HELM - holistic evaluation across NLP tasks, accuracy and beyond.
  • BIG-bench - 200+ tasks probing language model capabilities.
  • MMLU - multitask knowledge evaluation across 57 subjects.
  • MMLU-Pro (2024) - harder, more discriminative successor to MMLU.
  • GPQA - graduate-level Q&A, "Google-proof" reasoning evaluation.
  • REFUTE (2026) - scientific reasoning benchmark for evidence-grounded critique, overclaim detection, missing-evidence refusal, and calibration.
  • IFEval - verifiable instruction-following evaluation.
  • Chatbot Arena (LMSYS) - human-preference ELO leaderboard for chat models.
  • LiveBench (2024) - contamination-resistant benchmark with monthly refresh.
  • LM Evaluation Harness - unified framework for LM benchmark evaluation.
  • MMLU-ProX (2025) - multilingual extension of MMLU-Pro to 29 typologically diverse languages; reveals up to 24.3% performance gap between high- and low-resource languages.
  • MultiChallenge (2025) - multi-turn conversational benchmark exposing simultaneous instruction-following and in-context-reasoning failures; all tested frontier models score below 50%.
  • FRAMES (2025) - unified RAG evaluation: 824 multi-hop questions requiring factuality, retrieval accuracy, and cross-document reasoning together.

Long-context evaluation:

  • Needle in a Haystack - retrieval probe for long-context windows.
  • RULER (2024) - synthetic long-context tasks beyond simple retrieval.
  • LongBench - bilingual long-context benchmark across NLP tasks.
  • LongBench v2 (2025) - 503 expert-crafted multiple-choice questions spanning 8K-2M-word contexts with deep multi-hop reasoning; humans score 53.7% under time pressure.
  • U-NIAH (2025) - extends needle-in-haystack with multi-needle and nested configurations; shows RAG mitigates lost-in-the-middle for smaller LLMs but degrades reasoning models.

Reasoning and Test-Time Compute

A trend-defining direction in 2024-2026: models that produce explicit reasoning traces and benefit from extra inference compute.

  • Chain-of-Thought Prompting - foundational result; intermediate reasoning steps improve performance.
  • Self-Consistency - majority vote over sampled CoT chains.
  • Tree of Thoughts - search over reasoning trees.
  • Self-Refine and Reflexion - self-correction at inference time.
  • Large Language Models are Zero-Shot Reasoners - chain-of-thought for NLP reasoning tasks.
  • Let's Verify Step by Step - process-supervised reward models for reasoning.
  • DeepSeek-R1 (2025) - open reasoning model trained with pure RL; replicated o1-style behavior in the open.
  • OpenAI o1 / o3 (2024-2025) - test-time-compute reasoning systems.
  • Scaling LLM Test-Time Compute Optimally (2024) - systematic study of inference-time compute tradeoffs.
  • s1: Simple Test-Time Scaling (2025) - small open reasoning recipe via budget-forcing.
  • Kimi k1.5 (2025) - long-context RL with policy optimization (no MCTS, no PRM) reaching o1-level performance; introduces long-CoT distillation into short-CoT models.
  • rStar-Math (2025) - small policy model paired with a process preference model trained via MCTS rollouts; enables small LMs to bootstrap reasoning without distilling from larger models.
  • DAPO (2025) - open GRPO-based RL training system with four key improvements (decoupled clipping, dynamic sampling, token-level loss, entropy bonus); reproduces and surpasses DeepSeek-R1-Zero-level reasoning.
  • VAPO (2025) - value-model-based RL with length-adaptive GAE and token-level clipping; surpasses value-free GRPO methods on AIME 2024 with stable training.
  • ThinkPRM (2025) - generative process reward models that produce chain-of-thought verification per step, matching discriminative PRMs with 1% of the supervision labels.
  • OpenThoughts (2025) - 1000+ controlled experiments on data recipes for open reasoning models; SOTA on AIME 2025 matching closed distillation baselines.

Long Context and Alternative Architectures

  • Mamba and Mamba-2 - selective state-space models, linear-time long-context alternative to attention.
  • RWKV - RNN-transformer hybrid scaling to large parameter counts.
  • Jamba (2024) - hybrid Mamba-Transformer-MoE architecture.
  • RoPE and YaRN - rotary position embeddings and context-length extension.
  • Position Interpolation - extending context windows with minimal fine-tuning.
  • Lost in the Middle - long-context degradation patterns in NLP tasks.
  • RAG vs Long-Context LLMs (2024) - tradeoffs for QA over long inputs.
  • Titans: Learning to Memorize at Test Time (2025) - neural long-term memory module that learns to memorize historical context at test time; scales beyond 2M tokens, outperforms transformers and modern linear-recurrent models on language modeling and reasoning.
  • MiniMax-01 (2025) - 456B-parameter hybrid combining lightning (linear) attention with sparse softmax attention; matches GPT-4o-level NLP performance at up to 4M-token inference contexts.
  • Native Sparse Attention (NSA) (2025) - trainable sparse attention combining coarse-grained compression with fine-grained selection; large speedups at 64K with no NLP-benchmark degradation.
  • LongRoPE2 (2025) - identifies undertraining of high-frequency RoPE dimensions and applies evolutionary-search rescaling; extends LLaMA3-8B to 128K with 80x fewer training tokens than Meta's recipe.
  • Characterizing SSM and Hybrid LM Long-Context Performance (2025) - first comprehensive memory and speed analysis of transformer, SSM, and hybrid models up to 220K tokens; SSMs are up to 4x faster, hybrids balance recall and efficiency.

Factuality, Hallucination, Calibration

  • Survey of Hallucination in Natural Language Generation - taxonomy and mitigation strategies.
  • TruthfulQA - benchmark for truthfulness in question answering.
  • FActScore - fine-grained factual precision in long-form generation.
  • LongFact / SAFE (2024) - long-form factuality benchmark and search-augmented evaluator.
  • SelfCheckGPT - sampling-based hallucination detection.
  • RAGAS - reference-free evaluation for RAG and QA pipelines.
  • Lookback Lens (2024) - attention-pattern-based hallucination detection in long-context generation.
  • Calibration of LLMs on Multiple Choice (2024) - calibration analysis under format effects.
  • HalluLens (2025) - hallucination benchmark with extrinsic/intrinsic taxonomy and dynamic test-set regeneration to resist data leakage.
  • Atomic Calibration (2025) - claim-level calibration analysis for long-form generation; models are substantially worse-calibrated on extended outputs than on single claims.
  • FRANQ (2025) - faithfulness-aware uncertainty quantification for RAG fact-checking; formally separates faithfulness from factuality.
  • MUCH (2025) - multilingual claim-hallucination benchmark across English, French, Spanish, German with token-level logits released for principled UQ evaluation.
  • HalluHard (2026) - hard multi-turn hallucination benchmark for citation-required responses; ~30% hallucination rates persist even with web search.
  • CURE: Think Through Uncertainty (2026) - trains models to reason about claim-level uncertainty before generating; large gains on biography factuality and FactBench AUROC.

Probing and Interpretability

Efficient and Small Language Models

Distillation and small models:

  • DistilBERT and MiniLM - distilled encoders for production NLP.
  • Phi-3 / Phi-4 (Microsoft, 2024) - small models trained on curated data, competitive with much larger ones on NLP benchmarks.
  • SmolLM2 (HuggingFace, 2025) - fully open small-LM family with reproducible training data.
  • SmolLM3 (HuggingFace, 2025) - 3B fully open decoder pretrained on 11.2T tokens with NoPE and YaRN for 128K context; competitive with 4B-class models.
  • Gemma 3 Technical Report (Google, 2025) - 1B-27B open models with high local-to-global attention ratio to keep KV-cache tractable at 128K context.
  • Qwen3 Technical Report (Alibaba, 2025) - dense and MoE models 0.6B-235B with unified thinking/non-thinking modes; the 30B-A3B MoE matches larger dense models while activating only 3B parameters.
  • Apple Intelligence Foundation Language Models (Apple, 2025) - on-device 3B model using KV-cache sharing and 2-bit QAT for 37.5% cache memory reduction without accuracy loss.
  • Sentence-Transformers - sentence and paragraph embeddings via Siamese BERT.
  • SetFit - few-shot text classification without prompts.
  • FastFit - fast few-shot classification for many-class settings.
  • GTE, BGE, and Stella - compact text embedding models near the top of MTEB.

Quantization and serving (relevant when deploying NLP models at scale):

  • GPTQ - post-training quantization for transformers.
  • AWQ - activation-aware weight quantization.
  • KVTuner (ICML 2025) - sensitivity-aware layer-wise mixed-precision KV-cache quantization; up to 21% throughput improvement over uniform KV8.
  • GGUF / llama.cpp - portable quantized inference.
  • vLLM - PagedAttention-based high-throughput LM serving.
  • SGLang - structured generation and efficient serving.
  • Text Generation Inference (TGI) - HF production serving for LMs.

Parameter-efficient fine-tuning:

  • LoRA and QLoRA - low-rank adapters and quantized fine-tuning; the standard for adapting LMs to NLP tasks on modest hardware.
  • DoRA (2024) - weight-decomposed low-rank adaptation.
  • PEFT - HuggingFace library bundling LoRA, prefix tuning, IA3, and others.

Instruction Tuning and Preference Optimization

  • FLAN - finetuned language models as zero-shot learners.
  • InstructGPT - training LMs to follow instructions with human feedback.
  • Self-Instruct - bootstrapping instruction data from LMs.
  • Super-NaturalInstructions - 1600+ NLP tasks with instructions.
  • Constitutional AI - training LMs with AI-generated feedback against a written constitution.
  • Direct Preference Optimization - simpler alternative to RLHF; widely adopted.
  • Tülu 3 (AI2, 2024) - fully open post-training recipe with state-of-the-art results among open models.
  • LIMA - "less is more for alignment"; small high-quality SFT data goes a long way.
  • TRL - reference library for SFT, DPO, GRPO, and RLHF.
  • Magpie (2024-2025) - synthesizes high-quality instruction-response pairs by prompting aligned LMs with nothing; SFT on the filtered subset matches official Llama-3-Instruct.

Bias, Fairness, Safety in NLP

  • StereoSet - measuring stereotypical bias in pretrained LMs.
  • CrowS-Pairs - social bias measurement in masked LMs.
  • WinoBias - gender bias in coreference resolution.
  • HolisticBias - bias measurement across many demographic axes.
  • RealToxicityPrompts - toxicity in LM generation.
  • Sycophancy in Language Models - models tailoring answers to user beliefs.
  • Alignment Faking in Large Language Models (Anthropic, 2024) - models strategically complying during training.
  • WildGuard (2024) - open safety moderation model and benchmark.
  • Emergent Misalignment (2025) - finetuning on a narrow task (insecure code) unexpectedly produces broad alignment failures across unrelated domains.
  • SafeDialBench (2025) - multilingual (Chinese/English) safety benchmark of 4000+ multi-turn dialogues across 22 scenarios and 7 jailbreak strategies.
  • TeleAI-Safety (2025) - modular jailbreak evaluation framework integrating 19 attacks, 29 defenses, and 19 evaluation methods across 14 models and 12 risk categories.
  • IndicSafe (2026) - multilingual safety benchmark across 12 Indic languages; reveals 12.8% cross-language agreement, with over-refusal in low-resource scripts.
  • VLAF: Value-Conflict Alignment Faking (2026) - alignment faking occurs in models as small as 7B in 37% of cases when policy conflicts with internalized values; steering-vector mitigation reduces it 94%.

NLP per Language

Back to Top

Resources organized by human language. Click a section to expand.

NLP in Arabic

Back to Top

Libraries

  • CAMeL Tools - Python toolkit for Arabic NLP including dialect ID, morphology, NER.
  • goarabic - Go package for Arabic text processing.
  • jsastem - JavaScript Arabic stemmer.
  • PyArabic - Python library for Arabic.
  • RFTokenizer - trainable segmenter for Arabic, Hebrew, and Coptic.
  • Farasa - QCRI segmentation, POS tagging, and NER for Arabic.

Models and Embeddings

  • AraBERT - Arabic BERT family.
  • CAMeLBERT - BERT models for MSA, dialectal, and Classical Arabic.
  • AraELECTRA - efficient Arabic pretraining (released alongside AraBERT).
  • Jais (2023-2024) - bilingual Arabic-English open LM family.
  • ALLaM (SDAIA, 2024) - Arabic-first foundation models.

Datasets

NLP in Chinese

Back to Top

Libraries

  • jieba - Python package for Chinese word segmentation.
  • SnowNLP - Python package for Chinese NLP.
  • FudanNLP - Java library for Chinese text processing.
  • HanLP - multilingual NLP library with strong Chinese support.
  • LTP - HIT Language Technology Platform: segmentation, POS, NER, parsing.

Models and Embeddings

Anthology

  • funNLP - large collection of Chinese NLP tools and resources.

NLP in Danish

Back to Top

NLP in Dutch

Back to Top

  • python-frog - Python binding to Frog, an NLP suite for Dutch (POS tagging, lemmatization, dependency parsing, NER).
  • SimpleNLG_NL - Dutch surface realiser for natural language generation, based on the SimpleNLG implementation.
  • Alpino - dependency parser for Dutch (also does POS tagging and lemmatization).
  • Kaldi NL - Dutch speech-recognition models based on Kaldi.
  • spaCy Dutch model - industrial-strength NLP with a Dutch pipeline.

NLP in German

Back to Top

  • German-NLP - curated list of open-access, open-source, and off-the-shelf resources and tools developed with a focus on German.

NLP in Hungarian

Back to Top

NLP in Indic Languages

Back to Top

Data, Corpora and Treebanks

Corpora/Datasets that need a login/access can be gained via email

Language Models and Word Embeddings

Libraries and Tooling

Models and Embeddings

  • IndicBERT v2 (2022-2024) - multilingual BERT for 23 Indic languages.
  • IndicTrans2 (2023-2024) - high-quality MT for 22 Indic languages.
  • OpenHathi (Sarvam AI, 2023) - bilingual Hindi-English LLaMA continuation.
  • Airavata (2024) - instruction-tuned Hindi LLM.
  • Sarvam-1 (2024) - multilingual LM trained from scratch on 10 Indic languages.
  • BharatGPT / Krutrim (2024) - Indic-focused foundation models.

NLP in Indonesian

Back to Top

Libraries and Embeddings

Models

  • IndoBERT (IndoNLU) - pretrained Indonesian LM with the IndoNLU benchmark suite.
  • IndoBERT (IndoLEM) - alternative IndoBERT with the IndoLEM benchmark.
  • NusaCrowd / Cendol (2023-2024) - large-scale community datasets and Cendol instruction-tuned LMs for Indonesian and regional languages.
  • Sailor - open Southeast-Asian LMs covering Indonesian.
  • SEA-LION (2024) - Singapore AI's open Southeast-Asian LM with strong Indonesian.

Datasets

NLP in Korean

Back to Top

Libraries

  • KoNLPy - Python package for Korean natural language processing.
  • Mecab (Korean) - C++ library for Korean NLP.
  • KoalaNLP - Scala library for Korean NLP.
  • KoNLP - R package for Korean NLP.
  • kss - Korean sentence splitter.
  • Kiwi - fast Korean morphological analyzer.
  • Garu - browser-native Korean morphological analyzer running fully client-side via WebAssembly (1MB model, offline, MIT).

Models and Embeddings

Blogs and Tutorials

Datasets

NLP in Persian

Back to Top

Libraries

  • Hazm - Persian NLP toolkit.
  • Parsivar - Persian language processing toolkit.
  • Perke - Persian keyphrase extraction.
  • Perstem - Persian stemmer, morphological analyzer, and partial POS tagger.
  • ParsiAnalyzer - Persian analyzer for Elasticsearch.
  • virastar - Persian text cleaning.

Models

  • ParsBERT - Persian BERT.
  • PersianMind (2023-2024) - Persian instruction-tuned LM.
  • Dorna (Part AI, 2024) - Llama-3-based Persian instruction model.

Datasets

NLP in Polish

Back to Top

  • Polish-NLP - curated list of resources dedicated to Polish NLP: models, tools, and datasets.

NLP in Portuguese

Back to Top

  • Portuguese-nlp - curated list of Portuguese NLP resources and tools.

Models

  • BERTimbau - BERT for Brazilian Portuguese.
  • Sabiá (Maritaca AI, 2023-2024) - Portuguese-focused open LMs.
  • Albertina (PORTULAN, 2023-2024) - encoder-only Portuguese LMs for both PT-PT and PT-BR.

NLP in Spanish

Back to Top

Libraries

  • spanlp - Python library to detect, censor, and clean profanity, hate speech, and bullying in Spanish, with data from 21 Spanish-speaking countries.

Data

Models and Embeddings

NLP in Thai

Back to Top

Libraries

  • PyThaiNLP - Thai NLP in Python.
  • JTCC - character cluster library in Java.
  • CutKum - word segmentation with deep learning in TensorFlow.
  • Thai Language Toolkit - tokenization and POS tagging.
  • SynThai - word segmentation and POS tagging using deep learning.

Models

  • WangchanBERTa - pretrained Thai language model.
  • Typhoon (SCB 10X, 2024) - open Thai LLM family.
  • OpenThaiGPT (2023-2024) - open Thai instruction-tuned models.
  • Sailor - open Southeast-Asian LM family covering Thai.

Data

  • Inter-BEST - text corpus with 5M words and word segmentation.
  • Prime Minister 29 - dataset of speeches by the current Prime Minister of Thailand.

NLP in Ukrainian

Back to Top

NLP in Urdu

Back to Top

Libraries

Datasets

NLP in Uzbek

Back to Top

Datasets

NLP in Vietnamese

Back to Top

Libraries

  • underthesea - Vietnamese NLP toolkit.
  • vn.vitk - Vietnamese text processing toolkit.
  • VnCoreNLP - Vietnamese NLP toolkit.
  • pyvi - Python Vietnamese core NLP toolkit.
  • VieNeu-TTS - on-device Vietnamese text-to-speech with voice cloning.

Models and Embeddings

  • PhoBERT - pretrained LM for Vietnamese.
  • BARTpho - sequence-to-sequence pretrained model for Vietnamese.
  • PhoGPT (VinAI, 2023-2024) - open generative LM for Vietnamese.
  • Vistral (2024) - Mistral-based Vietnamese chat model.
  • Sailor (2024) - open multilingual LM family covering Vietnamese, Thai, Indonesian, and other Southeast Asian languages.

Data

  • Vietnamese Treebank - 10K sentences for the constituency parsing task.
  • BKTreeBank - Vietnamese dependency treebank.
  • UD_Vietnamese - Vietnamese Universal Dependency Treebank.
  • VIVOS - free Vietnamese speech corpus, 15 hours of recorded speech (HCMUS AILab).
  • VNTQcorpus(big).txt - 1.75M news sentences.
  • ViText2SQL - Vietnamese Text-to-SQL semantic parsing dataset (EMNLP-2020 Findings).
  • EVB Corpus - 20M words across 15 bilingual books, 100 parallel English-Vietnamese texts, 250 parallel law texts, 5K news articles, and 2K film subtitles.

Other Languages

  • Russian: pymorphy2 - a good pos-tagger for Russian
  • Asian Languages: Thai, Lao, Chinese, Japanese, and Korean ICU Tokenizer implementation in ElasticSearch
  • Ancient Languages: CLTK: The Classical Language Toolkit is a Python library and collection of texts for doing NLP in ancient languages
  • Hebrew: NLPH_Resources - A collection of papers, corpora and linguistic resources for NLP in Hebrew

Back to Top

See Also

Adjacent curated lists for topics out of scope here:

Citation

If you find this repository useful, please consider citing this list:

bibtex
@misc{awesome-nlp,
  title  = {Awesome NLP},
  author = {Kim, Keon Woo},
  year   = {2018},
  url    = {https://github.com/keon/awesome-nlp},
  note   = {GitHub repository}
}

License

License - CC0

File: README-ZH-TW.md

令人讚嘆的自然語言處理 [](https://github.com/sindresorhus/awesome)

專門用於自然語言處理的精選資源列表

請在提交之前閱讀 貢獻指南 。請隨時創建 拉取請求.

內容

研究摘要和趨勢

教學

返回頂部

閱讀內容

通用機器學習

自然語言處理介紹與指南

部落格與簡報

影片和課程

深度學習與自然語言處理

用於自然語言處理的詞嵌入, 遞歸神經網絡, 長短期記憶神經網絡與卷積神經網路 | 返回頂部

經典自然語言處理

自然語言處理的貝葉斯，統計和語言學方法| | 返回頂部

書籍

函式庫

返回頂部

  • Node.js and Javascript - 用於自然語言的 Node.js 函式庫 | 返回頂部

    • Twitter-text - 使用 JavaScript 實現的 Twitter 文本處理庫。
    • Knwl.js - JS中的自然語言處理器。
    • Retext - 用於分析和操縱自然語言的可​​擴展系統。
    • NLP Compromise - 瀏覽器中的自然語言處理。
    • Natural - 節點的一般自然語言設施。
    • Poplar - 一種基於 Web 的自然語言處理註釋工具（NLP）。

  • Python - 用於自然語言的 Python 函式庫 | 返回頂部

    • TextBlob - 為專研常見的自然語言處理（NLP）任務提供一致的 API。 站在自然語言工具包 (NLTK) 和 模式膀上，並與兩者很好地配合 :+1:

    • spaCy - 使用 Python 與 Cython 產業強度的自然語言處理 :+1:

      • textacy - 在spaCy上構建的更高級別的自然與儼處理。

    • gensim - 用於從純文本進行無監督語義建模的 函式庫 :+1:

    • scattertext - 用於生成語料庫之間語言差異的 d3 可視化的 Python 函式庫。

    • AllenNLP - 一個架構在 PyTorch 上的自然語言處理函式庫，用於開發各種語言任務最先進的深度學習模型。

    • PyTorch-NLP - 自然語言處理研究工具包設計來支援快速建立更好的數據加載器，詞向量加載器，神經網路層表示，常見的自然語言處理指標（如BLEU）原型。

    • Rosetta - 文本處理工具和包裝 (例如： Vowpal Wabbit)

    • PyNLPl - Python 自然語言處理函式庫. 適用於 Python 的通用自然語言處理函式庫。 還包含一些用於解析常見自然語言處理格式的特定模塊, 最常見的是用於 FoLiA，還包括 ARPA 語言模型，Moses 短語表，GIZA ++對齊。

    • jPTDP - 用於聯合詞性（POS）標記和依賴性解析的工具包。jPTDP 提供40多種語言的預訓練模型。

    • BigARTM - 一個用於主題建模的快速函式庫。

    • Snips NLU - 用於意圖解析的產品就緒函式庫。

    • Chazutsu - 用於下載和解析標準自然語言處理研究數據集的函式庫。

    • Word Forms - Word forms 可以準確生成所有可能的英語單詞形式。

    • Multilingual Latent Dirichlet Allocation (LDA) - 一種多語言和可擴展的文檔聚類管道。

    • NLP Architect - 用於探索 NLP 和 NLU 最先進的深度學習拓撲和技術的函式庫。

    • Flair - 一個非常簡單的框架，用於在 PyTorch 上構建最先進的多語言 NLP。包括 BERT，ELMo 和 Flair 嵌入。

    • Kashgari - 簡單的，基於 Keras 的多語言自然語言處理框架，允許您在5分鐘內構建模型，用於命名實體識別（NER），詞性標註（PoS）和文本分類任務。 包括 BERT 和 word2vec 嵌入。

  • C++ - C++ 函式庫 | 返回頂部

    • MIT 資訊提取工具包 - 用於命名實體識別和關係提取的 C，C++ 和Python 工具。
    • CRF++ - 條件隨機場（CRF）的開源專案，用於實現分割/標記順序數據和其他自然語言處理任務。
    • CRFsuite - CRFsuite 實現用於標記順序數據的條件隨機字段（CRF）。
    • BLLIP Parser - BLLIP 自然語言解析器（也稱為 Charniak-Johnson 解析器）
    • colibri-core - C++ 函式庫，命令行工具和 Python 綁定用於快速且內存有效的方式提取和使用基本語言結構，如 n-gram 和 skipgrams。
    • ucto - 適用於各種語言的基於 Unicode 的常規表達式標記生成器。工具和 C++函式庫。支持 FoLiA 格式。
    • libfolia - 用於 FoLiA 格式的 C++ 函式庫。
    • frog - 為荷蘭語開發的基於內存的自然語言處理套件：PoS 標記器，lemmatiser，依賴解析器，NER，淺層解析器，形態分析器。
    • MeTA - MeTA : ModErn Text Analysis 是一個 C++ 數據科學工具包，可以幫助挖掘大文本數據。
    • Mecab (日文)
    • Moses
    • StarSpace - 一個來自 Facebook 的函式庫用於創建單詞級，段級，文檔級和文本分類的嵌入

  • Java - Java 自然語言處理函式庫 | 返回頂部

    • 斯坦福大學 NLP
    • OpenNLP
    • NLP4J
    • Java 中的 Word2vec
    • ReVerb Web-Scale 開放信息提取。
    • OpenRegex 一種高效靈活的基於 token 的正則表達式語言和引擎。
    • CogcompNLP - 在伊利諾伊大學的認知計算組開發的核心函式庫。
    • MALLET - 用於 LanguagE Toolkit 的機器學習 - 用於統計自然語言處理，文檔分類，聚類，主題建模，資訊提取和其他機器學習應用程序的文本包。
    • RDRPOSTagger - 一個穩健的 POS 標記工具包（包括 Java 和 Python）以及40多種語言的預訓練模型。

  • Kotlin - Kotlin 自然語言處理函式庫 | 返回頂部

    • Lingua 適用於 Kotlin 和 Java 的語言檢測函式庫，適用於長文本和短文本。
    • Kotidgy — 一種用 Kotlin 編寫基於索引的文本數據生成器。

  • Scala - Scala 自然語言處理函式庫 | 返回頂部

    • Saul - 用於開發自然語言處理系統的函式庫，包括內置模塊，如 SRL，POS 等。
    • ATR4S - 具有最先進的自動術語識別方法的工具包。
    • tm - 基於正則化多語言 PLSA 的主題建模實現。
    • word2vec-scala - word2vec 模型的 Scala 接口; 包括對詞距離和詞類比等向量的操作。
    • Epic - Epic 是一個用 Scala 編寫的高性能統計解析器，以及用於構建複雜結構化預測模型的框架。

  • R - R 自然語言處理函式庫 | 返回頂部

    • text2vec - R 中的快速矢量化，主題建模，距離和 GloVe 字嵌入。
    • wordVectors - 用於創建和探索 word2vec 和其他單詞嵌入模型的 R 包。
    • RMallet - 與 Java 機器學習工具 MALLET 接口的 R 包。
    • dfr-browser - 創建用於在 Web 瀏覽器中瀏覽文本主題模型的 d3 可視化。
    • dfrtopics - 用於探索文本主題模型的 R 包。
    • sentiment_classifier - 使用Word Sense Disambiguation 和 WordNet Reader 的情感分類。
    • jProcessing - 日本自然語言處理庫，具有日語情感分類。

  • Clojure | 返回頂部

    • Clojure-openNLP - Clojure 中的自然語言處理（opennlp）。
    • Infections-clj - 用於 Clojure 和 ClojureScript 的類似 Rails 的變形函式庫。
    • postagga - 用於解析 Clojure 和 ClojureScript 中的自然語言的函式庫。

  • Ruby | 返回頂部

  • Rust | 返回頂部

    • whatlang — 基於三元組的自然語言識別函式庫。
    • snips-nlu-rs - 用於意圖解析的生產就緒等級函示庫。

服務

自然語言處理作為具有更高級功能的 API，例如 NER，主題標記等 | 返回頂部

註釋工具

  • GATE - 通用架構和文本工程已有15年歷史，免費開源。
  • Anafora 是免費的開源，基於 Web 的原始文本註釋工具。
  • brat - brat 快速註解工具是一個用於協作文本註釋的在線環境。
  • tagtog, 需花 $。
  • prodigy 是一個由主動學習驅動的註釋工具，需花 $。
  • LightTag - 為團隊提供託管和管理的文本註釋工具，需花 $。

技術

文本嵌入

返回頂部

文本嵌入允許深度學習在較小的數據集上有效。這些通常是深入學習的第一步輸入和自然語言處理中最流行的遷移學習方式。嵌入只是簡單的向量，比實際值的字符串表示更為通用的方式。Word嵌入被認為是大多數深度NLP任務的一個很好的起點。

單詞嵌入中最流行的名字是 Google（Mikolov）的 word2vec 和史丹佛的 PenVe（Pennington，Socher 和Manning）。fastText 似乎是一種非常流行的多語言子詞嵌入。

詞嵌入

返回頂部

嵌入 論文 組織 gensim - 培訓支援 部落格
word2vec 官方實作, T.Mikolove et al. 2013. 分散式詞語表達及其組合性。pdf Google 是 :heavy_check_mark: colah 在深度學習，自然語言處理和陳述中的視覺會解釋; gensim 的理解 word2vec
GloVe Jeffrey Pennington, Richard Socher 與 Christopher D. Manning. 2014. GloVe: 全局向量的字詞表示 pdf 史丹佛 否 :negative_squared_cross_mark: acoyler 的 GloVe 早報
fastText 官方實作, T. Mikolov et al. 2017. 使用子詞資訊豐富單詞向量。 pdf Facebook 是 :heavy_check_mark: Fasttext: 深入解析

給初學者的筆記:

基於句子和語言模型的詞嵌入

返回頂部

回答問題與知識提取

返回頂部

資料集

返回頂部

多語言自然語言處理框架

返回頂部

  • UDPipe 是一個可訓練的管道，用於標記，標記，解釋和解析通用樹庫和其他 CoNLL-U 文件。主要用 C++ 編寫，為多語言NLP處理提供快速可靠的解決方案。
  • NLP-Cube : 自然語言處理流水線 - 句子分裂，標記化，詞形還原，詞性標註和依賴性分析。用 Dynet 2.0 用 Python 編寫的新平台。提供獨立（CLI / Python 綁定）和服務器功能（REST API）。

自然語言處理-韓文

返回頂部

函式庫

  • KoNLPy - 用於韓語自然語言處理的Python包。
  • Mecab (Korean) - 韓文的自然語言處理 C++ 函式庫
  • KoalaNLP - 韓國自然語言處理的 Scala 函式庫。
  • KoNLP - 韓文的自然語言處理 R 包。

部落格與教學

資料集

自然語言處理-阿拉伯語

返回頂部

函式庫

  • goarabic - Go包用於阿拉伯語文本處理。
  • jsastem - 用於阿拉伯詞幹的Javascript。
  • PyArabic - 阿拉伯語的 Python 函式庫。

資料集

  • 多域數據集 - 阿拉伯語情感分析的最大可用多域資源。
  • LABR - LArge阿拉伯書籍評論數據集。
  • Arabic 停用詞 - 來自各種資源的阿拉伯語停用詞列表。

自然語言處理-中文

返回頂部

函式庫

  • jieba - 中文詞彙分割實用程序的 Python 包。
  • SnowNLP - 中文自然語言處理 Python 包。
  • FudanNLP - 用於中文文本處理的 Java 函式庫。

自然語言處理-德文

返回頂部

自然語言處理-西班牙語

返回頂部

資料

自然語言處理-印度語

返回頂部

印地語

資料, 文集與樹庫

自然語言處理-泰語

返回頂部

函式庫

  • PyThaiNLP - Python 包中的泰語自然語言處理。
  • JTCC - Java 中的字符集群庫。
  • CutKum - 在 TensorFlow 中使用深度學習進行分詞。
  • 泰語工具包 - 基於 Wirote Aroonmanakun 於2002年撰寫的一篇論文，其中包括數據集。
  • SynThai - 在 Python 中使用深度學習進行分詞和 POS 標記。

資料

自然語言處理-丹麥語

返回頂部

自然語言處理-越南語

返回頂部

函式庫

  • underthesea - 越南自然語言處理工具包。
  • vn.vitk - 越南文本處理工具包。
  • VnCoreNLP - 越南自然語言處理工具包。

資料

自然語言處理-印度尼西亞

返回頂部

資料集

函式庫與嵌入

其他語言

返回頂部

  • 俄語: pymorphy2 - - 俄語好的詞性標記。
  • 亞洲語言: ElasticSearch 中的泰語，老撾語，中文，日語和韓語 ICU Tokenizer 實現。
  • 古代語言: CLTK: 古典語言工具包是一個 Python 函式庫和用於在古代語言中進行自然語言處理的文本集合。
  • Dutch: python-frog - Python 綁定到 Frog，一個荷蘭語的自然語言處理套件。（pos 標記，詞形還原，依賴解析，NER
  • 希伯來語: NLPH_Resources - 希伯來語自然語言處理的論文，語料庫和語言資源的集合。

貢獻

初始策展人和來源的貢獻。

2. Official Technical Reference & Guides (keon/docs)

Mintlify Starter Kit

Click on Use this template to copy the Mintlify starter kit. The starter kit contains examples including

  • Guide pages
  • Navigation
  • Customizations
  • API Reference pages
  • Use of popular components

Development

Install the Mintlify CLI to preview the documentation changes locally. To install, use the following command

text
npm i mintlify

Run the following command at the root of your documentation (where mint.json is)

text
mintlify dev

Publishing Changes

Install our Github App to autopropagate changes from youre repo to your deployment. Changes will be deployed to production automatically after pushing to the default branch. Find the link to install on your dashboard.

Troubleshooting

  • Mintlify dev isn't running - Run mintlify install it'll re-install dependencies.
  • Page loads as a 404 - Make sure you are running in a folder with mint.json

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#awesome Browse awesome catalog → #awesome-list Browse awesome-list catalog → #deep-learning Browse deep-learning catalog → #language Browse language catalog → #machine-learning Browse machine-learning catalog → #natural-language-processing Browse natural-language-processing catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →