## File: README.md # WeNet [](https://opensource.org/licenses/Apache-2.0) [](https://github.com/wenet-e2e/wenet) [](#discussion--communication) [**Roadmap**](https://github.com/wenet-e2e/wenet/issues/1683) | [**Docs**](https://wenet-e2e.github.io/wenet) | [**Papers**](https://wenet-e2e.github.io/wenet/papers.html) | [**Runtime**](https://github.com/wenet-e2e/wenet/tree/main/runtime) | [**Pretrained Models**](docs/pretrained_models.md) | [**HuggingFace**](https://huggingface.co/spaces/wenet/wenet_demo) | [**Ask WeNet Guru**](https://gurubase.io/g/wenet) **We** share **Net** together. ## Highlights * **Production first and production ready**: The core design principle, WeNet provides full stack production solutions for speech recognition. * **Accurate**: WeNet achieves SOTA results on a lot of public speech datasets. * **Light weight**: WeNet is easy to install, easy to use, well designed, and well documented. ## Install ### Install python package ``` sh pip install git+https://github.com/wenet-e2e/wenet.git ``` **Command-line usage** (use `-h` for parameters): ``` sh wenet -m paraformer audio.wav ``` You can set `-m` with `paraformer` or `firered` or `wenetspeech` for chinese, and set it to `whisper-large-v3` or `whisper-large-v3-turbo` for english. **Python programming usage**: ``` python import wenet model = wenet.load_model('paraformer') result = model.transcribe('audio.wav') print(result.text) ``` Please refer [python usage](docs/python_package.md) for more command line and python programming usage. ### Install for training & deployment - Clone the repo ``` sh git clone https://github.com/wenet-e2e/wenet.git ``` - Install Conda: please see https://docs.conda.io/en/latest/miniconda.html - Create Conda env: ``` sh conda create -n wenet python=3.10 conda activate wenet conda install conda-forge::sox ``` - Install CUDA: please follow this [link](https://icefall.readthedocs.io/en/latest/installation/index.html#id1), It's recommended to install CUDA 12.1 - Install torch and torchaudio, It's recomended to use 2.2.2+cu121: ``` sh pip install torch==2.2.2+cu121 torchaudio==2.2.2+cu121 -f https://download.pytorch.org/whl/torch_stable.html ``` **For Ascend NPU users:** - Install CANN: please follow this [link](https://ascend.github.io/docs/sources/ascend/quick_install.html) to install CANN toolkit and kernels. - Install WeNet with torch-npu dependencies: ``` sh pip install -e .[torch-npu] ``` - Related version control table: | Requirement | Minimum | Recommend | | ------------ | ---------------- | ----------- | | CANN | 8.0.RC2.alpha003 | latest | | torch | 2.1.0 | 2.2.0 | | torch-npu | 2.1.0 | 2.2.0 | | torchaudio | 2.1.0 | 2.2.0 | | deepspeed | 0.13.2 | latest | - Install other python packages ``` sh pip install -r requirements.txt pre-commit install # for clean and tidy code ``` - Frequently Asked Questions (FAQs) ``` sh # If you encounter sox compatibility issues RuntimeError: set_buffer_size requires sox extension which is not available. # ubuntu sudo apt-get install sox libsox-dev # centos sudo yum install sox sox-devel # conda env conda install conda-forge::sox ``` **Build for deployment** Optionally, if you want to use x86 runtime or language model(LM), you have to build the runtime as follows. Otherwise, you can just ignore this step. ``` sh # runtime build requires cmake 3.14 or above cd runtime/libtorch mkdir build && cd build && cmake -DGRAPH_TOOLS=ON .. && cmake --build . ``` Please see [doc](https://github.com/wenet-e2e/wenet/tree/main/runtime) for building runtime on more platforms and OS. ## Discussion & Communication You can directly discuss on [Github Issues](https://github.com/wenet-e2e/wenet/issues). For Chinese users, you can also scan the QR code on the left to follow our official account of WeNet. We created a WeChat group for better discussion and quicker response. Please scan the personal QR code on the right, and the guy is responsible for inviting you to the chat group. | || | | ---- | ---- | ---- | ## Acknowledge 1. We borrowed a lot of code from [ESPnet](https://github.com/espnet/espnet) for transformer based modeling. 2. We borrowed a lot of code from [Kaldi](http://kaldi-asr.org/) for WFST based decoding for LM integration. 3. We referred [EESEN](https://github.com/srvk/eesen) for building TLG based graph for LM integration. 4. We referred to [OpenTransformer](https://github.com/ZhengkunTian/OpenTransformer/) for python batch inference of e2e models. ## Citations ``` bibtex @inproceedings{yao2021wenet, title={WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit}, author={Yao, Zhuoyuan and Wu, Di and Wang, Xiong and Zhang, Binbin and Yu, Fan and Yang, Chao and Peng, Zhendong and Chen, Xiaoyu and Xie, Lei and Lei, Xin}, booktitle={Proc. Interspeech}, year={2021}, address={Brno, Czech Republic }, organization={IEEE} } @article{zhang2022wenet, title={WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit}, author={Zhang, Binbin and Wu, Di and Peng, Zhendong and Song, Xingchen and Yao, Zhuoyuan and Lv, Hang and Xie, Lei and Yang, Chao and Pan, Fuping and Niu, Jianwei}, journal={arXiv preprint arXiv:2203.15455}, year={2022} } ``` --- ## File: docs/context.md ## Context Biasing In the practical application of ASR, the recognition effect of commonly used words is better, but for some unique words, the recognition accuracy may be low. Contextual biasing is the problem of injecting prior knowledge into an ASR system during inference, for example a user’s favorite songs, contacts, apps or location. Conventional ASR systems perform contextual biasing by building an n-gram finite state transducer (FST) from a list of biasing phrases, which is composed on-the-fly with the decoder graph during decoding. This helps to bias the recognition result towards the n-grams contained in the contextual FST, and thus improves accuracy in certain scenarios. In WeNet, we compute the biasing scores $P_C(\mathbf y)$, which are interpolated with the base model $P(\mathbf y|\mathbf x)$ using shallow-fusion during beam search, including CTC prefix beam search and CTC WFST beam search. $$ \mathbf y^*=\mathrm{arg\,max\,log}P(\mathbf y|\mathbf x)+\lambda\,\mathrm{log}\,P_C(\mathbf y) $$ where, $\lambda$ is a tunable hyperparameter controlling how much the contextual LM influences the overall model score during beam search. ### Context Graph If we want to improve the score of the word "cat", and the biasing score $\lambda\,\mathrm{log}\,P_C(\mathbf y)$ of each character is 0.25. The context graph can be constructed as follow: In the decoding process, when the corresponding prefix is matched, the corresponding score reward will be obtained. In order to avoid artificially boosting prefixes which match early on but do not match the entire phrase, we add a special failure arc which removes the boosted score. WeNet records only one state for each prefix, to easily determine the boundary of the matched hot word. That is, only one hot word can be matched at the same time, and only after the hot word matching succeeds or fails can other hot words start matching. ``` c++ int ContextGraph::GetNextState(int cur_state, int word_id, float* score, bool* is_start_boundary, bool* is_end_boundary) { int next_state = 0; // Traverse the arcs of current state. for (fst::ArcIterator aiter(*graph_, cur_state); !aiter.Done(); aiter.Next()) { const fst::StdArc& arc = aiter.Value(); if (arc.ilabel == 0) { // Record the score of the backoff arc. It might will be covered. *score = arc.weight.Value(); } else if (arc.ilabel == word_id) { // If they match, record the next state and the score. next_state = arc.nextstate; *score = arc.weight.Value(); // Check whether is the boundary of the hot word. if (cur_state == 0) { *is_start_boundary = true; } if (graph_->Final(arc.nextstate) == fst::StdArc::Weight::One()) { *is_end_boundary = true; } break; } } return next_state; } ``` ### CTC Prefix Beam Search In the process of CTC prefix beam search, each prefix needs to record the hot word matching information. After appending the current output character, if the prefix changes, call the above function `GetNextState` to update the state and score of the hot word. If it is the start or end of a hot word, it is also necessary to record the position, which are used to insert the start tag and end tag in the result, such as: "The \cat\ is in the bag". ### CTC WFST Beam Search WeNet adopts the Lattice Faster Online Decoder from Kaldi for WFST beam search. We have to modify the `lattice-faster-decoder.cc` to support context biasing. WFST beam search decodes in the TLG graph according to the CTC outputs. If we bias the input label of the TLG, we need to compose the context graph with the Token graph. Finally, we decide to bias TLG's output towards the contextual fst. We need to modify the `ProcessEmitting` and `ProcessNonemitting` functions as follow: ```c++ Elem *e_next = FindOrAddToken(arc.nextstate, frame + 1, tot_cost, tok, NULL); // NULL: no change indicator needed // ========== Context code BEGIN =========== bool is_start_boundary = false; bool is_end_boundary = false; float context_score = 0; if (context_graph_) { if (arc.olabel == 0) { e_next->val->context_state = tok->context_state; } else { e_next->val->context_state = context_graph_->GetNextState( tok->context_state, arc.olabel, &context_score, &is_start_boundary, &is_end_boundary); graph_cost -= context_score; } } // ========== Context code END ========== // Add ForwardLink from tok to next_tok (put on head of list // tok->links) tok->links = new ForwardLinkT(e_next->val, arc.ilabel, arc.olabel, graph_cost, ac_cost, is_start_boundary, is_end_boundary, tok->links); tok->links->context_score = context_score; ``` ### Pruning The backoff arc will return the accumulated scores to a single ForwardLink. It leads to the cost of that ForwardLink is too large. We have to remove the cost returned by backoff arc before pruning. ```c++ void LatticeFasterDecoderTpl::PruneForwardLinks( int32 frame_plus_one, bool *extra_costs_changed, bool *links_pruned, BaseFloat delta) { ... BaseFloat link_extra_cost = next_tok->extra_cost + ((tok->tot_cost + link->acoustic_cost + link->graph_cost) - next_tok->tot_cost); // difference in brackets is >= 0 // ========== Context code BEGIN =========== // graph_cost contains the score of hot word // link->context_score < 0 means the hot word of the link is returned from backoff arc if (link->context_score < 0) { link_extra_cost += link->context_score; } // ========== Context code END ========== // link_exta_cost is the difference in score between the best paths // through link source state and through link destination state ``` ### Usage 1. Specify the `--context_path` to a text file. - Each line of the file contains a context. - Each context can be split into words with the symbol_table of the ASR model (It means there is no oov in the context). 2. Specify the `--context_score`, the reward of each word in the context. ```bash cd /home/wenet/runtime/libtorch export GLOG_logtostderr=1 export GLOG_v=2 wav_path=docker_resource/test.wav context_path=docker_resource/context.txt model_dir=docker_resource/model ./build/decoder_main \ --chunk_size -1 \ --wav_path $wav_path \ --model_path $model_dir/final.zip \ --context_path $context_path \ --context_score 3 \ --unit_path $model_dir/units.txt 2>&1 | tee log.txt ``` --- ## File: docs/index.rst .. wenet documentation master file, created by sphinx-quickstart on Thu Dec 3 11:43:53 2020. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. Welcome to wenet's documentation! ================================= wenet is an tansformer-based end-to-end ASR toolkit. .. toctree:: :maxdepth: 2 :caption: Contents: ./python_package.md ./train.rst ./production.rst ./reference.rst Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` --- ## File: docs/jit_in_wenet.md # JIT in WeNet We want that our PyTorch model can be directly exported by torch.jit.script method, which is essential for deploying the model to production. See the following resource for how to deploy PyTorch models in production in details. - [INTRODUCTION TO TORCHSCRIPT](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) - [TORCHSCRIPT LANGUAGE REFERENCE](https://pytorch.org/docs/stable/jit_language_reference.html#language-reference) - [LOADING A TORCHSCRIPT MODEL IN C++](https://pytorch.org/tutorials/advanced/cpp_export.html) - [TorchScript and PyTorch JIT | Deep Dive](https://www.youtube.com/watch?v=2awmrMRf0dA&t=574s) - [Research to Production: PyTorch JIT/TorchScript Updates](https://www.youtube.com/watch?v=St3gdHJzic0) To ensure that, we will try to export the model before training stage. If it fails, we should modify the training code to satisfy the export requirements. ``` python # See in wenet/bin/train.py script_model = torch.jit.script(model) script_model.save(os.path.join(args.model_dir, 'init.zip')) ``` Two principles should be taken into consideration when we contribute our python code to WeNet, especially for the subclass of torch.nn.Module, and for the forward function. 1. Know what is allowed and what is disallowed. - [Torch and Tensor Unsupported Attributes](https://pytorch.org/docs/master/jit_unsupported.html#jit-unsupported) - [Python Language Reference Coverage](https://pytorch.org/docs/master/jit_python_reference.html#python-language-reference) 2. Try to use explicit typing as much as possible. You can try to do type checking forced by typeguard, see https://typeguard.readthedocs.io/en/latest/userguide.html for details. --- ## File: docs/lm.md # LM for WeNet WeNet uses n-gram based statistical language model and the WFST framework to support the custom language model. And LM is only supported in runtime of WeNet. ## Motivation Why n-gram based LM? This may be the first question many people will ask. Now that LM based on RNN and Transformer is in full swing, why does WeNet go backward? The reason is simple, it is for productivity. The n-gram-based language model has mature and complete training tools, any amount of corpus can be trained, the training is very fast, the hotfix is easy, and it has a wide range of mature applications in actual products. Why WFST? It may be the second question many people will ask. Since both industry and research have been working so hard to abandon traditional speech recognition, especially the complex decoding technology. Why does WeNet back? The reason is also very simple, it is for productivity. WFST is a standard and powerful tool in traditional speech recognition. And based on this solution, we have mature and complete bug fix solutions and product solutions, such as that we can use the replace function in WFST for class-based personalization such as contact recognition. Therefore, just like WeNet's design goal "Production first and Production Ready", LM in WeNet also puts productivity as the first priority. So it draws on many very productive tools and solutions accumulated in traditional speech recognition. The difference to traditional speech recognition are: 1. The training in WeNet is pure end to end. 2. As described below, LM is optional in decoding, you can choose whether to use LM according to your needs and application scenarios. ## System Design The whole system is shown in the bellowing picture. There are two ways to generate N-best. 1. Without LM, we use CTC prefix beam search to generate N-best. 2. With LM, we use CTC WFST search to generate N-best and CTC WFST search is the traditional WFST based decoder. There are two main parts of the CTC WFST based search. The first is building the decoding graph, which is to compose the model unit T, the lexicon L and the language model G into one unified graph TLG. And in which: 1. T is the model unit in E2E training. Typically it's char in Chinese, char or BPE in English. 2. L is the lexicon, the lexicon is very simple. What we need to do is just split a word into its modeling unit sequence. For example, the word "我们" is split into two chars "我 们", and the word "APPLE" is split into five letters "A P P L E". We can see there is no phonemes and there is no need to design pronunciation on purpose. 3. G is the language model, namely compiling the n-gram to standard WFST representation. The second is the decoder, which is the same as the traditional decoder, which uses the standard Viterbi beam search algorithm in decoding. ## Implementation WeNet draws on the decoder and related tools in Kaldi to support LM and WFST based decoding. For ease of using and keeping independence, we directly migrated the code related to decoding in Kaldi to [this directory](https://github.com/wenet-e2e/wenet/tree/main/runtime/core/kaldi) in WeNet runtime. And modify and organize according to the following principles: 1. To minimize changes, the migrated code remains the same directory structure as the original. 2. We use GLOG to replace the log system in Kaldi. 3. We modify the code format to meet the lint requirements of the code style in WeNet. The core code is https://github.com/wenet-e2e/wenet/blob/main/runtime/core/decoder/ctc_wfst_beam_search.cc, which wraps the LatticeFasterDecoder in Kaldi. And we use blank frame skipping to speed up decoding. In addition, WeNet also migrated related tools for building the decoding graph, such as arpa2fst, fstdeterminizestar, fsttablecompose, fstminimizeencoded, and other tools. So all the tools related to LM are built-in tools and can be used out of the box. ## Results We get consistent gain (3%~10%) on different datasets, including aishell, aishell2, and librispeech, please go to the corresponding example dataset for the details. ## How to use? Here is an example from aishell, which shows how to prepare the dictionary, how to train the LM, how to build the graph, and how to decode with the runtime. ``` sh # 7.1 Prepare dict unit_file=$dict mkdir -p data/local/dict cp $unit_file data/local/dict/units.txt tools/fst/prepare_dict.py $unit_file ${data}/resource_aishell/lexicon.txt \ data/local/dict/lexicon.txt # 7.2 Train lm lm=data/local/lm mkdir -p $lm tools/filter_scp.pl data/train/text \ $data/data_aishell/transcript/aishell_transcript_v0.8.txt > $lm/text local/aishell_train_lms.sh # 7.3 Build decoding TLG tools/fst/compile_lexicon_token_fst.sh \ data/local/dict data/local/tmp data/local/lang tools/fst/make_tlg.sh data/local/lm data/local/lang data/lang_test || exit 1; # 7.4 Decoding with runtime ./tools/decode.sh --nj 16 \ --beam 15.0 --lattice_beam 7.5 --max_active 7000 \ --blank_skip_thresh 0.98 --ctc_weight 0.5 --rescoring_weight 1.0 \ --fst_path data/lang_test/TLG.fst \ --dict_path data/lang_test/words.txt \ data/test/wav.scp data/test/text $dir/final.zip \ data/lang_test/units.txt $dir/lm_with_runtime ``` --- ## File: docs/papers.md ## Papers * [WeNet: Production Oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://arxiv.org/pdf/2102.01547.pdf), accepted by InterSpeech 2021. * [WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit](https://arxiv.org/pdf/2203.15455.pdf), accepted by InterSpeech 2022. --- ## File: docs/pretrained_models.md # Pretrained Models in WeNet ## Model Types We provide two types of pretrained model in WeNet to facilitate users with different requirements. 1. **Checkpoint Model**, with suffix **.pt**, the model trained and saved as checkpoint by WeNet python code, you can reproduce our published result with it, or you can use it as checkpoint to continue. 2. **Runtime Model**, with suffix **.zip**, you can directly use `runtime model` in our [x86](https://github.com/wenet-e2e/wenet/tree/main/runtime/libtorch) or [android](https://github.com/wenet-e2e/wenet/tree/main/runtime/android) runtime, the `runtime model` is export by Pytorch JIT on the `checkpoint model`. And the runtime models has been quantized to reduce the model size and network traffic. ## Model License The pretrained model in WeNet follows the license of it's corresponding dataset. For example, the pretrained model on LibriSpeech follows `CC BY 4.0`, since it is used as license of the LibriSpeech dataset, see http://openslr.org/12/. ## Model List Here is a list of the pretrained models on different datasets. The model structure, model size, and download link are given. | Datasets | Languages | Checkpoint Model | Runtime Model | Contributor | |--- |--- |--- |--- |--- | | [aishell](../examples/aishell/s0/README.md) | CN | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=aishell_u2pp_conformer_exp.tar.gz) | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=aishell_u2pp_conformer_libtorch.tar.gz) | [](https://www.chumenwenwen.com) | | [aishell2](../examples/aishell2/s0/README.md) | CN | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=aishell2_u2pp_conformer_exp.tar.gz) | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=aishell2_u2pp_conformer_libtorch.tar.gz) | [](https://www.chumenwenwen.com) | | [gigaspeech](../examples/gigaspeech/s0/README.md) | EN | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=gigaspeech_u2pp_conformer_exp.tar.gz) | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=gigaspeech_u2pp_conformer_libtorch.tar.gz) | [](https://www.chumenwenwen.com) | | [librispeech](../examples/librispeech/s0/README.md) | EN | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=librispeech_u2pp_conformer_exp.tar.gz) | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=librispeech_u2pp_conformer_libtorch.tar.gz) | [](https://www.chumenwenwen.com) | | [multi_cn](../examples/multi_cn/s0/README.md) | CN | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=multi_cn_unified_conformer_exp.tar.gz) | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=multi_cn_unified_conformer_libtorch.tar.gz) | [](https://www.jd.com) | | [wenetspeech](../examples/wenetspeech/s0/README.md) | CN | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=wenetspeech_u2pp_conformer_exp.tar.gz) | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=wenetspeech_u2pp_conformer_libtorch.tar.gz) | [](https://horizon.ai) | | paraformer | CN&EN | [Model](https://wenet.org.cn/downloads?models=wenet&version=paraformer.tar.gz) | NA | Alibaba | | firered | CN&EN | [Model](https://wenet.org.cn/downloads?models=wenet&version=firered.tar.gz) | NA | FireRed | | whisper-large-v3 | multi-lingual | [Model](https://wenet.org.cn/downloads?models=wenet&version=whisper-large-v3.tar.gz) | NA | OpenAI | | whisper-large-v3-turbo | multi-lingual | [Model](https://wenet.org.cn/downloads?models=wenet&version=whisper-large-v3-turbo.tar.gz) | NA | OpenAI | --- ## File: docs/production.rst Production Runtime ================== .. toctree:: :maxdepth: 1 :caption: Contents: ./lm.md ./context.md ./runtime.md ./jit_in_wenet.md --- ## File: docs/python_package.md # Python Package ## Install ``` sh pip install git+https://github.com/wenet-e2e/wenet.git ``` ## Development Install ``` sh git clone https://github.com/wenet-e2e/wenet.git cd wenet pip install -e . ``` ## Command line Usage ``` sh wenet --language chinese audio.wav ``` You can specify the following parameters. * `-l` or `--language`: chinese/english are supported now. * `-m` or `--model_dir`: your own model dir * `-g` or `--gpu`: the device id of gpu, default value -1 represents for cpu. * `-t` or `--show_tokens_info`: show the token level information such as timestamp, confidence, etc. * `--align`: force align the input audio and transcript * `--label`: the input label to align * `--paraformer`: use the best Chinese model * `--device`: specify the backend accelerator (cuda/npu/cpu) ## Python Programming Usage ``` python import wenet model = wenet.load_model('chinese') # or model = wenet.load_model(model_dir='xxx') result = model.transcribe('audio.wav') print(result['text']) ```