wenet Documentation

Technical Documentation: wenet-e2e/wenet

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

WeNet

Roadmap
| Docs
| Papers
| Runtime
| Pretrained Models
| HuggingFace
| Ask WeNet Guru

We share Net together.

Highlights Copy Chapter

• Production first and production ready: The core design principle, WeNet provides full stack production solutions for speech recognition.
• Accurate: WeNet achieves SOTA results on a lot of public speech datasets.
• Light weight: WeNet is easy to install, easy to use, well designed, and well documented.

Install Copy Chapter

Install python package Copy Section

pip install git+https://github.com/wenet-e2e/wenet.git

This pulls in torch>=1.13.0 with no upper bound, so pip may install a torch
build newer than your installed GPU driver supports. If you plan to use a GPU,
verify it after install:

python -c "import torch; print(torch.cuda.is_available())"

If this prints False on a machine with a GPU, torch silently falls back to
CPU with no error. Reinstall a torch build matching your driver's CUDA
version, e.g.:

pip install torch==2.4.0+cu121 torchaudio==2.4.0+cu121 --index-url https://download.pytorch.org/whl/cu121 --force-reinstall

Command-line usage (use -h for parameters):

wenet -m paraformer audio.wav

You can set -m with paraformer or firered or wenetspeech for chinese,
and set it to whisper-large-v3 or whisper-large-v3-turbo for english.

Note: this runs on CPU by default (--device defaults to cpu), even if
a GPU is available. Pass --device cuda explicitly to use the GPU (Python API:
wenet.load_model(model_name, device='cuda')). See
python usage for GPU/Turing (e.g.
Tesla T4) specific notes on dtype and attention-backend options.

Python programming usage:

import wenet

model = wenet.load_model('paraformer')
result = model.transcribe('audio.wav')
print(result.text)

Please refer python usage for more command line and python programming usage.

Install for training & deployment Copy Section

• Clone the repo

git clone https://github.com/wenet-e2e/wenet.git

• Install Conda: please see https://docs.conda.io/en/latest/miniconda.html
• Create Conda env:

conda create -n wenet python=3.10
conda activate wenet
conda install conda-forge::sox

• Install CUDA: please follow this link, It's recommended to install CUDA 12.1
• Install torch and torchaudio, It's recomended to use 2.2.2+cu121:

pip install torch==2.2.2+cu121 torchaudio==2.2.2+cu121 -f https://download.pytorch.org/whl/torch_stable.html

For Ascend NPU users:

• Install CANN: please follow this link to install CANN toolkit and kernels.

• Install WeNet with torch-npu dependencies:

pip install -e .[torch-npu]

• Related version control table:

• Install other python packages

pip install -r requirements.txt
pre-commit install  # for clean and tidy code

• Frequently Asked Questions (FAQs)

# If you encounter sox compatibility issues
RuntimeError: set_buffer_size requires sox extension which is not available.
# ubuntu
sudo apt-get install sox libsox-dev
# centos
sudo yum install sox sox-devel
# conda env
conda install  conda-forge::sox

Build for deployment

Optionally, if you want to use x86 runtime or language model(LM),
you have to build the runtime as follows. Otherwise, you can just ignore this step.

# runtime build requires cmake 3.14 or above
cd runtime/libtorch
mkdir build && cd build && cmake -DGRAPH_TOOLS=ON .. && cmake --build .

Please see doc for building
runtime on more platforms and OS.

Discussion & Communication Copy Chapter

You can directly discuss on Github Issues.

For Chinese users, you can also scan the QR code on the left to follow our official account of WeNet.
We created a WeChat group for better discussion and quicker response.
Please scan the personal QR code on the right, and the guy is responsible for inviting you to the chat group.

Acknowledge Copy Chapter

1. We borrowed a lot of code from ESPnet for transformer based modeling.
2. We borrowed a lot of code from Kaldi for WFST based decoding for LM integration.
3. We referred EESEN for building TLG based graph for LM integration.
4. We referred to OpenTransformer for python batch inference of e2e models.

Citations Copy Chapter

@inproceedings{yao2021wenet,
title={WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit},
author={Yao, Zhuoyuan and Wu, Di and Wang, Xiong and Zhang, Binbin and Yu, Fan and Yang, Chao and Peng, Zhendong and Chen, Xiaoyu and Xie, Lei and Lei, Xin},
  booktitle={Proc. Interspeech},
  year={2021},
  address={Brno, Czech Republic },
  organization={IEEE}
}

@article{zhang2022wenet,
  title={WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit},
  author={Zhang, Binbin and Wu, Di and Peng, Zhendong and Song, Xingchen and Yao, Zhuoyuan and Lv, Hang and Xie, Lei and Yang, Chao and Pan, Fuping and Niu, Jianwei},
  journal={arXiv preprint arXiv:2203.15455},
  year={2022}
}

Overview (`runtime/README.md`) Copy Section

Runtime on WeNet

This is the runtime of WeNet.

We are going to support the following platforms:

1. Various deep learning inference engines, such as LibTorch, ONNX, OpenVINO, TVM, and so on.
2. Various OS, such as android, iOS, Harmony, and so on.
3. Various AI chips, such as GPU, Horzion BPU, and so on.
4. Various hardware platforms, such as Raspberry Pi.
5. Various language binding, such as python and go.

Feel free to volunteer yourself if you are interested in trying out some items(they do not have to be on the list).

Introduction Copy Chapter

Here is a brief summary of all platforms and OSs. please note the corresponding working OS and inference engine.

Overview (`wenet/README.md`) Copy Section

Module Introduction

Here is a brief introduction of each module(directory).

• bin: training and recognition binaries
• dataset: IO design
• utils: common utils
• transformer: the core of WeNet, in which the standard transformer/conformer is implemented. It contains the common blocks(backbone) of speech transformers.
  • transformer/attention.py: Standard multi head attention
  • transformer/embedding.py: Standard position encoding
  • transformer/positionwise_feed_forward.py: Standard feed forward in transformer
  • transformer/convolution.py: ConvolutionModule in Conformer model
  • transformer/subsampling.py: Subsampling implementation for speech task
• transducer: transducer implementation
• squeezeformer: squeezeformer implementation, please refer paper
• efficient_conformer: efficient conformer implementation, please refer paper
• paraformer: paraformer implementation, please refer paper
  • paraformer/cif.py: Continuous Integrate-and-Fire implemented, please refer paper
• branchformer: branchformer implementation, please refer paper
• whisper: whisper implementation, please refer paper
• ssl: Self-supervised speech model implementation. e.g. wav2vec2, bestrq, w2vbert.
• ctl_model: Enhancing the Unified Streaming and Non-streaming Model with with Contrastive Learning implementation paper

transducer, squeezeformer, efficient_conformer, branchformer and cif are all based on transformer,
they resue a lot of the common blocks of tranformer.

If you want to contribute your own x-former, please reuse the current code as much as possible.

Overview (`runtime/android/README.md`) Copy Section

WeNet On-device ASR Android Demo

This Android demo shows we can run on-device streaming ASR with WeNet. You can download our prebuilt APK or build your APK from source code.

Prebuilt APK Copy Chapter

• Chinese ASR Demo APK, with model trained on AIShell data
• English ASR Demo APK, with model trained on GigaSpeech data

Build your APK from source code Copy Chapter

1) Build model Copy Section

You can use our pretrained model (click the following link to download):

中文(WenetSpeech)
| English(GigaSpeech)

Or you can train your own model using WeNet training pipeline on your data.

2) Build APK Copy Section

When your model is ready, put final.zip and units.txt into Android assets (app/src/main/assets) folder,
then just build and run the APK. Here is a gif demo, which shows how our on-device streaming e2e ASR runs with low latency.
Please note the wifi and data has been disabled in the demo so there is no network connection ^_^.

Compute the RTF Copy Chapter

Step 1, connect your Android phone, and use adb push command to push your model, wav scp, and waves to the sdcard.

Step 2, build the binary and the APK with Android Studio directly, or with the commands as follows:

cd runtime/android
./gradlew build

Step 3, push your binary and the dynamic library to /data/local/tmp as follows:

adb push app/.cxx/cmake/release/arm64-v8a/decoder_main /data/local/tmp
adb push app/build/pytorch_android-1.10.0.aar/jni/arm64-v8a/* /data/local/tmp

Step 4, change to the directory /data/local/tmp of your phone, and export the library path by:

adb shell
cd /data/local/tmp
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.

Step 5, execute the same command as the x86 demo to run the binary to decode and compute the RTF.

Overview (`runtime/gpu/README.md`) Copy Section

Introduction Copy Chapter

The below example shows how to deploy WeNet offline and online ASR models on GPUs.

Instructions Copy Chapter

• Step 1. Convert your model/pretrained model to onnx models. For example:

conda activate wenet
pip install onnxruntime-gpu onnxmltools
cd wenet/examples/aishell2 && . ./path.sh
model_dir=<absolute path to>/20211025_conformer_exp
onnx_model_dir=<absolute path>
mkdir $onnx_model_dir
python3 wenet/bin/export_onnx_gpu.py --config=$model_dir/train.yaml --checkpoint=$model_dir/final.pt --cmvn_file=$model_dir/global_cmvn --ctc_weight=0.5 --output_onnx_dir=$onnx_model_dir --fp16
cp $model_dir/words.txt $model_dir/train.yaml $onnx_model_dir/

If you want to export streaming model (u2/u2++) for streaming inference (inference by chunks) instead of offline inference (inference by audio segments/utterance), you should add --streaming option:

python3 wenet/bin/export_onnx_gpu.py --config=$model_dir/train.yaml --checkpoint=$model_dir/final.pt --cmvn_file=$model_dir/global_cmvn  --ctc_weight=0.1 --reverse_weight=0.4 --output_onnx_dir=$onnx_model_dir --fp16 --streaming

Congratulations! You've successully exported the onnx models and now you are able to deploy them. Please also ensure you have set up NGC account before the next step!

• Step 2. Build server docker and start server on one gpu:

docker build . -f Dockerfile/Dockerfile.server -t wenet_server:latest --network host
# offline model
docker run --gpus '"device=0"' -it -v $PWD/model_repo:/ws/model_repo -v $onnx_model_dir:/ws/onnx_model -p 8000:8000 -p 8001:8001 -p 8002:8002 --shm-size=1g --ulimit memlock=-1  wenet_server:latest /workspace/scripts/convert_start_server.sh
# streaming model
docker run --gpus '"device=0"' -it -v $PWD/model_repo_stateful:/ws/model_repo -v $onnx_model_dir:/ws/onnx_model -p 8000:8000 -p 8001:8001 -p 8002:8002 --shm-size=1g --ulimit memlock=-1  wenet_server:latest /workspace/scripts/convert_start_server.sh

Whenever there's something wrong starting the server, you may check the config.pbtxt files in every model in model_repo or model_repo_stateful to find if the settings are right.

• Step 3. Start client

docker build . -f Dockerfile/Dockerfile.client -t wenet_client:latest --network host

AUDIO_DATA=<path to your wav data>
docker run -ti --net host --name wenet_client -v $PWD/client:/ws/client -v $AUDIO_DATA:/ws/test_data wenet_client:latest
# In docker
# offline model test
cd /ws/client
# test one wav file
python3 client.py --audio_file=/ws/test_data/mid.wav --url=localhost:8001

# test a list of wav files & cer
python3 client.py --wavscp=/ws/dataset/test/wav.scp --data_dir=/ws/dataset/test/ --trans=/ws/dataset/test/text

Similarly, if your model is exported with --streaming, you should add this option when calling your streaming model.
For example,

python3 client.py --wavscp=/ws/test_data/data_aishell2/test/wav.scp --data_dir=/ws/test_data/data_aishell2/test/ --trans=/ws/test_data/data_aishell2/test/trans.txt --model_name=streaming_wenet --streaming

Precision Impact Copy Chapter

Some of you may be worried about whether fp16 will affect the final accuracy. We did several experiments and we may find the accuracy is acceptable.

Perf Test Copy Chapter

We use the below command to do our testing and we run the below command several times to warm up:

cd /ws/client
# generate the test data, input to our feature extractor
# offline model
python3 generate_perf_input.py --audio_file=input.wav
# offline_input.json generated
perf_analyzer -m attention_rescoring -b 1 -p 20000 --concurrency-range 100:200:50 -i gRPC --input-data=offline_input.json  -u localhost:8001

# streaming input
python3 generate_perf_input.py --audio_file=input.wav --streaming
# online_input.json generated
perf_analyzer -u "localhost:8001" -i gRPC --streaming --input-data=online_input.json -m streaming_wenet -b 1 --concurrency-range 100:200:50

Where:

• input.wav the input test audio, we tested 5 seconds, 8 seconds, 10 seconds audio;
• -m option indicates the name of the served model;
• -p option is the mearsurement window, which indicates in what time duration to collect the metrics;
• -v option turns on the verbose model;
• -i option is for choosing the networking protocol, you can choose HTTP or gRPC here;
• -u option sets the url of the service in the form of :, but notice that port 8000 corresponds to HTTP protocol while port 8001 corresponds to gRPC protocol;
• -b option indicates the batch size of the input requests used fo testing; since we simulate individual users sending requests, we set batch size here to 1;
• --input-data option points to the path of the json file containing the real input data
• --concurrency-range option is an important one, it indicates the concurrency of the requests which defines the pressure we will give to the server.
• You can also set -f option to set the path of testing result file;
• You can also set --max-threads option to set the number of threads used to send test request, it should be set to the number of CPU cores in your test machine.

Tested ENV Copy Section

• NVIDIA DRIVER: 470.57.02
• GPU: V100 & T4 & A30
• CUDA: 11.4
• Triton Inference Server: 22.03

Offline Model Perf Copy Section

Here are the wenetspeech conformer model and aishell2 u2++ perf on T4.

• Aishell2, FP16, onnx, U2++ Conformer

• Wenetspeech, FP16, onnx, Conformer

• The offline model_repo's overview:

Online Model Perf Copy Section

• Aishell2, FP16, onnx, U2++ Conformer
  Our chunksize is 16 * 4 * 10 = 640 ms, so we should care about the perf of latency less than 640ms so that it can be a realtime application.

Improve Accuracy Copy Section

Add Lanaguage Model

• Add language model: set --lm_path in the convert_start_server.sh. Notice the path of your language model is the path in docker.
• You may refer to wenet/bin/recognize_onnx.py to run inference locally. If you want to add language model locally, you may refer to here

Add Hotwords

• Add hotwords: If you use offline model, modify hotwords_path in model_repo/scoring/config_template.pbtxt (None->/ws/model_repo/scoring/hotwords.yaml). Otherwise, in streaming model, modify hotwords_path (None->/ws/model_repo/wenet/hotwords.yaml) in model_repo_stateful/wenet/config_template.pbtxt.
• Then follow the steps in Instructions to start the hotwords server.

We use client.py to test the effect of hotwords on AISHELL-1 Test dataset and AISHELL-1 hostwords sub-testsets.

AISHELL-1 Test dataset

• Test set contains 7176 utterances (5 hours) from 20 speakers.

AISHELL-1 hostwords sub-testsets

• Test set contains 235 utterances with 187 entities words.

Decoding result

Tested ENV

• CPU：40 Core, Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
• GPU：NVIDIA GeForce RTX 2080 Ti
• Acoustic model: 20210601_u2++_conformer_exp (AISHELL-1)

Refer to more results: https://huggingface.co/58AILab/wenet_u2pp_aishell1_with_hotwords/tree/main/results

Hotwords usage

Please refer to the following steps how to use hotwordsboosting.

• Step 1. Initialize HotWordsScorer

# if you don't want to use hotwords. set hotwords_scorer=None(default),
# vocab_list is Chinese characters.
hot_words = {'再接': 10, '再厉': -10, '好好学习': 100}
hotwords_scorer = HotWordsScorer(hot_words, vocab_list, is_character_based=True)

If you set is_character_based is True (default mode), the first step is to combine Chinese characters into words, if words in hotwords dictionary then add hotwords score. If you set is_character_based is False, all words in the fixed window will be enumerated.

• Step 2. Add hotwords_scorer when decoding

result = ctc_beam_search_decoder_batch(batch_chunk_log_prob_seq,
                                        batch_chunk_log_probs_idx,
                                        batch_root_trie,
                                        batch_start,
                                        beam_size, num_processes,
                                        blank_id, space_id,
                                        cutoff_prob, scorer, hotwords_scorer)

Please refer to swig/test/test_zh.py for how to decode with hotwordsboosting.

Hotwords evaluation

Prepare decode result file with_hotwords_ali.log and label file aishell1_text_hotwords

# utt \t text
BAC009S0764W0179    国务院发展研究中心市场经济研究所副所长邓郁松认为
BAC009S0764W0205    本报记者王颖春国家发改委近日发出通知

Run the script for evaluation, with multiple result files separated by ';'.

cd runtime/gpu/scripts
python compute_hotwords_f1.py \
    --label="aishell1_text_hotwords" \
    --preds="with_hotwords_ali.log;data/without_hotwords_ali.log" \
    --hotword="../model_repo/scoring/hotwords.yaml"

Output ner file:

BAC009S0764W0179    国务院发展研究中心市场经济研究所副所长邓郁松认为    邓郁松
BAC009S0764W0205    本报记者王颖春国家发改委近日发出通知  王颖春

Dynamic Left Chunks

For online model, training with dynamic left chunk option on will help further improve the model accuracy.
Let's take a look at the below table.

WenetSpeech
chunksize = 16

With dynamic left = False in training and chunk size=16 when inferencing, we take all the left chunks:

Warning Copy Section

• We only tested mandarin models in wenetspeech and aishell2 and haven't tested models built with English corpus.
• The bpe models currently are not supported.
• You need to add a VAD module to split your long audio into segments.
• Please pad your audio in order to leverage triton inference dynamic batching feature. We suggest you to pad your audio to nearest length such as 2s, 5s, 10s seconds.
• We're still working on improving this solution. If you find any issue, please raise an issue.
• For streaming pipeline, we haven't implemented endpoint feature, which means you have to cut your audio manually.
• Please use client.py as a reference for building an intermediate server between the true clients (e.g., web application) and triton inference server. You may add the padding strategy (for offline model), buffering, vad, endpoint or other preprocessing (for streaming model) in this intermediate server.

More info can be found on Triton documents. For example, if you want to deploy your application on scale, you may refer to triton k8s deploy.

Reference Copy Chapter

• Triton inference server & Client

Acknowledge Copy Chapter

This part originates from NVIDIA CISI project. We also have TTS and NLP solutions deployed on triton inference server.
If you are interested, please contact us.

Thanks to @RiverLiu @Jiawei and @day9011 for great effort in testing.

Overview (`runtime/horizonbpu/README.md`) Copy Section

WeNet & Horizon BPU (Cross Compile)

• Step 1. Setup environment (install horizon packages and cross compile tools) in the PC. (~10min)

# Conda env (This conda env is only used for converting bpu models, not for training torch models,
#   It's OK to install cpu-version pytorch)
conda create -n horizonbpu python=3.8
conda activate horizonbpu
git clone https://github.com/wenet-e2e/wenet.git
cd wenet/runtime/horizonbpu
pip install -r ../../requirements.txt -i https://mirrors.aliyun.com/pypi/simple
pip install torch==1.13.0 torchaudio==0.13.0 torchvision==0.14.0 onnx onnxruntime -i https://mirrors.aliyun.com/pypi/simple

# Horizon packages
wget https://gitee.com/xcsong-thu/toolchain_pkg/releases/download/resource/wheels.tar.gz
tar -xzf wheels.tar.gz
pip install wheels/* -i https://mirrors.aliyun.com/pypi/simple

# Cross compile tools
sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

• Step 2. Build decoder_main. It requires cmake 3.14 or above. and Send the binary/libraries to Horizon X3PI. (~20min)

# Assume current dir is `wenet/runtime/horizonbpu`
cmake -B build -DBPU=ON -DONNX=OFF -DTORCH=OFF -DWEBSOCKET=OFF -DGRPC=OFF -DCMAKE_TOOLCHAIN_FILE=toolchains/aarch64-linux-gnu.toolchain.cmake
cmake --build build

# Send binary and libraries
export BPUIP=xxx.xxx.xxx.xxx
export DEMO_PATH_ON_BOARD=/path/to/demo
scp build/bin/decoder_main sunrise@$BPUIP:$DEMO_PATH_ON_BOARD
scp fc_base/easy_dnn-src/dnn/*j3*/*/*/lib/libdnn.so sunrise@$BPUIP:$DEMO_PATH_ON_BOARD
scp fc_base/easy_dnn-src/easy_dnn/*j3*/*/*/lib/libeasy_dnn.so sunrise@$BPUIP:$DEMO_PATH_ON_BOARD
scp fc_base/easy_dnn-src/hlog/*j3*/*/*/lib/libhlog.so sunrise@$BPUIP:$DEMO_PATH_ON_BOARD

• Step 3. Export model to ONNX and convert ONNX to Horizon .bin and Send the model/dict/test_wav to Horizon X3PI. (~40min)

/* Detailed source-code truncated for AI context efficiency. */

• Step 4. Testing on X3PI, the RTF(real time factor) is shown in Horizon X3PI's console. (~1min)

cd /path/to/demo
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH \
  GLOG_logtostderr=1 GLOG_v=2 \
  ./decoder_main \
      --chunk_size 8 \
      --num_left_chunks 16 \
      --rescoring_weight 0.0 \
      --wav_path ./test_wav.wav \
      --bpu_model_dir ./ \
      --unit_path ./units.txt 2>&1 | tee log.txt

Overview (`runtime/ios/README.md`) Copy Section

WeNet On-device ASR iOS Demo

Build application from source code Copy Chapter

1) Generate cmake project, install LibTorch pod and build static libraries Copy Section

cd runtime/ios/build
cmake .. -G Xcode -DTORCH=ON -DONNX=OFF -DIOS=ON -DGRAPH_TOOLS=OFF -DBUILD_TESTING=OFF -DCMAKE_TOOLCHAIN_FILE=../toolchains/ios.toolchain.cmake -DPLATFORM=OS64 -DENABLE_BITCODE=FALSE
pod install

# Build debug version
cmake --build . --config Debug

# Build release version
cmake --build . --config Release

2) Build and run iOS application Copy Section

You can use our pretrained model (click the following link to download):

AISHELL-1
| AISHELL-2
| GigaSpeech
| LibriSpeech
| Multi-CN

Or you can train your own model using WeNet training pipeline on your data.

When your model is ready, put final.zip and units.txt into model (WenetDemo/WenetDemo/model) folder.

Open WenetDemo.xcodeproj with Xcode, build and run on iOS device.

Overview (`runtime/ipex/README.md`) Copy Section

WeNet Server (x86) ASR Demo With Intel® Extension for PyTorch\* Optimization Copy Chapter

Intel® Extension for PyTorch* (IPEX) extends PyTorch* with up-to-date optimization features for extra performance boost on Intel hardware. The optimizations take advantage of AVX-512, Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel X^e Matrix Extensions (XMX) AI engines on Intel discrete GPUs.

In the following we are introducing how to accelerate WeNet model inference performance on Intel® CPU machines with the adoption of Intel® Extension for PyTorch*. The adoption mainly includes the export of pretrained models with IPEX optimization, as well as the buildup of WeNet runtime executables with IPEX C++ SDK. The buildup can be processed from local source code, or directly build and run a docker container in which the runtime binaries are ready.

We provide a shell script for WeNet runtime environment checking. Please run the script

sh env_checking.sh

and see if the environment is suitable for leveraging resources to get optimal performance for WeNet runtime models.

Run in Docker Build Copy Section

We recommend using the docker environment to build the c++ binary to avoid
system or environment problems.

• Step 1. Build your docker image.

cd docker
docker build --no-cache -t wenet-ipex:latest .

• Step 2. The pretrained PyTorch model checkpoint file (.pt) needs to be converted to TorchScript runtime model (.zip) via export_ipex.py. The detail usage for this model exporting process are introduced in Run with Local Build section below. After model exporting, we can put all the resources, like runtime model, test wavs into a docker resource dir.

mkdir -p docker_resource
cp -r <your_model_dir> docker_resource/model
cp <your_test_wav> docker_resource/test.wav

• Step 3. Start docker container.

docker run --rm -v $PWD/docker_resource:/home/wenet/runtime/ipex/docker_resource -it wenet-ipex:latest bash

• Step 4. Test in docker container

cd /home/wenet/runtime/ipex
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=docker_resource/test.wav
model_dir=docker_resource/model
./build/bin/decoder_main \
    --chunk_size -1 \
    --wav_path $wav_path \
    --model_path $model_dir/<runtime_model_filename> \
    --unit_path $model_dir/units.txt 2>&1 | tee log.txt

Run with Local Build Copy Section

• Step 1. Environment Setup.

WeNet code cloning and default dependencies installation

git clone https://github.com/wenet-e2e/wenet
cd wenet
pip install -r requirements.txt

Upgrading of PyTorch and TorchAudio, followed by the installation of IPEX

pip install torch==2.3.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cpu --force-reinstall
pip install intel_extension_for_pytorch==2.3.0

Installation of related tools: Intel® OpenMP and TCMalloc

pip install intel-openmp

and

yum -y install gperftools or apt-get install -y google-perftools

based on the package manager of your system.

• Step 2. Download or prepare your pretrained model.

• Step 3. Export the pretrained model with IPEX optimization.

For exporting FP32 runtime model

source examples/aishell/s0/path.sh
export OMP_NUM_THREADS=1
python wenet/bin/export_ipex.py \
    --config <model_config_yaml_filename> \
    --checkpoint <model_ckpt_filename> \
    --output_file <runtime_model_output_filename>

If you have an Intel® 4th Generation Xeon (Sapphire Rapids) server, you can export a BF16 runtime model and get better performance by virtue of AMX instructions

source examples/aishell/s0/path.sh
export OMP_NUM_THREADS=1
python wenet/bin/export_ipex.py \
    --config <model_config_yaml_filename> \
    --checkpoint <model_ckpt_filename> \
    --output_file <runtime_model_output_filename> \
    --dtype bf16

And for exporting int8 quantized runtime model

source examples/aishell/s0/path.sh
export OMP_NUM_THREADS=1
python wenet/bin/export_ipex.py \
    --config <model_config_yaml_filename> \
    --checkpoint <model_ckpt_filename> \
    --output_quant_file <runtime_model_output_filename>

• Step 4. Build runtime executable binaries. The build requires cmake 3.14 or above. For building, please first change to wenet/runtime/ipex as your build directory, then type:

mkdir build && cd build && cmake .. && cmake --build .

• Step 5. Testing, the RTF(real time factor) would be shown in the console.

cd ..
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
model_dir=your_model_dir
ipexrun --no-python \
    ./build/bin/decoder_main \
        --chunk_size -1 \
        --wav_path $wav_path \
        --model_path $model_dir/<runtime_model_filename> \
        --unit_path $model_dir/units.txt 2>&1 | tee log.txt

NOTE: Please refer IPEX Launch Script Usage Guide for usage of advanced features.

For advanced usage of WeNet, such as building Web/RPC/HTTP services, please refer LibTorch Tutorial. The difference is that the executables should be invoked via IPEX launch script ipexrun.

Overview (`runtime/kunlun/README.md`) Copy Section

在昆仑芯片上运行Wenet

介绍 Copy Chapter

下面的示例展示了如何在XPU上部署WeNet离线或在线的ASR模型。XPU是一种由昆仑芯100%自主研发的通用人工智能计算核心架构。

准备XPU运行环境 Copy Chapter

在开始之前，请确认您获得以下必须的环境。

XRE(XPU Runtime Environment):昆仑芯片的基础运行环境，包括芯片驱动程序、runtime api库、固件FW工具等功能模块。
XDNN(XPU Deep Neural Network Library):加速深度神经网络的昆仑芯片库，提供应用程序中使用的高性能DNN功能库。

如果您需要任何帮助，或是想要进一步了解昆仑芯片，请通过官方网址联系我们：
https://www.kunlunxin.com.cn/

操作步骤 Copy Chapter

• 第一步：构建，需要cmake 3.14及以上版本

export CXX=${your_g++_path}
export CC=${your_gcc_path}
export XPU_API_PATH=${your_api_path}

# -r : release version; -d : debug version
bash ./compile.sh -r

• 第二步：测试，测试结果将在控制台输出

## set KUNLUN XPU visible device
export XPU_VISIBLE_DEVICES=0
export XPUSIM_DEVICE_MODEL=KUNLUN2
## set logging level
export GLOG_logtostderr=1
export GLOG_v=3
## set speech wav and model/weight path
wav_path=${your_test_wav_path}
xpu_model_dir=${your_xpu_weight_dir}
units=${your_units.txt}
## executive command
./build/bin/decoder_main \
    --chunk_size -1 \
    --wav_path ${wav_path} \
    --xpu_model_dir ${xpu_model_di} \
    --unit_path ${units}   \
    --device_id 0           \
    --nbest  3  2>&1 | tee log.txt

单条语音执行结果如下所示:

/* Detailed source-code truncated for AI context efficiency. */

Overview (`runtime/libtorch/README.md`) Copy Section

WeNet Server (x86) ASR Demo

中文版:x86 平台上使用 WeNet 进行语音识别

Run with Prebuilt Docker Copy Chapter

• Step 1. Download the pretrained models or prepare your trained model.

• Step 2. Start docker websocket server. Here is a demo.

model_dir=$PWD/aishell_u2pp_conformer_libtorch  # absolute path
docker run --rm -it -p 10086:10086 -v $model_dir:/home/wenet/model wenetorg/wenet-mini:latest bash /home/run.sh

• Step 3. Test with web browser. Open runtime/libtorch/web/templates/index.html in the browser directly, input your WebSocket URL, it will request some permissions, and start to record to test, as the following graph shows.

Run in Docker Build Copy Chapter

We recommend using the docker environment to build the c++ binary to avoid
system and environment problems.

• Step 1. Build your docker image.

cd docker
docker build --no-cache -t wenet:latest .

• Step 2. Put all the resources, like model, test wavs into a docker resource dir.

mkdir -p docker_resource
cp -r <your_model_dir> docker_resource/model
cp <your_test_wav> docker_resource/test.wav

• Step 3. Start docker container.

docker run --rm -v $PWD/docker_resource:/home/wenet/runtime/libtorch/docker_resource -it wenet bash

• Step 4. Testing in docker container

cd /home/wenet/runtime/libtorch
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=docker_resource/test.wav
model_dir=docker_resource/model
./build/bin/decoder_main \
    --chunk_size -1 \
    --wav_path $wav_path \
    --model_path $model_dir/final.zip \
    --unit_path $model_dir/units.txt 2>&1 | tee log.txt

Or you can do the WebSocket server/client testing as described in the WebSocket section.

Run with Local Build Copy Chapter

• Step 1. Download or prepare your pretrained model.

• Step 2. Build. The build requires cmake 3.14 or above. For building, please first change to wenet/runtime/libtorch as your build directory, then type:

mkdir build && cd build && cmake .. && cmake --build .

For building with GPU, you should turn on GPU:

mkdir build && cd build && cmake -DGPU=ON .. && cmake --build .

• Step 3. Testing, the RTF(real time factor) is shown in the console.

export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
model_dir=your_model_dir
./build/bin/decoder_main \
    --chunk_size -1 \
    --wav_path $wav_path \
    --model_path $model_dir/final.zip \
    --unit_path $model_dir/units.txt 2>&1 | tee log.txt

Advanced Usage Copy Chapter

WebSocket Copy Section

• Step 1. Download or prepare your pretrained model.
• Step 2. Build as in Run with Local Build
• Step 3. Start WebSocket server.

export GLOG_logtostderr=1
export GLOG_v=2
model_dir=your_model_dir
./build/bin/websocket_server_main \
    --port 10086 \
    --chunk_size 16 \
    --model_path $model_dir/final.zip \
    --unit_path $model_dir/units.txt 2>&1 | tee server.log

• Step 4. Start WebSocket client.

export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
./build/bin/websocket_client_main \
    --hostname 127.0.0.1 --port 10086 \
    --wav_path $wav_path 2>&1 | tee client.log

You can also start WebSocket client by web browser as described before.

Here is a demo for command line based websocket server/client interaction.

gRPC Copy Section

Why grpc? You may find your answer in https://grpc.io/.
Please follow the following steps to try gRPC.

• Step 1. Download or prepare your pretrained model.
• Step 2. Build

mkdir build && cd build && cmake -DGRPC=ON .. && cmake --build .

• Step 3. Start gRPC server

export GLOG_logtostderr=1
export GLOG_v=2
model_dir=your_model_dir
./build/bin/grpc_server_main \
    --port 10086 \
    --workers 4 \
    --chunk_size 16 \
    --model_path $model_dir/final.zip \
    --unit_path $model_dir/units.txt 2>&1 | tee server.log

• Step 4. Start gRPC client.

export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
./build/bin/grpc_client_main \
    --hostname 127.0.0.1 --port 10086 \
    --wav_path $wav_path 2>&1 | tee client.log

http Copy Section

• Step 1. Download or prepare your pretrained model.
• Step 2. Build

mkdir build && cd build && cmake -DHTTP=ON .. && cmake --build .

• Step 3. Start http server

simply replace grpc_server_main with http_server_main of Step 3 in gRPC

• Step 4. Start http client.

simply replace grpc_client_main with http_client_main of Step 4 in gRPC

Overview (`runtime/onnxruntime/README.md`) Copy Section

ONNX backend on WeNet

• Step 1. Export your experiment model to ONNX by https://github.com/wenet-e2e/wenet/blob/main/wenet/bin/export_onnx_cpu.py

exp=exp  # Change it to your experiment dir
onnx_dir=onnx
python -m wenet.bin.export_onnx_cpu \
  --config $exp/train.yaml \
  --checkpoint $exp/final.pt \
  --chunk_size 16 \
  --output_dir $onnx_dir \
  --num_decoding_left_chunks -1

# When it finishes, you can find `encoder.onnx`, `ctc.onnx`, and `decoder.onnx` in the $onnx_dir respectively.

• Step 2. Build. The build requires cmake 3.14 or above.

mkdir build && cd build
cmake -DONNX=ON -DTORCH=OFF -DWEBSOCKET=OFF -DGRPC=OFF ..
cmake --build .

• Step 3. Testing, the RTF(real time factor) is shown in the console.

export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
onnx_dir=your_model_dir
units=units.txt  # Change it to your model units path
# Make sure that the `chunk_size` and `num_left_chunks` variables are set to the corresponding values used when exporting the ONNX models.
./build/bin/decoder_main \
    --chunk_size 16 \
    --num_left_chunks -1 \
    --wav_path $wav_path \
    --onnx_dir $onnx_dir \
    --unit_path $units 2>&1 | tee log.txt

Overview (`runtime/openvino/README.md`) Copy Section

OpenVINO™ backend on WeNet

• Step 1. Get ONNX model.

Use wenet.bin.export_onnx_cpu to export ONNX model. Follow this guide

Please note, 3 ONNX models(encoder.onnx, ctc.onnx, and decoder.onnx) will be generated.

• Step 2. Convert ONNX model to OpenVINO™ IR(Intermediate Representation).

Install OpenVINO™ runtime with Model Optimizer (MO) command tool for model conversion:

mo --input_model onnx/encoder.onnx --input chunk[1,-1,80],att_cache[12,4,-1,128],cnn_cache[12,1,256,7] --output_dir openvino 
mo --input_model onnx/ctc.onnx --input_shape [1,-1,256] --output_dir openvino 
mo --input_model onnx/decoder.onnx --input hyps[-1,-1],hyps_lens[-1],encoder_out[1,-1,256]  --output_dir openvino

By using MO, user can specify shape of multiple inputs by `--input` option, or just directly provide `--input_shape` for single input model. To enable the dynamic shape support for inference, user can use `-1` or provide a range of input shape value, like `1..80`.Please refer the [usage guide of MO](https://docs.openvino.ai/latest/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html).


# When it finishes, you can find IR files(.xml and .bin) for encoder, ctc and decoder.

• Step 3. Build WeNet with OpenVINO™.

Please refer system requirement to check if the hardware platform available by OpenVINO™.
The OpenVINO Linux installation package will not provide Intel TBB runtime library and pugixml library, please check if your system already installed:

# For Ubuntu and Debian:
sudo apt install libtbb-dev libpugixml-dev
# For CentOS:
sudo yum install tbb-devel pugixml-devel

To build WeNet with OpenVINO:

mkdir build && cd build
cmake -DOPENVINO=ON -DTORCH=OFF -DWEBSOCKET=OFF -DGRPC=OFF ..
make --jobs=$(nproc --all)

(Optional) Some users may cannot easily download OpenVINO™ binary package from server due to firewall or proxy issue. If you failed to download by CMake script, you can download OpenVINO™ package by your selves and put the package to below path:

${wenet_path}/runtime/openvino/fc_base/openvino-subbuild/openvino-populate-prefix/src/l_openvino_toolkit_ubuntu20_2022.3.0.9052.9752fafe8eb_x86_64.tgz

• Step 4. Testing.

export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
openvino_dir=your_model_dir
units=units.txt  # Change it to your model units path
./build/bin/decoder_main \
    --chunk_size 16 \
    --wav_path $wav_path \
    --openvino_dir $openvino_dir \
    --unit_path $units 2>&1 | tee log.txt

Overview (`runtime/raspberrypi/README.md`) Copy Section

WeNet & Raspberry PI (Cross Compile)

• Step 1. Install cross compile tools in the PC.

sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

Or download, and install the binaries from: https://releases.linaro.org/components/toolchain/binaries/latest-7

• Step 2. Export your experiment model to ONNX by https://github.com/wenet-e2e/wenet/blob/main/wenet/bin/export_onnx_cpu.py

exp=exp  # Change it to your experiment dir
onnx_dir=onnx
python -m wenet.bin.export_onnx_cpu \
  --config $exp/train.yaml \
  --checkpoint $exp/final.pt \
  --chunk_size 16 \
  --output_dir $onnx_dir \
  --num_decoding_left_chunks -1

# When it finishes, you can find `encoder.onnx(.quant)`, `ctc.onnx(.quant)`, and `decoder.onnx(.quant)` in the $onnx_dir respectively.
# We use the quantified to speed up the inference, so rename it without the suffix `.quant`

• Step 3. Build. The build requires cmake 3.14 or above. and Send the binary and libraries to Raspberry PI.

cmake -B build -DONNX=ON -DTORCH=OFF -DWEBSOCKET=OFF -DGRPC=OFF -DCMAKE_TOOLCHAIN_FILE=toolchains/aarch64-linux-gnu.toolchain.cmake
cmake --build build
scp build/bin/decoder_main [email protected]:/path/to/wenet
scp fc_base/onnxruntime-src/lib/libonnxruntime.so* [email protected]:/path/to/wenet

• Step 4. Testing, the RTF(real time factor) is shown in Raspberry PI's console.

cd /path/to/wenet
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
onnx_dir=your_model_dir
units=units.txt  # Change it to your model units path
./build/bin/decoder_main \
    --chunk_size 16 \
    --wav_path $wav_path \
    --onnx_dir $onnx_dir \
    --unit_path $units 2>&1 | tee log.txt

Overview (`runtime/web/README.md`) Copy Section

WeNet Web Demo Copy Chapter

• How to install? pip install -r requirements.txt
• How to start?
  • Non-streaming: python app.py

Overview (`examples/aishell/NST/README.md`) Copy Section

Recipe to run Noisy Student Training with LM filter in WeNet

Noisy Student Training (NST) has recently demonstrated extremely strong performance in Automatic Speech Recognition (ASR).

Here, we provide a recipe to run NST with LM filter strategy using AISHELL-1 as supervised data and WenetSpeech as unsupervised data from this paper, where hypotheses with and without Language Model are generated and CER differences between them are utilized as a filter threshold to improve the ASR performances of non-target domain datas.

Table of Contents Copy Chapter

• Guideline
  • Data preparation
  • Initial supervised teacher
  • Noisy student interations
• Performance Record
  • Supervised baseline and standard NST
  • Supervised AISHELL-1 and unsupervised 1khr WenetSpeech
  • Supervised AISHELL-2 and unsupervised 4khr WenetSpeech
• Citations

Guideline Copy Chapter

First, you have to prepare supervised and unsupervised data for NST. Then in stage 1 of run.sh, you will train an initial supervised teacher and generate pseudo labels for unsupervised data.
After that, you can run the noisy student training iteratively in stage 2. The whole pipeline is illustrated in the following picture.

Data preparation Copy Section

To run this recipe, you should follow the steps from WeNet examples to prepare AISHELL1 and WenetSpeech data.
We extract 1khr data from WenetSpeech and data should be prepared and stored in the following format:

data/
├── train/
├──── data_aishell.list
├──── wenet_1khr.list
├──── wav_dir/
├──── utter_time.json (optional)
├── dev/
└── test/

• Files *.list contain paths for all the data shards for training.
• A Json file containing the audio length should be prepared as utter_time.json if you want to apply the speaking rate filter.
• A wav_dir contains all the audio data (id.wav) and labels (id.txt which is optional) for unsupervised data.

Initial supervised teacher Copy Section

To train an initial supervised teacher model, run the following command:

bash run.sh --stage 1 --stop-stage 1

Full arguments are listed below, you can check run.sh and run_nst.sh for more information about steps in each stage and their arguments. We used num_split = 60 and generate shards with different cpu for the experiments in our paper which saved us lots of inference time and data shards generation time.

bash run.sh --stage 1 --stop-stage 1 --dir exp/conformer_test_fully_supervised --supervised_data_list data_aishell.list --enable_nst 0 --num_split 1 --unsupervised_data_list wenet_1khr.list --dir_split wenet_split_60_test/ --job_num 0 --hypo_name hypothesis_nst0.txt --label 1 --wav_dir data/train/wenet_1k_untar/ --cer_hypo_dir wenet_cer_hypo --cer_label_dir wenet_cer_label --label_file label.txt --cer_hypo_threshold 10 --speak_rate_threshold 0 --utter_time_file utter_time.json --untar_dir data/train/wenet_1khr_untar/ --tar_dir data/train/wenet_1khr_tar/ --out_data_list data/train/wenet_1khr.list

• dir contains the training parameters.
• data_list contains paths for the training data list.
• supervised_data_list contains paths for supervised data shards.
• unsupervised_data_listcontains paths for unsupervised data shards which is used for inference.
• dir_split is the directory stores split unsupervised data for parallel computing.
• out_data_list is the pseudo label data list file path.
• enable_nst indicates whether we train with pseudo label and split data, for initial teacher we set it to 0.
• This recipe uses the default num_split=1 while we strongly recommend use larger number to decrease the inference and shards generation time.

HINTS If num_split is set to N larger than 1, you need to modify the script in step 4-8 in run_nst.sh to submit N tasks into your own clusters (such as slurm,ngc etc..).
We strongly recommend to do so since inference and pseudo-data generation is time-consuming.

Noisy student interations Copy Section

After finishing the initial fully supervised baseline, we now have the mixed list contains both supervised and pseudo data which is wenet_1khr_nst0.list.
We will use it as the data_list in the training step and the data_list for next NST iteration will be generated.

Here is an example command:

bash run.sh --stage 2 --stop-stage 2 --iter_num 2

Here we add extra argument iter_num for number of NST iterations. Intermediate files are named with iter_num as a suffix.
Please check the run.sh and run_nst.sh scripts for more information about each stage and their arguments.

Performance Record Copy Chapter

Supervised baseline and standard NST Copy Section

• Non-streaming conformer model with attention rescoring decoder.
• Without filter strategy, first iteration
• Feature info: using FBANK feature, dither, cmvn, online speed perturb
• Training info: lr 0.002, batch size 32, 8 gpu, acc_grad 4, 240 epochs, dither 0.1
• Decoding info: ctc_weight 0.3, average_num 30

Supervised AISHELL-1 and unsupervised 1khr WenetSpeech Copy Section

• Non-streaming conformer model with attention rescoring decoder.
• Feature info: using FBANK feature
• Training info: lr=0.002, batch_size=32, 8 GPUs, acc_grad=4, 120 epochs, dither=0.1
• Decoding info: ctc_weight=0.3, average_num=30, pseudo_ratio=0.75

Supervised AISHELL-2 and unsupervised 4khr WenetSpeech Copy Section

• Non-streaming conformer model with attention rescoring decoder.
• Feature info: using FBANK feature
• Training info: lr=0.002, batch_size=32, 8 GPUs, acc_grad=4, 120 epochs, dither=0.1
• Decoding info: ctc_weight=0.3, average_num=30, pseudo_ratio=0.75

Citations Copy Chapter

@article{chen2022NST,
  title={Improving Noisy Student Training on Non-target Domain Data for Automatic Speech Recognition},
  author={Chen, Yu and Wen, Ding and Lai, Junjie},
  journal={arXiv preprint arXiv:2203.15455},
  year={2022}
}

---

### Overview (`examples/aishell/paraformer/README.md`)

# Preliminary
1. Run below command to convert funasr-style  ckpt to wenet-style ckpt:
```sh
output_dir=exp/paraformer/large
mkdir -p ${output_dir}
. ./path.sh && python wenet/paraformer/convert_paraformer_to_wenet_config_and_ckpt.py \
  --output_dir ${output_dir}
# init ctc and embed(used in sampler)
python local/modify_ckpt.py \
  --add_list "{\"ctc.ctc_lo.weight\": \"embed.weight\"}" \
  --input_ckpt exp/paraformer/large/wenet_paraformer.pt \
  --output_ckpt exp/paraformer/large/wenet_paraformer.init-ctc.init-embed.pt

Performance Record

Paraformer (original) Result Copy Chapter

Paraformer (full-parameter tuning) Result Copy Chapter

• Training info: torch_ddp fp32, batch size 28, ctc_weight: 0.3, acc_grad 1, 8 * 3090 gpu, 60 epochs (about 8h)
• Decoding info: ctc_weight 0.3, average_num 5
• Git hash: TBD

Paraformer-dynamic training (full-parameter tuning) Result Copy Chapter

• Training info: torch_ddp fp32, batch size 28, ctc_weight: 0.3, acc_grad 1, 8 * 3090 gpu, 60 epochs (about 8h)
• Decoding info: ctc_weight 0.3, average_num 5
• Git hash: TBD

Overview (`examples/aishell/rnnt/README.md`) Copy Section

Performance Record

Conformer Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, online speed perturb
• Training info: lr 0.001, batch size 8, 8 gpu, acc_grad 1, 100 epochs, dither 0.1
• Training weight info: transducer_weight 0.75, ctc_weight 0.1, attention_weight 0.15, average_num 10
• Predictor type: lstm

• after 165 epochs and avg 30

Conformer Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, online speed perturb
• Training info: lr 0.001, batch size 20, 8 gpu, acc_grad 1, 140 epochs, dither 0.1
• Training weight info: transducer_weight 0.4, ctc_weight 0.2, attention_weight 0.4, average_num 10
• Predictor type: lstm
• Model link: https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/aishell/20220728_conformer_rnnt_exp.tar.gz

U2++ Conformer Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, oneline speed perturb
• Training info: lr 0.001, batch size 4, 32 gpu, acc_grad 1, 360 epochs
• Training weight info: transducer_weight 0.75, ctc_weight 0.1, reverse_weight 0.15 average_num 30
• Predictor type: lstm

Pretrain Copy Chapter

• Pretrain model: https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/aishell/20210601_u2%2B%2B_conformer_exp.tar.gz
• Feature info: using fbank feature, dither, cmvn, oneline speed perturb
• Training info: lr 0.001, batch size 8, 8 gpu, acc_grad 1, 140 epochs
• Training weight info: transducer_weight 0.4, ctc_weight 0.2 , attention_weight 0.4, reverse_weight 0.3 average_num 30
• Predictor type: lstm

Training loss ablation study Copy Chapter

note:

• If rnnt is checked, greedy means rnnt greedy search; so is beam

• if rnnt is checked, rescoring means rnnt beam & attention rescoring

• if only 'ctc & att' is checked, greedy means ctc gredy search; so is beam

• if only 'ctc & att' (AED) is checked, rescoring means ctc beam & attention rescoring

• what if rnnt model do search of wenet's style, comming soon

Overview (`examples/aishell/s0/README.md`) Copy Section

Performance Record

Conformer Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, online speed perturb
• Training info: lr 0.002, batch size 18, 4 gpu, acc_grad 4, 240 epochs, dither 0.1
• Decoding info: ctc_weight 0.5, average_num 20
• Git hash: 919f07c4887ac500168ba84b39b535fd8e58918a

U2++ Conformer Result Copy Chapter

• Feature info: using fbank feature, dither=1.0, cmvn, oneline speed perturb
• Training info: lr 0.001, batch size 16, 8 gpu, acc_grad 1, 360 epochs
• Decoding info: ctc_weight 0.3, reverse_weight 0.5 average_num 30, lm_scale 0.7, decoder_scale 0.1, r_decoder_scale 0.7
• Git hash: 5a1342312668e7a5abb83aed1e53256819cebf95

U2++ lite Conformer Result (uio shard) Copy Chapter

• Feature info: using fbank feature, dither=1.0, cmvn, oneline speed perturb
• Training info: lr 0.001, batch size 16, 8 gpu, acc_grad 1, load a well trained model and continue training 80 epochs with u2++ lite config
• Decoding info: ctc_weight 0.3, reverse_weight 0.5 average_num 30
• Git hash: 73185808fa1463b0163a922dc722513b7baabe9e

Unified Conformer Result Copy Chapter

• Feature info: using fbank feature, dither=0, cmvn, oneline speed perturb
• Training info: lr 0.001, batch size 16, 8 gpu, acc_grad 1, 180 epochs, dither 0.0
• Decoding info: ctc_weight 0.5, average_num 20
• Git hash: 919f07c4887ac500168ba84b39b535fd8e58918a

U2++ Transformer Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, online speed perturb.
• Training info: lr 0.001, batch size 26, 8 gpu, acc_grad 1, 360 epochs, dither 0.1
• Decoding info: ctc_weight 0.2, reverse_weight 0.5, average_num 30
• Git hash: 65270043fc8c2476d1ab95e7c39f730017a670e0

Transformer Result Copy Chapter

• Feature info: using fbank feature, dither, with cmvn, online speed perturb.
• Training info: lr 0.002, batch size 26, 4 gpu, acc_grad 4, 240 epochs, dither 0.1
• Decoding info: ctc_weight 0.5, average_num 20
• Git hash: 919f07c4887ac500168ba84b39b535fd8e58918a

Unified Transformer Result Copy Chapter

• Feature info: using fbank feature, dither=0, with cmvn, online speed perturb.
• Training info: lr 0.002, batch size 16, 4 gpu, acc_grad 1, 240 epochs, dither 0.1
• Decoding info: ctc_weight 0.5, average_num 20
• Git hash: 919f07c4887ac500168ba84b39b535fd8e58918a

AMP Training Transformer Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, online speed perturb
• Training info: lr 0.002, batch size, 4 gpus, acc_grad 4, 240 epochs, dither 0.1, warm up steps 25000
• Decoding info: ctc_weight 0.5, average_num 20
• Git hash: 1bb4e5a269c535340fae5b0739482fa47733d2c1

Muilti-machines Training Conformer Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, online speed perturb
• Training info: lr 0.004, batch size 16, 2 machines, 8*2=16 gpus, acc_grad 4, 240 epochs, dither 0.1, warm up steps 10000
• Decoding info: ctc_weight 0.5, average_num 20
• Git hash: f6b1409023440da1998d31abbcc3826dd40aaf35

Conformer with/without Position Encoding Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, online speed perturb
• Training info: lr 0.002, batch size 16, 8 gpu, acc_grad 4, 240 epochs, dither 0.1
• Decoding info: ctc_weight 0.5, average_num 20

Efficient Conformer v1 Result Copy Chapter

• Feature info:
  • using fbank feature, cmvn, speed perturb, dither
• Training info:
  • train_u2++_efficonformer_v1.yaml
  • 8 gpu, batch size 16, acc_grad 1, 200 epochs
  • lr 0.001, warmup_steps 25000
• Model info:
  • Model Params: 48,488,347
  • Downsample rate: 1/4 (conv2d) * 1/2 (efficonformer block)
  • encoder_dim 256, output_size 256, head 8, linear_units 2048
  • num_blocks 12, cnn_module_kernel 15, group_size 3
• Decoding info:
  • ctc_weight 0.5, reverse_weight 0.3, average_num 20
• Model Download: wenet_efficient_conformer_aishell_v1

Efficient Conformer v2 Result Copy Chapter

• Feature info:
  • using fbank feature, cmvn, speed perturb, dither
• Training info:
  • train_u2++_efficonformer_v2.yaml
  • 8 gpu, batch size 16, acc_grad 1, 200 epochs
  • lr 0.001, warmup_steps 25000
• Model info:
  • Model Params: 49,354,651
  • Downsample rate: 1/2 (conv2d2) * 1/4 (efficonformer block)
  • encoder_dim 256, output_size 256, head 8, linear_units 2048
  • num_blocks 12, cnn_module_kernel 15, group_size 3
• Decoding info:
  • ctc_weight 0.5, reverse_weight 0.3, average_num 20
• Model Download: wenet_efficient_conformer_aishell_v2

U2++ Branchformer Result Copy Chapter

• Feature info: using fbank feature, dither=1.0, cmvn, oneline speed perturb
• • Model info:
    • Model Params: 48,384,667
    • Num Encoder Layer: 24
    • CNN Kernel Size: 63
    • Merge Method: concat
• Training info: lr 0.001, weight_decay: 0.000001, batch size 16, 3 gpu, acc_grad 1, 360 epochs
• Decoding info: ctc_weight 0.3, reverse_weight 0.5 average_num 30, lm_scale 0.7, decoder_scale 0.1, r_decoder_scale 0.7
• Git hash: 5a1342312668e7a5abb83aed1e53256819cebf95

E-Branchformer Result Copy Chapter

• Feature info: using fbank feature, dither=1.0, cmvn, online speed perturb
• • Model info:
    • Model Params: 47,570,132
    • Num Encoder Layer: 17
    • CNN Kernel Size: 31
• Training info: lr 0.001, weight_decay: 0.000001, batch size 16, 4 gpu, acc_grad 1, 240 epochs
• Decoding info: ctc_weight 0.3, average_num 30
• Git hash: 89962d1dcae18dd3a281782a40e74dd2721ae8fe

Overview (`examples/aishell/sensevoice/README.md`) Copy Section

Performance Record

SenseVoice-small (original) Result Copy Chapter

SenseVoice-small (full-parameter tuning) Result Copy Chapter

• Training info: torch_ddp fp32, ctc_weight: 1.0, acc_grad 1, 8 * 3090 gpu, 49 epochs
• Decoding info: ctc_weight 1.0, average_num 5
• Git hash: TBD

Overview (`examples/aishell/whisper/README.md`) Copy Section

Preliminary

1. Download whisper ckpt from this link

2. We assume you have run stage0~stage3 using aishell/s0/run.sh and here we are simply creating a symbolic link

ln -s ../s0/data .

3. Run below command to convert openai-style ckpt to wenet-style ckpt:

mkdir -p exp
mkdir -p exp/whisper
mkdir -p exp/whisper/large-v3
. ./path.sh && python wenet/whisper/convert_whisper_to_wenet_config_and_ckpt.py \
  --whisper_ckpt downloaded-large-v3.pt \
  --output_dir exp/whisper/large-v3
# remove conv1d2
python local/modify_ckpt.py \
  --remove_list "encoder.embed.conv" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.remove-subsample.pt
# init ctc
python local/modify_ckpt.py \
  --add_list "{\"ctc.ctc_lo.weight\": \"decoder.embed.0.weight\"}" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.init-ctc.pt
# remove conv1d2 and init ctc
python local/modify_ckpt.py \
  --remove_list "encoder.embed.conv" \
  --add_list "{\"ctc.ctc_lo.weight\": \"decoder.embed.0.weight\"}" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.remove-subsample.init-ctc.pt

Performance Record

Whisper-largev2 (original) Result Copy Chapter

Whisper-largev3 (conv1d2, full-parameter tuning) Result Copy Chapter

• Feature info: using log_mel_spectrogram feature, no cmvn
• Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 1, 8 * 3090 gpu, 30 epochs (about 16 hours), conf/finetune_whisper_largev3_onlyattn.yaml
• Decoding info: ctc_weight 0.0, average_num 2
• Git hash: TBD

• Feature info: using log_mel_spectrogram feature, no cmvn, no speed perturb
• Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 1, 8 * 3090 gpu, 80 epochs (about 43 hours), conf/finetune_whisper_largev3.yaml
• Decoding info: ctc_weight 0.3, average_num 5
• Git hash: TBD

Whisper-largev3 (conv2d4, full-parameter tuning) Result Copy Chapter

• Feature info: using log_mel_spectrogram feature, no cmvn
• Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 1, 8 * 3090 gpu, 30 epochs (about 14 hours), conf/finetune_whisper_largev3_conv2d4_onlyattn.yaml
• Decoding info: ctc_weight 0.0, average_num 2
• Git hash: TBD

• Feature info: using log_mel_spectrogram feature, no cmvn, no speed perturb
• Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 1, 8 * 3090 gpu, 80 epochs (about 36 hours), conf/finetune_whisper_largev3_conv2d4.yaml
• Decoding info: ctc_weight 0.3, average_num 5
• Git hash: TBD

Frequently Asked Questions

• Q: Why are there so many insertion errors in the decoding results of CTC?
• A: Because Chinese characters are composed of multiple bytes, in Whisper's tokenizer, one Chinese character might be represented by multiple tokens (for example, 3 tokens). During the CTC decoding process, it's possible that only two of these tokens are decoded. This not only causes garbled text (see https://github.com/wenet-e2e/wenet/issues/2308 ) but also leads to insertion errors.

Overview (`examples/aishell2/rnnt/README.md`) Copy Section

Performance Record

U2++ Conformer Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, oneline speed perturb
• Training info: lr 0.001, dynamic batch with max_frames_in_batch 15000, 4 gpu, acc_grad 1, 130 epochs
• Training weight info: transducer_weight 0.75, ctc_weight 0.1, reverse_weight 0.30, average_num 30
• Predictor type: lstm

Overview (`examples/aishell2/s0/README.md`) Copy Section

Performance Record

U2++ Conformer Result Copy Chapter

• Feature info: using fbank feature, with cmvn, no speed perturb, dither
• Training info: lr 0.001, batch size 32, 8 gpus, acc_grad 1, 240 epochs, dither 1.0
• Decoding info: ctc_weight 0.1, reverse_weight 0.4, average_num 30
• Git hash: 5a1342312668e7a5abb83aed1e53256819cebf95

U2++ Transformer Result Copy Chapter

• Feature info: using fbank feature, with cmvn, no speed perturb
• Training info: lr 0.002, batch size 22, 8 gpus, acc_grad 1, 240 epochs, dither 0.0
• Decoding info: ctc_weight 0.1, reverse_weight 0.5, average_num 30
• Git hash: 5a1342312668e7a5abb83aed1e53256819cebf95

Unified Conformer Result Copy Chapter

• Feature info: using fbank feature, with cmvn, no speed perturb.
• Training info: lr 0.002, batch size 16, 8 gpus, acc_grad 1, 120 epochs, dither 1.0
• Decoding info: ctc_weight 0.5, average_num 20
• Git hash: 14d38085a8d966cf9e9577ffafc51d578dce954f

Unified Transformer Result Copy Chapter

• Feature info: using fbank feature, with cmvn, no speed perturb.
• Training info: lr 0.002, batch size 22, 8 gpus, acc_grad 1, 180 epochs, dither 0.0
• Decoding info: ctc_weight 0.5, average_num 30
• Git hash: 14d38085a8d966cf9e9577ffafc51d578dce954f

Overview (`examples/aishell4/s0/README.md`) Copy Section

Performance Record

Conformer Result Copy Chapter

• Feature info: using fbank feature, cmvn, without speed perturb (not supported segments yet)
• Training info: lr 0.001, max_frames_in_batch 15000, 8 gpu, acc_grad 4, 100 epochs
• Decoding info: ctc_weight 0.5, average_num 30

U2pp **Conformer Result Copy Chapter

• Feature info: using fbank feature, cmvn
• Training info: lr 0.001, batchsize static16, 8 * 3090 gpu, gradient_checkpoint, 100 epochs, about 8 hours
• Decoding info: ctc_weight 0.5, average_num 30

Aishell4-test Copy Section

WenetSpeech-test_meeting Copy Section

Overview (`examples/chime4/s0/README.md`) Copy Section

Performance Record

Conformer Result Copy Chapter

• Feature info: dither + specaug + speed perturb
• Training info: lr 0.0005, batch size 8, 1 gpu, acc_grad 4, 80 epochs
• Decoding info: average_num 10

Overview (`examples/commonvoice/fr/README.md`) Copy Section

Performance Record

Should be installed ffmpeg , pandas !!!

Conformer Result Copy Chapter

• Feature info: dither + specaug + speed perturb
• Training info: lr 0.0005, warmup_steps 20000 batch size 8, 3 gpu, 30 epochs
• Decoding info: average_num 20

Overview (`examples/csj/s0/README.md`) Copy Section

Performance Record

Conformer Result Bidecoder (large) Copy Chapter

Conformer Result Copy Chapter

• Feature info: using fbank feature, cmvn, dither, online speed perturb
• Training info: train_conformer.yaml, kernel size 15, lr 0.004, batch size 12, 8 gpu, acc_grad 1, 50 epochs, dither 0.0
• Decoding info: ctc_weight 0.5, average_num 10

note that "+" means we removed two <0.1s wav files in test1 before decoding.

Conformer U2++ Result Copy Chapter

Conformer U2 Result Copy Chapter

Overview (`examples/gigaspeech/s0/README.md`) Copy Section

GigaSpeech

A Large, modern and evolving dataset for automatic speech recognition. More details about GigaSpeech can be found: https://github.com/SpeechColab/GigaSpeech

Performance Record

Conformer bidecoder Result Copy Chapter

• Feature info: using fbank feature, dither 1.0, cmvn, 16k
• Training info: conf/train_conformer_bidecoder.yaml, subsample 4, kernel size 31, lr 0.001, batch size 24, 8 gpu, acc_grad 4, 40 epochs
• Decoding info: ctc_weight 0.3, reverse_weight 0.5, average_num 10
• Git hash: 9a0c270f9f976d7e887f777690e6c358a45a1c27

test set gigaspeech scoring Copy Section

dev set gigaspeech scoring Copy Section

Conformer U2++ Result Copy Chapter

• Feature info: using fbank feature, dither 1.0, cmvn, 16k
• Training info: conf/train_u2++_conformer.yaml, subsample 6, kernel size 31, lr 0.001, batch size 28, 8 gpu, acc_grad 1, 50 epochs
• Decoding info: ctc_weight 0.3, reverse_weight 0.5, average_num 10
• Git hash: 9a0c270f9f976d7e887f777690e6c358a45a1c27

test set gigaspeech scoring, full chunk (non-streaming) Copy Section

test set gigaspeech scoring, chunk 8 (latency range from 0 to 480ms) Copy Section

Conformer Result Copy Chapter

• Feature info: using fbank feature, dither 1.0, no cmvn, 48k
• Training info: conf/train_conformer.yaml, kernel size 31, lr 0.001, batch size 24, 8 gpu, acc_grad 4, 30 epochs
• Decoding info: ctc_weight 0.5, average_num 5
• Git hash: 9a0c270f9f976d7e887f777690e6c358a45a1c27

test set gigaspeech scoring Copy Section

Overview (`examples/hkust/s0/README.md`) Copy Section

Performance Record

Conformer Result (Old IO) Copy Chapter

• Feature info: using fbank feature, with cmvn, with speed perturb.
• Training info: lr 0.002, batch size 16, 1 machines, 1*4 = 4 gpu, acc_grad 4, 240 epochs, dither 0.1
• Decoding info: ctc_weight 0.5, average_num 30

Conformer Result (New IO) Copy Chapter

• Feature info: using fbank feature, with cmvn, with speed perturb.
• Training info: lr 0.002, batch size 16, 1 machines, 1*4 = 4 gpu, acc_grad 4, 133 epochs, dither 0.1
• Decoding info: ctc_weight 0.5, average_num 30

Overview (`examples/librispeech/rnnt/README.md`) Copy Section

Performance Record

Conformer Bidecoder Transducer Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, online speed perturb
• Training info: lr 0.001, dynamic batch with max_frames_in_batch 4000, 8 gpu, acc_grad 1, 60 epochs
• Training weight info: transducer_weight 0.75, ctc_weight 0.1, reverse_weight 0.30, average_num 10
• Predictor type: lstm

Pretrained model: https://huggingface.co/yuekai/wenet-asr-librispeech-conformer-transducer-mtl/blob/main/exp/conformer_transducer/avg_10.pt

Overview (`examples/librispeech/s0/README.md`) Copy Section

Performance Record

Conformer Result Bidecoder (large) Copy Chapter

• Encoder FLOPs(30s): 96,238,430,720, params: 85,709,704
• Feature info: using fbank feature, cmvn, dither, online speed perturb
• Training info: train_conformer_bidecoder_large.yaml, kernel size 31, lr 0.002, batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 1.0
• Decoding info: ctc_weight 0.3, reverse weight 0.5, average_num 30
• Git hash: 65270043fc8c2476d1ab95e7c39f730017a670e0
• LM-tgmed: 3-gram.pruned.1e-7.arpa.gz
• LM-tglarge: 3-gram.arpa.gz
• LM-fglarge: 4-gram.arpa.gz

SqueezeFormer Result (U2++, FFN:2048) Copy Chapter

• Encoder info:
  • SM12, reduce_idx 5, recover_idx 11, conv1d, batch_norm, syncbn
  • encoder_dim 512, output_size 512, head 8, ffn_dim 512*4=2048
  • Encoder FLOPs(30s): 82,283,704,832, params: 85,984,648
• Feature info:
  • using fbank feature, cmvn, dither, online speed perturb, spec_aug
• Training info:
  • train_squeezeformer_bidecoder_large.yaml, kernel size 31
  • batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 1.0
  • adamw, lr 8e-4, NoamHold, warmup 0.2, hold 0.3, lr_decay 1.0
• Decoding info:
  • ctc_weight 0.3, reverse weight 0.5, average_num 30

Conformer Result Copy Chapter

• Encoder FLOPs(30s): 34,085,088,512, params: 34,761,608
• Feature info: using fbank feature, cmvn, dither, online speed perturb
• Training info: train_conformer.yaml, kernel size 31, lr 0.004, batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 0.1
• Decoding info: ctc_weight 0.5, average_num 30
• Git hash: 90d9a559840e765e82119ab72a11a1f7c1a01b78
• LM-fglarge: 4-gram.arpa.gz

Conformer Result (12 layers, FFN:2048) Copy Chapter

• Encoder FLOPs(30s): 34,085,088,512, params: 34,761,608
• Feature info: using fbank feature, cmvn, dither, online speed perturb
• Training info: train_squeezeformer.yaml, kernel size 31,
• batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 0.1
• AdamW, lr 1e-3, NoamHold, warmup 0.2, hold 0.3, lr_decay 1.0
• Decoding info: ctc_weight 0.3, reverse weight 0.5, average_num 30

SqueezeFormer Result (SM12, FFN:1024) Copy Chapter

• Encoder info:
  • SM12, reduce_idx 5, recover_idx 11, conv2d, w/o syncbn
  • encoder_dim 256, output_size 256, head 4, ffn_dim 256*4=1024
  • Encoder FLOPs(30s): 21,158,877,440, params: 22,219,912
• Feature info:
  • using fbank feature, cmvn, dither, online speed perturb
• Training info:
  • train_squeezeformer.yaml, kernel size 31,
  • batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 0.1
  • adamw, lr=1e-3, noamhold, warmup=0.2, hold=0.3, lr_decay=1.0
• Decoding info: ctc_weight 0.3, reverse weight 0.5, average_num 30

SqueezeFormer Result (SM12, FFN:2048) Copy Chapter

• Encoder info:
  • SM12, reduce_idx 5, recover_idx 11, conv2d, w/o syncbn
  • encoder_dim 256, output_size 256, head 4, ffn_dim 256*8=2048
  • encoder FLOPs(30s): 28,230,473,984, params: 34,827,400
• Feature info: using fbank feature, cmvn, dither, online speed perturb
• Training info:
  • train_squeezeformer.yaml, kernel size 31
  • batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 0.1
  • adamw, lr 1e-3, noamhold, warmup 0.2, hold 0.3, lr_decay 1.0
• Decoding info:
  • ctc_weight 0.3, reverse weight 0.5, average_num 30

SqueezeFormer Result (SM12, FFN:1312) Copy Chapter

• Encoder info:
  • SM12, reduce_idx 5, recover_idx 11, conv1d, w/o syncbn
  • encoder_dim 328, output_size 256, head 4, ffn_dim 328*4=1312
  • encoder FLOPs(30s): 34,103,960,008, params: 35,678,352
• Feature info:
  • using fbank feature, cmvn, dither, online speed perturb
• Training info:
  • train_squeezeformer.yaml, kernel size 31,
  • batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 1.0
  • adamw, lr 1e-3, noamhold, warmup 0.2, hold 0.3, lr_decay 1.0
• Decoding info:
  • ctc_weight 0.3, reverse weight 0.5, average_num 30

Conformer U2++ Result Copy Chapter

• Feature info: using fbank feature, cmvn, no speed perturb, dither
• Training info: train_u2++_conformer.yaml lr 0.001, batch size 24, 8 gpu, acc_grad 1, 120 epochs, dither 1.0
• Decoding info: ctc_weight 0.3, reverse weight 0.5, average_num 30
• Git hash: 65270043fc8c2476d1ab95e7c39f730017a670e0

test clean

test other

SqueezeFormer Result (U2++, FFN:2048) Copy Chapter

• Encoder info:
  • SM12, reduce_idx 5, recover_idx 11, conv1d, layer_norm
  • do_rel_shift false, warp_for_time, syncbn
  • encoder_dim 256, output_size 256, head 4, ffn_dim 256*8=2048
  • Encoder FLOPs(30s): 28,255,337,984, params: 34,893,704
• Feature info:
  • using fbank feature, cmvn, dither, online speed perturb
• Training info:
  • train_squeezeformer.yaml, kernel size 31
  • batch size 12, 8 gpu, acc_grad 2, 120 epochs, dither 1.0
  • adamw, lr 8e-4, NoamHold, warmup 0.2, hold 0.3, lr_decay 1.0
• Decoding info:
  • ctc_weight 0.3, reverse weight 0.5, average_num 30

test clean

test other

Branchformer U2++ Result Copy Chapter

• Feature info: using fbank feature, cmvn, online speed perturb, dither
• Encoder info: layer num 24, cnn_kernel size 63, cgmlp linear units: 2048
• Training info: train_u2++_branchformer.yaml lr 0.001, batch size 16, 8 gpu, acc_grad 1, 120 epochs, dither 1.0
• Decoding info: ctc_weight 0.3, reverse weight 0.5, average_num 30

test clean

test other

Conformer U2 Result Copy Chapter

• Feature info: using fbank feature, cmvn, speed perturb, dither
• Training info: train_unified_conformer.yaml lr 0.001, batch size 10, 8 gpu, acc_grad 1, 120 epochs, dither 1.0
• Decoding info: ctc_weight 0.5, average_num 30
• Git hash: 90d9a559840e765e82119ab72a11a1f7c1a01b78
• LM-tgmed: 3-gram.pruned.1e-7.arpa.gz
• LM-tglarge: 3-gram.arpa.gz
• LM-fglarge: 4-gram.arpa.gz

test clean

test other

Efficient Conformer V1 Result Copy Chapter

• Feature info:
  • using fbank feature, cmvn, speed perturb, dither
• Training info:
  • train_u2++_efficonformer_v1.yaml
  • 8 gpu, batch size 16, acc_grad 1, 120 epochs
  • lr 0.001, warmup_steps 35000
• Model info:
  • Model Params: 49,474,974
  • Downsample rate: 1/4 (conv2d) * 1/2 (efficonformer block)
  • encoder_dim 256, output_size 256, head 8, linear_units 2048
  • num_blocks 12, cnn_module_kernel 15, group_size 3
• Decoding info:
  • ctc_weight 0.5, reverse_weight 0.3, average_num 20
• Model Download: wenet_efficient_conformer_librispeech_v1

test clean

test other

Efficient Conformer V2 Result Copy Chapter

• Feature info:
  • using fbank feature, cmvn, speed perturb, dither
• Training info:
  • train_u2++_efficonformer_v2.yaml
  • 8 gpu, batch size 16, acc_grad 1, 120 epochs
  • lr 0.001, warmup_steps 35000
• Model info:
  • Model Params: 50,341,278
  • Downsample rate: 1/2 (conv2d2) * 1/4 (efficonformer block)
  • encoder_dim 256, output_size 256, head 8, linear_units 2048
  • num_blocks 12, cnn_module_kernel 15, group_size 3
• Decoding info:
  • ctc_weight 0.5, reverse_weight 0.3, average_num 20
• Model Download: wenet_efficient_conformer_librispeech_v2

test clean

test other

Overview (`examples/multi_cn/s0/README.md`) Copy Section

Performance Record

This is a Chinese speech recognition recipe that trains on all Chinese corpora including:

Unified Transformer Result Copy Chapter

Data info: Copy Section

• Dataset: Aidatatang, Aishell, MagicData, Primewords, ST-CMDS, and THCHS-30.
• Feature info: using fbank feature, with cmvn, no speed perturb.
• Training info: lr 0.004, batch size 18, 3 machines, 3*8 = 24 GPUs, acc_grad 1, 220 epochs, dither 0.1
• Decoding info: ctc_weight 0.5, average_num 30
• Git hash: 013794572a55c7d0dbea23a66106ccf3e5d3b8d4

WER Copy Section

Unified Conformer Result Copy Chapter

Data info: Copy Section

• Dataset: Aidatatang, Aishell, MagicData, Primewords, ST-CMDS, and THCHS-30.
• Feature info: using fbank feature, with cmvn, speed perturb.
• Training info: lr 0.001, batch size 8, 1 machines, 1*8 = 8 GPUs, acc_grad 12, 60 epochs
• Decoding info: ctc_weight 0.5, average_num 10
• Git hash: 5bdf436e671ef4c696d1b039f29cc33109e072fa

WER Copy Section

Unified Conformer Result Copy Chapter

Data info: Copy Section

• Dataset: Aidatatang, Aishell, MagicData, Primewords, ST-CMDS, THCHS-30, TAL-ASR, and AISHELL2.
• Feature info: using fbank feature, dither=0, cmvn, speed perturb
• Training info: lr 0.001, batch size 22, 4 GPUs, acc_grad 4, 120 epochs, dither 0.1
• Decoding info: ctc_weight 0.5, average_num 10
• Git hash: 66f30c197d00c59fdeda3bc8ada801f867b73f78

WER Copy Section

Overview (`examples/openasr2021/s0/README.md`) Copy Section

w2v-conformer based end-to-end model for Openasr2021 challenge

This is a example to use unsupervised pretrained w2v-conformer model to fintune OpenASR2021 constrained-plus tasks.

We pretrain conformer encoders using wav2vec 2.0 pre-training method , which we called ch-w2v-conformer. The original pre-training works take raw waveforms
as input. Unlike these works, we use MFCC features as inputs.

The ch-w2v-conformer model uses following datasets to pretrain:

ISML datasets (6 languages,70k hours): internal dataset contains 40k hours Chinese, Cantonese, Tibetan, Inner Mongolian, Inner Kazakh, Uighur.

Babel datasets (17 languages, 2k hours): Assamese, Bengali, Cantonese, Cebuano, Georgian, Haitian, Kazakh, Kurmanji, Lao, Pashto, Swahili, Tagalog, Tamil, Tok, Turkish, Vietnamese, Zulu

After pretraining, we build ASR system based on CTC-Attention structure. In very low resource task, we find that if too many initialization network structures are constructed in the upper layer of pre-training conformer encoder, the migration performance of the pre-training model will be destroyed, so we only build a single-layer transformer decoder for joint training.

pretrained model link: https://huggingface.co/emiyasstar/ch-w2v-conformer

constrained-plus Task Performance Copy Chapter

• Languages: Cantonese,mongolian,kazakh
• config: conf/train_conformer_large_10h.yaml
• Feature info: using mfcc feature, with dither 1.0, without cmvn
• Training info: lr 0.001, batch size 10, 4 gpus on V100, acc_grad 1, 80 epochs
• Decoding info: ctc_weight 0.5, average_num 35

dev set results trained only with 10 hours training set

w2v-Conformer Copy Chapter

Conformer （train from scratch） Copy Chapter

Overview (`examples/swbd/s0/README.md`) Copy Section

Performance Record

Conformer Result Copy Chapter

• Feature info: dither + specaug + speed perturb
• Training info: lr 0.001, warmup_steps 25000, batch size 16, 1 gpu, acc_grad 4, 240 epochs
• Decoding info: average_num 10

Overview (`examples/tedlium3/s0/README.md`) Copy Section

Performance Record

Conformer Result Copy Chapter

• Feature info: using fbank feature, dither, cmvn, without speed perturb (not supported segments yet)
• Training info: lr 0.001, batch size 20, 8 gpu, acc_grad 1, 240 epochs, dither 0.1
• Decoding info: ctc_weight 0.5, average_num 10

Overview (`examples/timit/s0/README.md`) Copy Section

Performance Record

Conformer Result Copy Chapter

• Feature info: dither + specaug + speed perturb
• Training info: lr 0.002, warmup_steps 5000 batch size 16, 1 gpu, acc_grad 4, 120 epochs
• Decoding info: average_num 20
• trans_type: phn

transformer Result Copy Chapter

• Feature info: dither + specaug + speed perturb
• Training info: lr 0.002, warmup_steps 5000 batch size 16, 1 gpu, acc_grad 4, 120 epochs
• Decoding info: average_num 20
• trans_type: phn

Overview (`examples/vkw2021/s0/README.md`) Copy Section

conformer based end-to-end model for VKW challenge

Standard E2E Results Copy Chapter

Conformer without speed perpurb and lm

• config: conf/train_train_vkw_bidirect_12conformer_hs2048_output256_att4_conv2d_char.yaml
• beam: 10
• num of gpu: 8
• num of averaged model: 5
• ctc weight (used for attention rescoring): 0.5

dev set results trained only with training set (785 keywords, 1505 hour train set)

dev set results trained with training set and finetune set (785 keywords, 1505 hour train set + 15 hour finetune set)

test set results trained only with training set (384 keywords, 1505 hour train set)

test set results trained with training set and finetune set (384 keywords, 1505 hour train set + 15 hour finetune set)

Overview (`examples/wenetspeech/paraformer/README.md`) Copy Section

Preliminary

1. Run below command to convert funasr-style ckpt to wenet-style ckpt:

output_dir=exp/paraformer/large
mkdir -p ${output_dir}
. ./path.sh && python wenet/paraformer/convert_paraformer_to_wenet_config_and_ckpt.py \
  --output_dir ${output_dir}

Performance Record

Paraformer (original) Result Copy Chapter

Paraformer-dynamic training (full-parameter tuning) Result Copy Chapter

• Training info: lr 0.0005, batch size dynamic36000, gradient checkpointing, torch_ddp, acc_grad 4, 8 * 3090 gpus, 20 epochs, about 4 days
• Decoding info: ctc_weight 0.5, average_num 3, blank penalty 2.5 for dev and 0.0 for others
• Git hash: TBD

NOTE Copy Section

This is our first attempt at fine-tuning the paraformer-large to enable stream inference through a wenet-like chunk method.
Although the non-streaming results deteriorated after fine-tuning compared to before, we believe there is still significant room for improvement for paraformer-large when fine-tuned within wenet, considering this is a very initial result.

Additionally, on the same training set (wenetspeech+aishell4), we trained a conformer-large model from scratch (see experimental results in examples/aishell/s0). Comparing it with the fine-tuned results of paraformer-large, we found that the CTC results of paraformer-large consistently outperformed those of conformer-large, and the NAR results of paraformer-large were always better than the rescore of conformer-large. This is mainly due to paraformer-large having been pre-trained on 6wh industrial data, giving the model a better initialization.

Overview (`examples/wenetspeech/s0/README.md`) Copy Section

Performance Record

Conformer Copy Chapter

• Feature info: using fbank feature, with dither 1.0, with cmvn
• Training info: lr 0.001, batch size 32, 24 gpus on V100, acc_grad 16, 26 epochs
• Decoding info: ctc_weight 0.5, average_num 10

Conformer bidecoder Copy Chapter

• Feature info: using fbank feature, with dither 1.0, with cmvn
• Training info: lr 0.001, batch size 32, 24 gpus on V100, acc_grad 16, 26 epochs
• Decoding info: ctc_weight 0.5, average_num 10

U2++ conformer Copy Chapter

• Feature info: using fbank feature, with dither 1.0, with cmvn
• Training info: lr 0.002, batch size dynamic24000, 24 gpus on 3090, acc_grad 16, 80 epochs, 4.5 days
• Decoding info: ctc_weight 0.5, reverse_weight 0.0, average_num 10, blank penalty 2.5, length penalty 8.5 for dev/testmeeting and 0.0 for testnet

U2++ conformer (text\_fixed, see https://github.com/wenet-e2e/WenetSpeech/discussions/54) Copy Chapter

• Feature info: using fbank feature, with dither 1.0, with cmvn
• Training info: lr 0.001, batch size dynamic36000, 8 gpus on 3090, acc_grad 4, 130k steps, 4.6 days
• Decoding info: ctc_weight 0.5, reverse_weight 0.0, average_num 5, blank penalty 0.0, length penalty 0.0
• PR link: https://github.com/wenet-e2e/wenet/pull/2371

U2++ conformer (wenetspeech plus aishell4) Copy Chapter

• Feature info: using fbank feature, with dither 1.0, with cmvn
• Training info: lr 0.001, batch size dynamic36000, gradient checkpointing, torch_ddp, 8 * 3090 gpus, acc_grad 4, 60 epochs, about 8.5 days
• Decoding info: ctc_weight 0.5, reverse_weight 0.0, average_num 10, blank penalty 2.5 for dev and 0.0 for others

Overview (`examples/wenetspeech/whisper/README.md`) Copy Section

Preliminary

1. Download whisper ckpt from this link

2. We assume you have run stage0~stage3 using wenetspeech/s0/run.sh and here we are simply creating a symbolic link

ln -s ../s0/data .

3. Run below command to convert openai-style ckpt to wenet-style ckpt:

mkdir -p exp
mkdir -p exp/whisper
mkdir -p exp/whisper/large-v3
. ./path.sh && python wenet/whisper/convert_whisper_to_wenet_config_and_ckpt.py \
  --whisper_ckpt downloaded-large-v3.pt \
  --output_dir exp/whisper/large-v3
# remove conv1d2
python local/modify_ckpt.py \
  --remove_list "encoder.embed.conv" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.remove-subsample.pt
# init ctc
python local/modify_ckpt.py \
  --add_list "{\"ctc.ctc_lo.weight\": \"decoder.embed.0.weight\"}" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.init-ctc.pt
# remove conv1d2 and init ctc
python local/modify_ckpt.py \
  --remove_list "encoder.embed.conv" \
  --add_list "{\"ctc.ctc_lo.weight\": \"decoder.embed.0.weight\"}" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.remove-subsample.init-ctc.pt

Performance Record

Whisper-largev2 (original) Result Copy Chapter

Whisper-largev3 (conv1d2, full-parameter tuning) Result Copy Chapter

• Feature info: using log_mel_spectrogram feature, no cmvn
• Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 8, 8 * 3090 gpu, 40k steps (about 5 days), conf/finetune_whisper_largev3.yaml
• Decoding info: ctc_weight 0.0, average_num 5
• PR link: https://github.com/wenet-e2e/wenet/pull/2356

Whisper-largev3 (conv1d2, full-parameter tuning) Result (text\_fixed, see https://github.com/wenet-e2e/WenetSpeech/discussions/54) Copy Chapter

• Feature info: using log_mel_spectrogram feature, no cmvn
• Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 8, 8 * 3090 gpu, 48k steps (about 6 days), conf/finetune_whisper_largev3.yaml
• Decoding info: ctc_weight 0.0, average_num 5
• PR link: https://github.com/wenet-e2e/wenet/pull/2371

Frequently Asked Questions

• Q: Why are there so many insertion errors in the decoding results of CTC and attention_rescoring?
• A: Because Chinese characters are composed of multiple bytes, in Whisper's tokenizer, one Chinese character might be represented by multiple tokens (for example, 3 tokens). During the CTC decoding process, it's possible that two or four tokens are decoded. This not only causes garbled text (see https://github.com/wenet-e2e/wenet/issues/2308 ) but also leads to insertion errors.

Overview (`examples/wsj/s0/README.md`) Copy Section

Performance Record

Conformer Result Copy Chapter

• Feature info: dither + specaug + speed perturb
• Training info: lr 0.002, warmup_steps 20000 batch size 16, 1 gpu, acc_grad 4, 120 epochs
• Decoding info: average_num 20

Overview (`runtime/core/api/README.md`) Copy Section

WeNet API

We refer vosk
for the interface design.

We are going to implement the following interfaces:

• non-streaming recognition
• [] streaming recognition
• [] nbest
• [] contextual biasing word
• [] alignment
• [] language support(post processor)
• [] label check

Overview (`runtime/core/kaldi/README.md`) Copy Section

We use Kaldi decoder to implement TLG based language model integration,
so we copied related files to this directory.
The main changes are:

1. To minimize the change, we use the same directories tree as Kaldi.

2. We replace Kaldi log system with glog in the following way.

#define KALDI_WARN \
  google::LogMessage(__FILE__, __LINE__, google::GLOG_WARNING).stream()
#define KALDI_ERR \
  google::LogMessage(__FILE__, __LINE__, google::GLOG_ERROR).stream()
#define KALDI_INFO \
  google::LogMessage(__FILE__, __LINE__, google::GLOG_INFO).stream()
#define KALDI_VLOG(v) VLOG(v)

#define KALDI_ASSERT(condition) CHECK(condition)

3. We lint all the files to satisfy the lint in WeNet.

Overview (`runtime/gpu/cuda_decoders/README.md`) Copy Section

Using CUDA based Decoders for Triton ASR Server Copy Chapter

Introduction Copy Section

The triton model repository model_repo_cuda_decoder here, integrates the CUDA WFST decoder originally described in https://arxiv.org/abs/1910.10032. We take small conformer fp16 onnx inference for offline ASR as an example.

Quick Start Copy Section

# using docker image runtime/gpu/Dockerfile/Dockerfile.server
docker pull soar97/triton-wenet:22.12
docker run -it --rm --name "wenet_tlg_test" --gpus all --shm-size 1g --net host soar97/triton-wenet:22.12
# inside the docker container
git clone https://github.com/wenet-e2e/wenet.git
cd wenet/runtime/gpu/cuda_wfst_decoder
# Use pip3 install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple if you encounter network issue
pip3 install -r requirements.txt

# For offline TLG ASR
bash run.sh
# For streaming TLG ASR
bash run_streaming.sh

Performance of Small Offline ASR Model using Different Decoders Copy Section

Benchmark(small offline conformer onnx fp16 model trained on Aishell1) based on Aishell1 test set with V100, the total audio duration is 36108.919 seconds.

(Note: 80 concurrent tasks, service has been fully warm up.)

Overview (`runtime/gpu/tensorrt/README.md`) Copy Section

Using Tensorrt for Triton ASR Server Copy Section

# using docker image runtime/gpu/Dockerfile/Dockerfile.server
docker pull soar97/triton-wenet:22.12
docker run -it --rm --name "wenet_trt_test" --gpus all --shm-size 1g --net host soar97/triton-wenet:22.12
# inside the docker container
git clone https://github.com/wenet-e2e/wenet.git
cd wenet/runtime/gpu/tensorrt
pip3 install nvidia-pyindex
# Use pip3 install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple if you encounter network issue
pip3 install -r requirements.txt

bash run_streaming_small_model.sh

Performance of Small u2pp Model for Streaming ASR

Benchmark(small u2pp onnx) based on Aishell1 test set with server-A10 (16vCPU 60GB Memory)/client(4vCPU 16GB Memory), the total audio duration is 36108.919 seconds.

(Note: using non-simulate-streaming mode)

Overview (`runtime/gpu/tensorrt_fastertransformer/README.md`) Copy Section

Recipe for use of FasterTransformer accelerated Conformer model with Triton Copy Chapter

# using docker image runtime/gpu/Dockerfile/Dockerfile.server
docker pull soar97/triton-wenet:22.12
docker run -it --rm --name "wenet_trt_test" --gpus all --net host --shm-size=1g soar97/triton-wenet:22.12
# inside the docker container
git clone https://github.com/wenet-e2e/wenet.git
cd wenet/runtime/gpu/tensorrt_fastertransformer
git submodule update --init
pip3 install nvidia-pyindex
# Use pip3 install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple if you encounter network issue
pip3 install -r requirements.txt
# Please modify the model parameters in run.sh. This recipe is for small conformer model.
bash run.sh

With help of FasterTransformer wenet tensorrt plugins, the overall throughput could get ~50% improvement comparing with onnx fp16 inference.

Performance of Small Model for AIShell2

The following benchmakr shows the performance on T4 of a small Conformer model for AIShell2 case as defined in WeNet AIShell2 Example.

Note: Currently, encoder only supports offline conformer model. Decoder only supports transformer decoder.

For directly using FasterTransformer built wenet, you may refer to https://github.com/NVIDIA/FasterTransformer/tree/main/examples/cpp/wenet

UIO (`docs/UIO.md`) Copy Section

Big Data Training: UIO

In order to support the model training of industrial tens of millions of hours of speech dataset, the data processing
method UIO (Unified IO) has been updated in WeNet. The document will introduce UIO from the following sections:
Necessity of upgrading IO mothod, System design of UIO, Validation experiments, Usage of UIO, Q&A.

Necessity of upgrading IO mothod Copy Chapter

The old IO method in WeNet is based on Pytorch's native Dataset. During training, it need to load all training audio
paths and correspondingly labels into the memory at one time, then randomly read data. In the case of industrial-grade
ultra-large-scale data (egs: more than 50,000 hours or 50 million or more audio), this method will cause the training
to fail to run for two reasons:

• Out of memory(OOM): The physical memory of the general machine is difficult to load the training data at one time.
• Slow down reading performance: In the case that the large-scale data memory cannot be used as a file cache, the training
  data reading speed is greatly reduced.

System design of UIO Copy Chapter

Overall design Copy Section

Inspired by the following industrial methods(egs: webdataset, TFRecord),
WeNet redesigned the IO method. Its core idea is to make the audio and labels of multiple small data(such as 1000 pieces),
into compressed packets (tar) and read them based on the IterableDataset of Pytorch. The advantages of this method is：

• Only the index information of the compressed package needs to be maintained in memory, which greatly saves memory and
  solves the problem of OOM.
• The on-the-fly decompression is performed in the memory, and the data in the same compressed package is read in
  sequence, which solves the problem of slow random reading performance. Different compressed packets can be read randomly
  to ensure the global randomness of data.

The new IO method takes into account both small datasets and large datasets， and provides two data reading methods.
We call it UIO. The overall design of UIO is shown in the figure below:

Some necessary explanations about the above figure:

• Small IO(raw) supports small datasets, which we call raw mode. This mode only supports local file reading.
  The required documents must be sorted into Kaldi style file: wav.scp and text.(It's the same as before)
• Big IO(shared) supports large datasets, which we call shard mode. This mode can support both local file
  reading and network cloud storage file reading. The required files must be sorted into compressed packages. Audio (wav)
  and label (txt) are stored in a single compressed package in sequence.

Chain IO Copy Section

Inspired by TFRecord chain IO, UIO also adopts chain implementation. In practical use, chain IO is more flexible,
easier to expand and easier to debug. TFRecord IO example as follows,

def read_dataset(filename, batch_size):
    dataset = tf.data.TFRecordDataset(filename)
    dataset = dataset.map(_parse_image_function, num_parallel_calls=tf.data.experimental.AUTOTUNE)
    dataset = dataset.shuffle(500)
    dataset = dataset.batch(batch_size, drop_remainder=True)
    dataset = dataset.repeat()
    dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
    return dataset

Refer to TFRecord IO, the UIO dataflow in WeNet is designed as the figure below:

It includes the following modules:

• tokenize module: convert the label into specify modeling unit(egs: char or BPE).

• filter module: filter out unqualified training data.

• resample module: optional resampling of training data.

• compute_fbank module: fbank feature extraction.

• spec_augmentation module: feature enhancement.

• shuffle module: disrupt local data.

• sort module: sort local data.

• batch module: organize multiple pieces of data into batch.

• padding module: padding data in the same batch.

what's more, There are several parameters to note. first, shuffle buffer and sort buffer in buffer size:

• Shuffle buffer: shuffle data. It is recommended that the size of this buffer be larger than
  the number of data contained in a single shard. Each time it is equivalent to shuffling data between two shards,
  which increases the randomness of the data.(egs: if each shard contains 1000, you can set shuffle buffer as 1500)
• Sort buffer: sort the data according to the number of frames. This operation is very important and can greatly
  improve the training speed.

then, Prefetch:
Prefetch is used in the Pytorch Dataloader to pre-read data. The granularity of prefetch is the batch of final training.
The default parameter is 2, that is, the data of two batches will be pre-read by default. In the design of the UIO,
due to the existence of the pre buffer, the pre-read data may already be in the buffer, so there is no real pre read.
Only when the data in the buffer is insufficient during the next training can the buffer be filled on the fly.
At this time, the training is blocked on the read data. In a word, when the prefetch is very small, the training will
block reading data in part of the time, because the previous data is still in cache. Therefore, you can set a large
prefetch to avoid this problem.

Validation experiments Copy Chapter

At present, we have verified the accuracy of UIO on aishell (200 hours) and wenetspeech (10000 hours) data respectively.

Aishell(``raw`` vs ``shard``) Copy Section

WenetSpeech(``shard``) Copy Section

WeNet and ESPnet use similar model structure and parameter configuration, and they achieve similar recognition rate,
which shows the correctness of UIO in WeNet. And during the training, we observed that the overall utilization rate of
GPU of UIO is more than 80% - 90%, indicating that the overall IO reading efficiency is very high.

Usage of UIO Copy Chapter

For detailed usage of UIO, please refer to the aishell dataset example:
https://github.com/wenet-e2e/wenet/blob/main/examples/aishell/s0/run.sh
At present, all datasets in WeNet have used UIO as the default data preparation.

There are three parameters related to UIO in the training script train.py：

• train_data(cv_data/test_data): data.list
• data_type: raw/shard
• symbol_table: specify modeling unit

For example:

python wenet/bin/train.py --gpu $gpu_id \
  --config $train_config \
  --data_type $data_type \
  --symbol_table $dict \
  --train_data $feat_dir/$train_set/data.list \
  --cv_data $feat_dir/dev/data.list \
  ...

If data_type is raw, the format of data.list is as follows:

{"key": "BAC009S0002W0122", "wav": "/export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0122.wav", "txt": "而对楼市成交抑制作用最大的限购"}
{"key": "BAC009S0002W0123", "wav": "/export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0123.wav", "txt": "也成为地方政府的眼中钉"}
{"key": "BAC009S0002W0124", "wav": "/export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0124.wav", "txt": "自六月底呼和浩特市率先宣布取消限购后"}

Each line is a json serialized string, which contains three fields: key, wav and txt.

If data_type is shard, the format of data.list is as follows:

# [option 1: local]
/export/maryland/binbinzhang/code/wenet/examples/aishell/s3/raw_wav/train/shards/shards_000000000.tar.gz
/export/maryland/binbinzhang/code/wenet/examples/aishell/s3/raw_wav/train/shards/shards_000000001.tar.gz
/export/maryland/binbinzhang/code/wenet/examples/aishell/s3/raw_wav/train/shards/shards_000000002.tar.gz

# [option 2: network(egs: OSS)]
https://examplebucket.oss-cn-hangzhou.aliyuncs.com/exampledir/1.tar.gz
https://examplebucket.oss-cn-hangzhou.aliyuncs.com/exampledir/2.tar.gz

Q&A Copy Chapter

Q1: How to operate distributed partition of training data?

A: According to rank and num_workers can segment the data. for example:

class DistributedSampler:
    def __init__(self, shuffle=True, partition=True):
        self.epoch = -1
        self.update()
        self.shuffle = shuffle
        self.partition = partition

    def set_epoch(self, epoch):
        self.epoch = epoch

    def sample(self, data):
        data = list(range(len(data)))
        if self.partition:
            if self.shuffle:
                random.Random(self.epoch).shuffle(data)
            data = data[self.rank::self.world_size]
        data = data[self.worker_id::self.num_workers]
        return data

Q2: How to deal with unbalanced data?

A: Use model.join() to handle the imbalance of data allocated on each rank. Please refer this.

Context (`docs/context.md`) Copy Section

Context Biasing Copy Chapter

In the practical application of ASR, the recognition effect of commonly used words is better, but for some unique words, the recognition accuracy may be low. Contextual biasing is the problem of injecting prior knowledge into an ASR system during inference, for example a user’s favorite songs, contacts, apps or location. Conventional ASR systems perform contextual biasing by building an n-gram finite state transducer (FST) from a list of biasing phrases, which is composed on-the-fly with the decoder graph during decoding. This helps to bias the recognition result towards the n-grams contained in the contextual FST, and thus improves accuracy in certain scenarios.

In WeNet, we compute the biasing scores $P_C(\mathbf y)$, which are interpolated with the base model $P(\mathbf y|\mathbf x)$ using shallow-fusion during beam search, including CTC prefix beam search and CTC WFST beam search.

$$
\mathbf y^*=\mathrm{arg,max,log}P(\mathbf y|\mathbf x)+\lambda,\mathrm{log},P_C(\mathbf y)
$$

where, $\lambda$ is a tunable hyperparameter controlling how much the contextual LM influences the overall model score during beam search.

Context Graph Copy Section

If we want to improve the score of the word "cat", and the biasing score $\lambda,\mathrm{log},P_C(\mathbf y)$ of each character is 0.25. The context graph can be constructed as follow:

In the decoding process, when the corresponding prefix is matched, the corresponding score reward will be obtained. In order to avoid artificially boosting prefixes which match early on but do not match the entire phrase, we add a special failure arc which removes the boosted score.

WeNet records only one state for each prefix, to easily determine the boundary of the matched hot word. That is, only one hot word can be matched at the same time, and only after the hot word matching succeeds or fails can other hot words start matching.

int ContextGraph::GetNextState(int cur_state, int word_id, float* score,
                               bool* is_start_boundary, bool* is_end_boundary) {
  int next_state = 0;
  // Traverse the arcs of current state.
  for (fst::ArcIterator<fst::StdFst> aiter(*graph_, cur_state); !aiter.Done();
       aiter.Next()) {
    const fst::StdArc& arc = aiter.Value();
    if (arc.ilabel == 0) {
      // Record the score of the backoff arc. It might will be covered.
      *score = arc.weight.Value();
    } else if (arc.ilabel == word_id) {
      // If they match, record the next state and the score.
      next_state = arc.nextstate;
      *score = arc.weight.Value();
      // Check whether is the boundary of the hot word.
      if (cur_state == 0) {
        *is_start_boundary = true;
      }
      if (graph_->Final(arc.nextstate) == fst::StdArc::Weight::One()) {
        *is_end_boundary = true;
      }
      break;
    }
  }
  return next_state;
}

CTC Prefix Beam Search Copy Section

In the process of CTC prefix beam search, each prefix needs to record the hot word matching information. After appending the current output character, if the prefix changes, call the above function GetNextState to update the state and score of the hot word. If it is the start or end of a hot word, it is also necessary to record the position, which are used to insert the start tag and end tag in the result, such as: "The <context>cat</context> is in the bag".

CTC WFST Beam Search Copy Section

WeNet adopts the Lattice Faster Online Decoder from Kaldi for WFST beam search. We have to modify the lattice-faster-decoder.cc to support context biasing.

WFST beam search decodes in the TLG graph according to the CTC outputs. If we bias the input label of the TLG, we need to compose the context graph with the Token graph. Finally, we decide to bias TLG's output towards the contextual fst. We need to modify the ProcessEmitting and ProcessNonemitting functions as follow:

Elem *e_next =
    FindOrAddToken(arc.nextstate, frame + 1, tot_cost, tok, NULL);
// NULL: no change indicator needed

// ========== Context code BEGIN ===========
bool is_start_boundary = false;
bool is_end_boundary = false;
float context_score = 0;
if (context_graph_) {
  if (arc.olabel == 0) {
    e_next->val->context_state = tok->context_state;
  } else {
    e_next->val->context_state = context_graph_->GetNextState(
      tok->context_state, arc.olabel, &context_score,
      &is_start_boundary, &is_end_boundary);
    graph_cost -= context_score;
  }
}
// ========== Context code END ==========

// Add ForwardLink from tok to next_tok (put on head of list
// tok->links)
tok->links = new ForwardLinkT(e_next->val, arc.ilabel, arc.olabel,
                              graph_cost, ac_cost, is_start_boundary,
                              is_end_boundary, tok->links);
tok->links->context_score = context_score;

Pruning Copy Section

The backoff arc will return the accumulated scores to a single ForwardLink. It leads to the cost of that ForwardLink is too large. We have to remove the cost returned by backoff arc before pruning.

void LatticeFasterDecoderTpl<FST, Token>::PruneForwardLinks(
    int32 frame_plus_one, bool *extra_costs_changed, bool *links_pruned,
    BaseFloat delta) {
  ...
  BaseFloat link_extra_cost =
            next_tok->extra_cost +
            ((tok->tot_cost + link->acoustic_cost + link->graph_cost) -
             next_tok->tot_cost);  // difference in brackets is >= 0
  // ========== Context code BEGIN ===========
  // graph_cost contains the score of hot word
  // link->context_score < 0 means the hot word of the link is returned from backoff arc
  if (link->context_score < 0) {
    link_extra_cost += link->context_score;
  }
  // ========== Context code END ==========
  // link_exta_cost is the difference in score between the best paths
  // through link source state and through link destination state

Usage Copy Section

1. Specify the --context_path to a text file.
   • Each line of the file contains a context.
   • Each context can be split into words with the symbol_table of the ASR model (It means there is no oov in the context).
2. Specify the --context_score, the reward of each word in the context.

cd /home/wenet/runtime/libtorch
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=docker_resource/test.wav
context_path=docker_resource/context.txt
model_dir=docker_resource/model
./build/decoder_main \
    --chunk_size -1 \
    --wav_path $wav_path \
    --model_path $model_dir/final.zip \
    --context_path $context_path \
    --context_score 3 \
    --unit_path $model_dir/units.txt 2>&1 | tee log.txt

Index (`docs/index.rst`) Copy Section

.. wenet documentation master file, created by
sphinx-quickstart on Thu Dec 3 11:43:53 2020.
You can adapt this file completely to your liking, but it should at least
contain the root toctree directive.

Welcome to wenet's documentation!

wenet is an tansformer-based end-to-end ASR toolkit.

.. toctree::
:maxdepth: 2
:caption: Contents:

./python_package.md
./train.rst
./production.rst
./reference.rst

Indices and tables

• :ref:genindex
• :ref:modindex
• :ref:search

Jit In Wenet (`docs/jit_in_wenet.md`) Copy Section

JIT in WeNet

We want that our PyTorch model can be directly exported by torch.jit.script method,
which is essential for deploying the model to production.

See the following resource for how to deploy PyTorch models in production in details.

• INTRODUCTION TO TORCHSCRIPT
• TORCHSCRIPT LANGUAGE REFERENCE
• LOADING A TORCHSCRIPT MODEL IN C++
• TorchScript and PyTorch JIT | Deep Dive
• Research to Production: PyTorch JIT/TorchScript Updates

To ensure that, we will try to export the model before training stage.
If it fails, we should modify the training code to satisfy the export requirements.

# See in wenet/bin/train.py
script_model = torch.jit.script(model)
script_model.save(os.path.join(args.model_dir, 'init.zip'))

Two principles should be taken into consideration when we contribute our python code
to WeNet, especially for the subclass of torch.nn.Module, and for the forward function.

1. Know what is allowed and what is disallowed.

   • Torch and Tensor Unsupported Attributes
   • Python Language Reference Coverage

2. Try to use explicit typing as much as possible. You can try to do type checking
   forced by typeguard, see https://typeguard.readthedocs.io/en/latest/userguide.html for details.

Lm (`docs/lm.md`) Copy Section

LM for WeNet

WeNet uses n-gram based statistical language model and the WFST framework to support the custom language model.
And LM is only supported in runtime of WeNet.

Motivation Copy Chapter

Why n-gram based LM? This may be the first question many people will ask.
Now that LM based on RNN and Transformer is in full swing, why does WeNet go backward?
The reason is simple, it is for productivity.
The n-gram-based language model has mature and complete training tools,
any amount of corpus can be trained, the training is very fast, the hotfix is easy,
and it has a wide range of mature applications in actual products.

Why WFST? It may be the second question many people will ask.
Since both industry and research have been working so hard to abandon traditional speech recognition,
especially the complex decoding technology. Why does WeNet back?
The reason is also very simple, it is for productivity.
WFST is a standard and powerful tool in traditional speech recognition.
And based on this solution, we have mature and complete bug fix solutions and product solutions,
such as that we can use the replace function in WFST for class-based personalization such as contact recognition.

Therefore, just like WeNet's design goal "Production first and Production Ready",
LM in WeNet also puts productivity as the first priority.
So it draws on many very productive tools and solutions accumulated in traditional speech recognition.
The difference to traditional speech recognition are:

1. The training in WeNet is pure end to end.
2. As described below, LM is optional in decoding, you can choose whether to use LM according to your needs and application scenarios.

System Design Copy Chapter

The whole system is shown in the bellowing picture. There are two ways to generate N-best.

1. Without LM, we use CTC prefix beam search to generate N-best.
2. With LM, we use CTC WFST search to generate N-best and CTC WFST search is the traditional WFST based decoder.

There are two main parts of the CTC WFST based search.

The first is building the decoding graph, which is to compose the model unit T, the lexicon L and the language model G into one unified graph TLG. And in which:

1. T is the model unit in E2E training. Typically it's char in Chinese, char or BPE in English.
2. L is the lexicon, the lexicon is very simple. What we need to do is just split a word into its modeling unit sequence.
   For example, the word "我们" is split into two chars "我 们", and the word "APPLE" is split into five letters "A P P L E".
   We can see there is no phonemes and there is no need to design pronunciation on purpose.
3. G is the language model, namely compiling the n-gram to standard WFST representation.

The second is the decoder, which is the same as the traditional decoder, which uses the standard Viterbi beam search algorithm in decoding.

Implementation Copy Chapter

WeNet draws on the decoder and related tools in Kaldi to support LM and WFST based decoding.
For ease of using and keeping independence, we directly migrated the code related to decoding in Kaldi to this directory in WeNet runtime.
And modify and organize according to the following principles:

1. To minimize changes, the migrated code remains the same directory structure as the original.
2. We use GLOG to replace the log system in Kaldi.
3. We modify the code format to meet the lint requirements of the code style in WeNet.

The core code is https://github.com/wenet-e2e/wenet/blob/main/runtime/core/decoder/ctc_wfst_beam_search.cc,
which wraps the LatticeFasterDecoder in Kaldi.
And we use blank frame skipping to speed up decoding.

In addition, WeNet also migrated related tools for building the decoding graph,
such as arpa2fst, fstdeterminizestar, fsttablecompose, fstminimizeencoded, and other tools.
So all the tools related to LM are built-in tools and can be used out of the box.

Results Copy Chapter

We get consistent gain (3%~10%) on different datasets,
including aishell, aishell2, and librispeech,
please go to the corresponding example dataset for the details.

How to use? Copy Chapter

Here is an example from aishell, which shows how to prepare the dictionary, how to train the LM,
how to build the graph, and how to decode with the runtime.

# 7.1 Prepare dict
unit_file=$dict
mkdir -p data/local/dict
cp $unit_file data/local/dict/units.txt
tools/fst/prepare_dict.py $unit_file ${data}/resource_aishell/lexicon.txt \
    data/local/dict/lexicon.txt
# 7.2 Train lm
lm=data/local/lm
mkdir -p $lm
tools/filter_scp.pl data/train/text \
     $data/data_aishell/transcript/aishell_transcript_v0.8.txt > $lm/text
local/aishell_train_lms.sh
# 7.3 Build decoding TLG
tools/fst/compile_lexicon_token_fst.sh \
    data/local/dict data/local/tmp data/local/lang
tools/fst/make_tlg.sh data/local/lm data/local/lang data/lang_test || exit 1;
# 7.4 Decoding with runtime
./tools/decode.sh --nj 16 \
    --beam 15.0 --lattice_beam 7.5 --max_active 7000 \
    --blank_skip_thresh 0.98 --ctc_weight 0.5 --rescoring_weight 1.0 \
    --fst_path data/lang_test/TLG.fst \
    --dict_path data/lang_test/words.txt \
    data/test/wav.scp data/test/text $dir/final.zip \
    data/lang_test/units.txt $dir/lm_with_runtime

Papers (`docs/papers.md`) Copy Section

Papers Copy Chapter

• WeNet: Production Oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit, accepted by InterSpeech 2021.
• WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit, accepted by InterSpeech 2022.

Pretrained Models (`docs/pretrained_models.md`) Copy Section

Pretrained Models in WeNet

Model Types Copy Chapter

We provide two types of pretrained model in WeNet to facilitate users with different requirements.

1. Checkpoint Model, with suffix .pt, the model trained and saved as checkpoint by WeNet python code, you can reproduce our published result with it, or you can use it as checkpoint to continue.

2. Runtime Model, with suffix .zip, you can directly use runtime model in our x86 or android runtime, the runtime model is export by Pytorch JIT on the checkpoint model. And the runtime models has been quantized to reduce the model size and network traffic.

Model License Copy Chapter

The pretrained model in WeNet follows the license of it's corresponding dataset.
For example, the pretrained model on LibriSpeech follows CC BY 4.0, since it is used as license of the LibriSpeech dataset, see http://openslr.org/12/.

Model List Copy Chapter

Here is a list of the pretrained models on different datasets. The model structure, model size, and download link are given.

Datasets	Languages	Checkpoint Model	Runtime Model	Contributor
aishell	CN	Conformer	Conformer
aishell2	CN	Conformer	Conformer
gigaspeech	EN	Conformer	Conformer
librispeech	EN	Conformer	Conformer
multi_cn	CN	Conformer	Conformer
wenetspeech	CN	Conformer	Conformer
paraformer	CN&EN	Model	NA	Alibaba
firered	CN&EN	Model	NA	FireRed
whisper-large-v3	multi-lingual	Model	NA	OpenAI
whisper-large-v3-turbo	multi-lingual	Model	NA	OpenAI

Production (`docs/production.rst`) Copy Section

Production Runtime

.. toctree::
:maxdepth: 1
:caption: Contents:

./lm.md
./context.md
./runtime.md
./jit_in_wenet.md

Python Package (`docs/python_package.md`) Copy Section

Python Package

Install Copy Chapter

pip install git+https://github.com/wenet-e2e/wenet.git

Development Install Copy Chapter

git clone https://github.com/wenet-e2e/wenet.git
cd wenet
pip install -e .

Command line Usage Copy Chapter

wenet --language chinese audio.wav

You can specify the following parameters.

• -l or --language: chinese/english are supported now.
• -m or --model_dir: your own model dir
• -g or --gpu: the device id of gpu, default value -1 represents for cpu.
• -t or --show_tokens_info: show the token level information such as timestamp, confidence, etc.
• --align: force align the input audio and transcript
• --label: the input label to align
• --paraformer: use the best Chinese model
• --device: specify the backend accelerator (cuda/npu/cpu)

Python Programming Usage Copy Chapter

import wenet

model = wenet.load_model('chinese')
# or model = wenet.load_model(model_dir='xxx')
result = model.transcribe('audio.wav')
print(result['text'])

GPU/Hardware Notes Copy Chapter

Measured on a Tesla T4 (16GB, Turing/sm_75), torch 2.4.0+cu121.

• GPU is not the default. --device (CLI) and device (wenet.load_model)
  both default to 'cpu'. On a GPU host, wenet -m <model> audio.wav still
  runs on CPU unless you pass --device cuda. There is no warning when this
  happens. Measured on paraformer: CUDA is 2.84x faster than CPU
  (0.205s vs 0.581s per utterance, cached model, T4).
• No dtype flag on the pip-installed CLI. The wenet console command
  (wenet/cli/transcribe.py) always runs fp32 and has no --dtype/--fp16
  flag. The training-recipe script wenet/bin/recognize.py does expose
  --dtype {fp16,fp32,bf16}, but that is a separate, more involved entry
  point (local recipe checkout, not the pip package).
• bf16 can be slower than fp32 on Turing-class GPUs (e.g. T4). T4 lacks
  bf16 tensor-core acceleration. On WeNet's own Conformer models (e.g.
  wenetspeech), casting to bf16 measured ~47% slower than fp32
  (46.6ms vs 31.7ms per forward pass). fp16 does not crash but shows no
  speedup at batch size 1 with short audio (launch-overhead bound), though it
  does cut VRAM by about 46%.
• use_sdpa is off by default in the shipped configs, and turning it on
  helps most on bf16. WeNet's own Conformer implementation supports a
  use_sdpa: true toggle (encoder_conf/decoder_conf) that switches
  attention from a manual matmul+softmax to
  torch.nn.functional.scaled_dot_product_attention, using the same
  checkpoint with no retraining. The train.yaml shipped with pretrained
  models (e.g. wenetspeech) does not set this key, so it loads as False.
  Enabling it measured 7-29% faster on a T4, with identical output text:
  fp32 31.7ms -> 29.4ms (~7%), fp16 32.2ms -> 30.0ms (~7%), bf16
  46.6ms -> 33.1ms (~29%, cutting most of the bf16 slowdown above). This
  toggle only applies to WeNet's own Conformer models; Paraformer (SANM
  attention) does not have a use_sdpa code path.
• int8 dynamic quantization is CPU-only. wenet/bin/export_jit.py uses
  torch.quantization.quantize_dynamic, which works on CPU but raises
  NotImplementedError: Could not run 'quantized::linear_dynamic' with arguments from the 'CUDA' backend if applied to a CUDA model — this is a
  PyTorch backend limitation, not a WeNet bug. (wenet/bin/export_onnx_cpu.py
  quantizes separately via onnxruntime.quantization.quantize_dynamic, a
  different, ONNX-graph-level API that isn't affected by this PyTorch
  limitation.) For GPU-side reduced precision, see
  wenet/bin/export_onnx_gpu.py --fp16 (ONNX export, a separate fp16-only
  workflow).

Reference (`docs/reference.rst`) Copy Section

Reference

.. toctree::
:maxdepth: 1
:caption: Contents:

./papers.md
./python_api/modules.rst

Requirements (`docs/requirements.txt`) Copy Section

jinja2
nbsphinx
sphinx
recommonmark
sphinx-markdown-tables
sphinx-rtd-theme
torch
torchaudio
typeguard

Runtime (`docs/runtime.md`) Copy Section

Runtime for WeNet

WeNet runtime uses Unified Two Pass (U2) framework for inference. U2 has the following advantages:

• Unified: U2 unified the streaming and non-streaming model in a simple way, and our runtime is also unified. Therefore you can easily balance the latency and accuracy by changing chunk_size (described in the following section).
• Accurate: U2 achieves better accuracy by CTC joint training.
• Fast: Our runtime uses attention rescoring based decoding method described in U2, which is much faster than a traditional autoregressive beam search.
• Other benefits: In practice, we find U2 is more stable on long-form speech than standard transformer which usually fails or degrades a lot on long-form speech; and we can easily get the word-level time stamps by CTC spikes in U2. Both of these aspects are favored for industry adoption.

Platforms Supported Copy Chapter

The WeNet runtime supports the following platforms.

• Server
  • x86
• Device
  • android

Architecture and Implementation Copy Chapter

Architecture Copy Section

The following picture shows how U2 works.

When input is not finished, the input frames $x_t$ are fed into the Shared Encoder module frame by frame to get the encoder output $e_t$, then $e_t$ is transformed by the CTC Activation module (typically, it's just a linear transform with a log_softmax) to get the CTC prob $y_t$ at current frame, and $y_t$ is further used by the CTC prefix beam search module to generate n-best results at current time $t$, and the best result is used as partial result of the U2 system.

When input is finished at time $T$, the n-best results from the CTC prefix beam search module and the encoder output $e_1, e_2, e_3, ..., e_T$ are fed into the Attention Decoder module, then the Attention Decoder module computes a score for every result. The result with the best score is selected as the final result of U2 system.

We can group $C$ continuous frames $x_t, x_{t+1}, x_{t+C}$ as one chunk for the Shared Encoder module, and $C$ is called chunk_size in the U2 framework. The chunk_size will affect the attention computation in the Shared Encoder module. When chunk_size is infinite, it is a non-streaming case. The system gives the best accuracy with infinite latency. When chunk_size is limited (typically less than 1s), it is a streaming case. The system has limited latency and also gives promising accuracy. So the developer can balance the accuracy and latency and setting a proper chunk_size.

Interface Design Copy Section

We use LibTorch to implement U2 runtime in WeNet, and we export several interfaces in PyTorch python code
by @torch.jit.export (see asr_model.py),
which are required and used in C++ runtime in torch_asr_model.cc.
Here we just list the interface and give a brief introduction.

Cache in Details Copy Section

For streaming scenario, the Shared Encoder module works in an incremental way. The current chunk computation requries the inputs and outputs of all the history chunks. We implement the incremental computation by using caches. Overall, two types of cache are used in our runtime.

• att_cache: the attention cache of the Shared Encoder(Conformer/Transformer) module.
• cnn_cache: the cnn cache of the Shared Encoder, which caches the left context for causal CNN computation in Conformer.

Please see encoder.py:forward_chunk() and torch_asr_model.cc for details of the caches.

In practice, CNN is also used for subsampling, we should handle the CNN cache in subsampling.
However, there are different CNN layers in subsampling with different left contexts, right contexts and strides, which makes it tircky to directly implement the CNN cache in subsampling.
In our implementation, we simply overlap the input to avoid subsampling CNN cache.
It is simple and straightforward with negligible additional cost since subsampling CNN only costs a very small fraction of the whole computation.
The following picture shows how it works, where the blue color is for the overlap part of current inputs and previous inputs.

References Copy Chapter

1. Sequence Modeling With CTC
2. First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs
3. Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition

Train (`docs/train.rst`) Copy Section

How to train models?

.. toctree::
:maxdepth: 1
:caption: Contents:

./tutorial_librispeech.md
./tutorial_aishell.md
./pretrained_models.md
./UIO.md

Tutorial Aishell (`docs/tutorial_aishell.md`) Copy Section

Tutorial on AIShell Copy Chapter

If you meet any problems when going through this tutorial, please feel free to ask in github issues. Thanks for any kind of feedback.

First Experiment Copy Section

We provide a recipe example/aishell/s0/run.sh on aishell-1 data.

The recipe is simple and we suggest you run each stage one by one manually and check the result to understand the whole process.

cd example/aishell/s0
bash run.sh --stage -1 --stop_stage -1
bash run.sh --stage 0 --stop_stage 0
bash run.sh --stage 1 --stop_stage 1
bash run.sh --stage 2 --stop_stage 2
bash run.sh --stage 3 --stop_stage 3
bash run.sh --stage 4 --stop_stage 4
bash run.sh --stage 5 --stop_stage 5
bash run.sh --stage 6 --stop_stage 6

You could also just run the whole script

bash run.sh --stage -1 --stop_stage 6

Stage -1: Download data

This stage downloads the aishell-1 data to the local path $data. This may take several hours. If you have already downloaded the data, please change the $data variable in run.sh and start from --stage 0.
Please set a absolute path for $data, e.g. /home/username/asr-data/aishell/

Stage 0: Prepare Training data

In this stage, local/aishell_data_prep.sh organizes the original aishell-1 data into two files:

• wav.scp each line records two tab-separated columns : wav_id and wav_path
• text each line records two tab-separated columns : wav_id and text_label

wav.scp

BAC009S0002W0122 /export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0122.wav
BAC009S0002W0123 /export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0123.wav
BAC009S0002W0124 /export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0124.wav
BAC009S0002W0125 /export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0125.wav
...

text

BAC009S0002W0122 而对楼市成交抑制作用最大的限购
BAC009S0002W0123 也成为地方政府的眼中钉
BAC009S0002W0124 自六月底呼和浩特市率先宣布取消限购后
BAC009S0002W0125 各地政府便纷纷跟进
...

If you want to train using your customized data, just organize the data into two files wav.scp and text, and start from stage 1.

Stage 1: Extract optinal cmvn features

example/aishell/s0 uses raw wav as input and and TorchAudio to extract the features just-in-time in dataloader. So in this step we just copy the training wav.scp and text file into the raw_wav/train/ dir.

tools/compute_cmvn_stats.py is used to extract global cmvn(cepstral mean and variance normalization) statistics. These statistics will be used to normalize the acoustic features. Setting cmvn=false will skip this step.

Stage 2: Generate label token dictionary

The dict is a map between label tokens (we use characters for Aishell-1) and
the integer indices.

An example dict is as follows

0
 1
一 2
丁 3
...
龚 4230
龟 4231
<sos/eos> 4232

• <blank> denotes the blank symbol for CTC.
• <unk> denotes the unknown token, any out-of-vocabulary tokens will be mapped into it.
• <sos/eos> denotes start-of-speech and end-of-speech symbols for attention based encoder decoder training, and they shares the same id.

Stage 3: Prepare WeNet data format

This stage generates the WeNet required format file data.list. Each line in data.list is in json format which contains the following fields.

1. key: key of the utterance
2. wav: audio file path of the utterance
3. txt: normalized transcription of the utterance, the transcription will be tokenized to the model units on-the-fly at the training stage.

Here is an example of the data.list, and please see the generated training feature file in data/train/data.list.

{"key": "BAC009S0002W0122", "wav": "/export/data/asr-data/OpenSLR/33//data_aishell/wav/train/S0002/BAC009S0002W0122.wav", "txt": "而对楼市成交抑制作用最大的限购"}
{"key": "BAC009S0002W0123", "wav": "/export/data/asr-data/OpenSLR/33//data_aishell/wav/train/S0002/BAC009S0002W0123.wav", "txt": "也成为地方政府的眼中钉"}
{"key": "BAC009S0002W0124", "wav": "/export/data/asr-data/OpenSLR/33//data_aishell/wav/train/S0002/BAC009S0002W0124.wav", "txt": "自六月底呼和浩特市率先宣布取消限购后"}

We aslo design another format for data.list named shard which is for big data training.
Please see gigaspeech(10k hours) or
wenetspeech(10k hours)
for how to use shard style data.list if you want to apply WeNet on big data set(more than 5k).

Stage 4: Neural Network training

The NN model is trained in this step.

• Multi-GPU mode

If using DDP mode for multi-GPU, we suggest using dist_backend="nccl". If the NCCL does not work, try using gloo or use torch==1.6.0
Set the GPU ids in CUDA_VISIBLE_DEVICES. For example, set export CUDA_VISIBLE_DEVICES="0,1,2,3,6,7" to use card 0,1,2,3,6,7.

• Resume training

If your experiment is terminated after running several epochs for some reasons (e.g. the GPU is accidentally used by other people and is out-of-memory ), you could continue the training from a checkpoint model. Just find out the finished epoch in exp/your_exp/, set checkpoint=exp/your_exp/$n.pt and run the run.sh --stage 4. Then the training will continue from the $n+1.pt

• Config

The config of neural network structure, optimization parameter, loss parameters, and dataset can be set in a YAML format file.

In conf/, we provide several models like transformer and conformer. see conf/train_conformer.yaml for reference.

• Use Tensorboard

The training takes several hours. The actual time depends on the number and type of your GPU cards. In an 8-card 2080 Ti machine, it takes about less than one day for 50 epochs.
You could use tensorboard to monitor the loss.

tensorboard --logdir tensorboard/$your_exp_name/ --port 12598 --bind_all

Stage 5: Recognize wav using the trained model

This stage shows how to recognize a set of wavs into texts. It also shows how to do the model averaging.

• Average model

If ${average_checkpoint} is set to true, the best ${average_num} models on cross validation set will be averaged to generate a boosted model and used for recognition.

• Decoding

Recognition is also called decoding or inference. The function of the NN will be applied on the input acoustic feature sequence to output a sequence of text.

Four decoding methods are provided in WeNet:

• ctc_greedy_search : encoder + CTC greedy search
• ctc_prefix_beam_search : encoder + CTC prefix beam search
• attention : encoder + attention-based decoder decoding
• attention_rescoring : rescoring the ctc candidates from ctc prefix beam search with encoder output on attention-based decoder.

In general, attention_rescoring is the best method. Please see U2 paper for the details of these algorithms.

--beam_size is a tunable parameter, a large beam size may get better results but also cause higher computation cost.

--batch_size can be greater than 1 for "ctc_greedy_search" and "attention" decoding mode, and must be 1 for "ctc_prefix_beam_search" and "attention_rescoring" decoding mode.

• WER evaluation

tools/compute-wer.py will calculate the word (or char) error rate of the result. If you run the recipe without any change, you may get WER ~= 5%.

Stage 6: Export the trained model

wenet/bin/export_jit.py will export the trained model using Libtorch. The exported model files can be easily used for inference in other programming languages such as C++.

Tutorial Librispeech (`docs/tutorial_librispeech.md`) Copy Section

Tutorial on LibriSpeech Copy Chapter

If you meet any problems when going through this tutorial, please feel free to ask in github issues. Thanks for any kind of feedback.

First Experiment Copy Section

We provide a recipe example/librispeech/s0/run.sh on librispeech data.

The recipe is simple and we suggest you run each stage one by one manually and check the result to understand the whole process.

cd example/librispeech/s0
bash run.sh --stage -1 --stop_stage -1
bash run.sh --stage 0 --stop_stage 0
bash run.sh --stage 1 --stop_stage 1
bash run.sh --stage 2 --stop_stage 2
bash run.sh --stage 3 --stop_stage 3
bash run.sh --stage 4 --stop_stage 4
bash run.sh --stage 5 --stop_stage 5
bash run.sh --stage 6 --stop_stage 6
bash run.sh --stage 7 --stop_stage 7

You could also just run the whole script

bash run.sh --stage -1 --stop_stage 7

Stage -1: Download data

data_url=www.openslr.org/resources/12
datadir=/export/data/en-asr-data/OpenSLR/
if [ ${stage} -le -1 ] && [ ${stop_stage} -ge -1 ]; then
  echo "stage -1: Data Download"
  for part in dev-clean test-clean dev-other test-other train-clean-100 train-clean-360 train-other-500; do
    local/download_and_untar.sh ${datadir} ${data_url} ${part}
  done
fi

This stage downloads the librispeech data to the local path $data. This may take several hours. If you have already downloaded the data, please change the $data variable in run.sh and start from --stage 0.

Stage 0: Prepare Training data

wave_data=data
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
  for part in dev-clean test-clean dev-other test-other train-clean-100 train-clean-360 train-other-500; do
    # use underscore-separated names in data directories.
    local/data_prep_torchaudio.sh ${datadir}/LibriSpeech/${part} $wave_data/${part//-/_}
  done
fi

In this stage, local/data_prep_torchaudio.sh organizes the original data into two files:

• wav.scp each line records two tab-separated columns : wav_id and wav_path
• text each line records two tab-separated columns : wav_id and text_label

wav.scp

1867-154075-0014 /export/data/en-asr-data/OpenSLR//LibriSpeech/train-clean-100/1867/154075/1867-154075-0014.flac
1970-26100-0022 /export/data/en-asr-data/OpenSLR//LibriSpeech/train-clean-100/1970/26100/1970-26100-0022.flac
...

text

1867-154075-0014 YOU SHOW HIM THAT IT IS POSSIBLE
1970-26100-0022 DID YOU SEE HIM AT THAT TIME
...

If you want to train using your customized data, just organize the data into two files wav.scp and text, and start from stage 1.

Stage 1: Extract optinal cmvn features

if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
  ### Task dependent. You have to design training and dev sets by yourself.
  ### But you can utilize Kaldi recipes in most cases
  echo "stage 1: Feature Generation"
  mkdir -p $wave_data/train_960
  # merge total training data
  for set in train_clean_100 train_clean_360 train_other_500; do
    for f in `ls $wave_data/$set`; do
      cat $wave_data/$set/$f >> $wave_data/train_960/$f
    done
  done
  mkdir -p $wave_data/dev
  # merge total dev data
  for set in dev_clean dev_other; do
    for f in `ls $wave_data/$set`; do
      cat $wave_data/$set/$f >> $wave_data/dev/$f
    done
  done

  tools/compute_cmvn_stats.py --num_workers 16 --train_config $train_config \
    --in_scp $wave_data/$train_set/wav.scp \
    --out_cmvn $wave_data/$train_set/global_cmvn

fi

The librispeech corpus contains 3 subsets for training, namely train_clean_100, train_clean_360, and train_other_500,
so we first merge them to get our final training data.

tools/compute_cmvn_stats.py is used to extract global cmvn(cepstral mean and variance normalization) statistics. These statistics will be used to normalize the acoustic features. Setting cmvn=false will skip this step.

Stage 2: Generate label token dictionary

dict=$wave_data/lang_char/${train_set}_${bpemode}${nbpe}_units.txt
bpemodel=$wave_data/lang_char/${train_set}_${bpemode}${nbpe}
echo "dictionary: ${dict}"
if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
  ### Task dependent. You have to check non-linguistic symbols used in the corpus.
  echo "stage 2: Dictionary and Json Data Preparation"
  mkdir -p data/lang_char/

  echo " 0" > ${dict} # 0 will be used for "blank" in CTC
  echo " 1" >> ${dict} #  must be 1

  # we borrowed these code and scripts which are related bpe from ESPnet.
  cut -f 2- -d" " $wave_data/${train_set}/text > $wave_data/lang_char/input.txt
  tools/spm_train --input=$wave_data/lang_char/input.txt --vocab_size=${nbpe} --model_type=${bpemode} --model_prefix=${bpemodel} --input_sentence_size=100000000
  tools/spm_encode --model=${bpemodel}.model --output_format=piece < $wave_data/lang_char/input.txt | tr ' ' '\n' | sort | uniq | awk '{print $0 " " NR+1}' >> ${dict}
  num_token=$(cat $dict | wc -l)
  echo "<sos/eos> $num_token" >> $dict # <eos>
  wc -l ${dict}
fi

The model unit of English e2e speech recognition system could be char or BPE(byte-pair-encoding).
Typically, BPE shows better result. So here we use BPE as model unit,
and the BPE is trained by sentencepiece tool on the librispeech training data.

The model unit is defined as a dict in WeNet, which maps the a BPE into integer index.
The librispeech dict is like:

0
 1
' 2
▁ 3
A 4
▁A 5
AB 6
▁AB 7
▁YOU 4995
▁YOUNG 4996
▁YOUR 4997
▁YOUTH 4998
Z 4999
ZZ 5000
<sos/eos> 5001

• <blank> denotes the blank symbol for CTC.
• <unk> denotes the unknown token, any out-of-vocabulary tokens will be mapped into it.
• <sos/eos> denotes start-of-speech and end-of-speech symbols for attention based encoder decoder training, and they shares the same id.

Stage 3: Prepare WeNet data format

if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then
  # Prepare wenet required data
  echo "Prepare data, prepare required format"
  for x in dev ${recog_set} $train_set ; do
    tools/make_raw_list.py $wave_data/$x/wav.scp $wave_data/$x/text \
        $wave_data/$x/data.list
  done

fi

This stage generates the WeNet required format file data.list. Each line in data.list is in json format which contains the following fields.

1. key: key of the utterance
2. wav: audio file path of the utterance
3. txt: normalized transcription of the utterance, the transcription will be tokenized to the model units on-the-fly at the training stage.

Here is an example of the data.list, and please see the generated training feature file in data/train/data.list.

{"key": "1455-134435-0000", "wav": "/mnt/nfs/ptm1/open-data/LibriSpeech/train-clean-100/1455/134435/1455-134435-0000.flac", "txt": "THE GIRL WHO CAME INTO THE WORLD ON THAT NIGHT WHEN JESSE RAN THROUGH THE FIELDS CRYING TO GOD THAT HE BE GIVEN A SON HAD GROWN TO WOMANHOOD ON THE FARM"}
{"key": "1455-134435-0001", "wav": "/mnt/nfs/ptm1/open-data/LibriSpeech/train-clean-100/1455/134435/1455-134435-0001.flac", "txt": "AND WHEN NOT ANGRY SHE WAS OFTEN MOROSE AND SILENT IN WINESBURG IT WAS SAID THAT SHE DRANK HER HUSBAND THE BANKER"}
{"key": "1455-134435-0002", "wav": "/mnt/nfs/ptm1/open-data/LibriSpeech/train-clean-100/1455/134435/1455-134435-0002.flac", "txt": "BUT LOUISE COULD NOT BE MADE HAPPY SHE FLEW INTO HALF INSANE FITS OF TEMPER DURING WHICH SHE WAS SOMETIMES SILENT SOMETIMES NOISY AND QUARRELSOME SHE SWORE AND CRIED OUT IN HER ANGER SHE GOT A KNIFE FROM THE KITCHEN AND THREATENED HER HUSBAND'S LIFE"}

We aslo design another format for data.list named shard which is for big data training.
Please see gigaspeech(10k hours) or
wenetspeech(10k hours)
for how to use shard style data.list if you want to apply WeNet on big data set(more than 5k).

Stage 4: Neural Network training

if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then
  # Training
  mkdir -p $dir
  INIT_FILE=$dir/ddp_init
  rm -f $INIT_FILE # delete old one before starting
  init_method=file://$(readlink -f $INIT_FILE)
  echo "$0: init method is $init_method"
  num_gpus=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
  # Use "nccl" if it works, otherwise use "gloo"
  dist_backend="nccl"
  cmvn_opts=
  $cmvn && cmvn_opts="--cmvn $wave_data/${train_set}/global_cmvn"
  # train.py will write $train_config to $dir/train.yaml with model input
  # and output dimension, train.yaml will be used for inference or model
  # export later
  for ((i = 0; i < $num_gpus; ++i)); do
  {
    gpu_id=$(echo $CUDA_VISIBLE_DEVICES | cut -d',' -f$[$i+1])
    python wenet/bin/train.py --gpu $gpu_id \
      --config $train_config \
      --data_type raw \
      --symbol_table $dict \
      --train_data $wave_data/$train_set/data.list \
      --cv_data $wave_data/dev/data.list \
      ${checkpoint:+--checkpoint $checkpoint} \
      --model_dir $dir \
      --ddp.init_method $init_method \
      --ddp.world_size $num_gpus \
      --ddp.rank $i \
      --ddp.dist_backend $dist_backend \
      --num_workers 1 \
      $cmvn_opts \
      --pin_memory
  } &
  done
  wait
fi

The NN model is trained in this step.

• Multi-GPU mode

If using DDP mode for multi-GPU, we suggest using dist_backend="nccl". If the NCCL does not work, try using gloo or use torch==1.6.0
Set the GPU ids in CUDA_VISIBLE_DEVICES. For example, set export CUDA_VISIBLE_DEVICES="0,1,2,3,6,7" to use card 0,1,2,3,6,7.

• Resume training

If your experiment is terminated after running several epochs for some reasons (e.g. the GPU is accidentally used by other people and is out-of-memory ), you could continue the training from a checkpoint model. Just find out the finished epoch in exp/your_exp/, set checkpoint=exp/your_exp/$n.pt and run the run.sh --stage 4. Then the training will continue from the $n+1.pt

• Config

The config of neural network structure, optimization parameter, loss parameters, and dataset can be set in a YAML format file.

In conf/, we provide several models like transformer and conformer. see conf/train_conformer.yaml for reference.

• Use Tensorboard

The training takes several hours. The actual time depends on the number and type of your GPU cards. In an 8-card 2080 Ti machine, it takes about less than one day for 50 epochs.
You could use tensorboard to monitor the loss.

tensorboard --logdir tensorboard/$your_exp_name/ --port 12598 --bind_all

Stage 5: Recognize wav using the trained model

/* Detailed source-code truncated for AI context efficiency. */

This stage shows how to recognize a set of wavs into texts. It also shows how to do the model averaging.

• Average model

If ${average_checkpoint} is set to true, the best ${average_num} models on cross validation set will be averaged to generate a boosted model and used for recognition.

• Decoding

Recognition is also called decoding or inference. The function of the NN will be applied on the input acoustic feature sequence to output a sequence of text.

Four decoding methods are provided in WeNet:

• ctc_greedy_search : encoder + CTC greedy search
• ctc_prefix_beam_search : encoder + CTC prefix beam search
• attention : encoder + attention-based decoder decoding
• attention_rescoring : rescoring the ctc candidates from ctc prefix beam search with encoder output on attention-based decoder.

In general, attention_rescoring is the best method. Please see U2 paper for the details of these algorithms.

--beam_size is a tunable parameter, a large beam size may get better results but also cause higher computation cost.

--batch_size can be greater than 1 for "ctc_greedy_search" and "attention" decoding mode, and must be 1 for "ctc_prefix_beam_search" and "attention_rescoring" decoding mode.

• WER evaluation

tools/compute-wer.py will calculate the word (or char) error rate of the result.

Stage 6(Optional): Export the trained model

if [ ${stage} -le 6 ] && [ ${stop_stage} -ge 6 ]; then
  # Export the best model you want
  python wenet/bin/export_jit.py \
    --config $dir/train.yaml \
    --checkpoint $dir/avg_${average_num}.pt \
    --output_file $dir/final.zip
fi

wenet/bin/export_jit.py will export the trained model using Libtorch.
The exported model files can be easily used for C++ inference in our runtime.
It is required if you want to integrate language model(LM), as shown in Stage 7.

Stage 7(Optional): Add LM and test it with runtime

/* Detailed source-code truncated for AI context efficiency. */

LM is only supported in runtime, you have to build the runtime as shown in Installation,
and please refer LM for WeNet for the details of LM design.

--- METRICS ---

• Files Extracted: 60
• Estimated Token Budget: ~51430 tokens
• Canonical Reference: https://codewiki.google/github.com/wenet-e2e/wenet