# Technical Documentation: wenet-e2e/wenet

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/wenet-e2e/wenet)

### Overview (`README.md`)

# WeNet

[](https://opensource.org/licenses/Apache-2.0)
[](https://github.com/wenet-e2e/wenet)
[](#discussion--communication)

[**Roadmap**](https://github.com/wenet-e2e/wenet/issues/1683)
| [**Docs**](https://wenet-e2e.github.io/wenet)
| [**Papers**](https://wenet-e2e.github.io/wenet/papers.html)
| [**Runtime**](https://github.com/wenet-e2e/wenet/tree/main/runtime)
| [**Pretrained Models**](docs/pretrained_models.md)
| [**HuggingFace**](https://huggingface.co/spaces/wenet/wenet_demo)
| [**Ask WeNet Guru**](https://gurubase.io/g/wenet)

**We** share **Net** together.

## Highlights

* **Production first and production ready**: The core design principle, WeNet provides full stack production solutions for speech recognition.
* **Accurate**: WeNet achieves SOTA results on a lot of public speech datasets.
* **Light weight**: WeNet is easy to install, easy to use, well designed, and well documented.


## Install

### Install python package

``` sh
pip install git+https://github.com/wenet-e2e/wenet.git
```

This pulls in `torch>=1.13.0` with no upper bound, so pip may install a torch
build newer than your installed GPU driver supports. If you plan to use a GPU,
verify it after install:

``` sh
python -c "import torch; print(torch.cuda.is_available())"
```

If this prints `False` on a machine with a GPU, torch silently falls back to
CPU with no error. Reinstall a torch build matching your driver's CUDA
version, e.g.:

``` sh
pip install torch==2.4.0+cu121 torchaudio==2.4.0+cu121 --index-url https://download.pytorch.org/whl/cu121 --force-reinstall
```

**Command-line usage** (use `-h` for parameters):

``` sh
wenet -m paraformer audio.wav
```

You can set `-m` with `paraformer` or `firered` or `wenetspeech` for chinese,
and set it to `whisper-large-v3` or `whisper-large-v3-turbo` for english.

Note: this runs on **CPU by default** (`--device` defaults to `cpu`), even if
a GPU is available. Pass `--device cuda` explicitly to use the GPU (Python API:
`wenet.load_model(model_name, device='cuda')`). See
[python usage](docs/python_package.md#gpuhardware-notes) for GPU/Turing (e.g.
Tesla T4) specific notes on dtype and attention-backend options.


**Python programming usage**:

``` python
import wenet

model = wenet.load_model('paraformer')
result = model.transcribe('audio.wav')
print(result.text)
```

Please refer [python usage](docs/python_package.md) for more command line and python programming usage.

### Install for training & deployment

- Clone the repo
``` sh
git clone https://github.com/wenet-e2e/wenet.git
```

- Install Conda: please see https://docs.conda.io/en/latest/miniconda.html
- Create Conda env:

``` sh
conda create -n wenet python=3.10
conda activate wenet
conda install conda-forge::sox
```

- Install CUDA: please follow this [link](https://icefall.readthedocs.io/en/latest/installation/index.html#id1), It's recommended to install CUDA 12.1
- Install torch and torchaudio, It's recomended to use 2.2.2+cu121:

``` sh
pip install torch==2.2.2+cu121 torchaudio==2.2.2+cu121 -f https://download.pytorch.org/whl/torch_stable.html
```

<details><summary><b>For Ascend NPU users:</b></summary>

- Install CANN: please follow this [link](https://ascend.github.io/docs/sources/ascend/quick_install.html) to install CANN toolkit and kernels.

- Install WeNet with torch-npu dependencies:

``` sh
pip install -e .[torch-npu]
```

- Related version control table:

| Requirement  |      Minimum     | Recommend   |
| ------------ | ---------------- | ----------- |
| CANN         | 8.0.RC2.alpha003 | latest      |
| torch        | 2.1.0            | 2.2.0       |
| torch-npu    | 2.1.0            | 2.2.0       |
| torchaudio   | 2.1.0            | 2.2.0       |
| deepspeed    | 0.13.2           | latest      |

</details>

- Install other python packages

``` sh
pip install -r requirements.txt
pre-commit install  # for clean and tidy code
```

- Frequently Asked Questions (FAQs)

``` sh
# If you encounter sox compatibility issues
RuntimeError: set_buffer_size requires sox extension which is not available.
# ubuntu
sudo apt-get install sox libsox-dev
# centos
sudo yum install sox sox-devel
# conda env
conda install  conda-forge::sox
```

**Build for deployment**

Optionally, if you want to use x86 runtime or language model(LM),
you have to build the runtime as follows. Otherwise, you can just ignore this step.

``` sh
# runtime build requires cmake 3.14 or above
cd runtime/libtorch
mkdir build && cd build && cmake -DGRAPH_TOOLS=ON .. && cmake --build .
```

Please see [doc](https://github.com/wenet-e2e/wenet/tree/main/runtime) for building
runtime on more platforms and OS.


## Discussion & Communication

You can directly discuss on [Github Issues](https://github.com/wenet-e2e/wenet/issues).

For Chinese users, you can also scan the QR code on the left to follow our official account of WeNet.
We created a WeChat group for better discussion and quicker response.
Please scan the personal QR code on the right, and the guy is responsible for inviting you to the chat group.

| <img src="https://github.com/robin1001/qr/blob/master/wenet.jpeg" width="250px"> |<img src="https://github.com/robin1001/qr/blob/master/chengdong.jpg" width="250px">| <img src="https://github.com/robin1001/qr/blob/master/binbin.jpeg" width="250px"> |
| ---- | ---- | ---- |


## Acknowledge

1. We borrowed a lot of code from [ESPnet](https://github.com/espnet/espnet) for transformer based modeling.
2. We borrowed a lot of code from [Kaldi](http://kaldi-asr.org/) for WFST based decoding for LM integration.
3. We referred [EESEN](https://github.com/srvk/eesen) for building TLG based graph for LM integration.
4. We referred to [OpenTransformer](https://github.com/ZhengkunTian/OpenTransformer/) for python batch inference of e2e models.

## Citations

``` bibtex
@inproceedings{yao2021wenet,
title={WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit},
author={Yao, Zhuoyuan and Wu, Di and Wang, Xiong and Zhang, Binbin and Yu, Fan and Yang, Chao and Peng, Zhendong and Chen, Xiaoyu and Xie, Lei and Lei, Xin},
  booktitle={Proc. Interspeech},
  year={2021},
  address={Brno, Czech Republic },
  organization={IEEE}
}

@article{zhang2022wenet,
  title={WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit},
  author={Zhang, Binbin and Wu, Di and Peng, Zhendong and Song, Xingchen and Yao, Zhuoyuan and Lv, Hang and Xie, Lei and Yang, Chao and Pan, Fuping and Niu, Jianwei},
  journal={arXiv preprint arXiv:2203.15455},
  year={2022}
}
```

---

### Overview (`runtime/README.md`)

# Runtime on WeNet

This is the runtime of WeNet.

We are going to support the following platforms:

1. Various deep learning inference engines, such as LibTorch, ONNX, OpenVINO, TVM, and so on.
2. Various OS, such as android, iOS, Harmony, and so on.
3. Various AI chips, such as GPU, Horzion BPU, and so on.
4. Various hardware platforms, such as Raspberry Pi.
5. Various language binding, such as python and go.

Feel free to volunteer yourself if you are interested in trying out some items(they do not have to be on the list).

## Introduction

Here is a brief summary of all platforms and OSs. please note the corresponding working `OS` and `inference engine`.

| runtime         | OS                  | inference engine     | Description                                                                                      |
|-----------------|---------------------|----------------------|--------------------------------------------------------------------------------------------------|
| core            | /                   | /                    | common core code of all runtime                                                                  |
| android         | android             | libtorch             | android demo, [English demo](https://www.youtube.com/shorts/viEnvmZf03s ), [Chinese demo](TODO)  |
| bingding/python | linux, windows, mac | libtorch             | python binding of wenet, mac M1/M2 are is not supported now.                                     |
| gpu             | linux               | onnxruntime/tensorrt | GPU inference with NV's Triton and TensorRT                                                      |
| horizonbpu      | linux               | bpu runtime          | Horizon BPU runtime                                                                              |
| ios             | ios                 | libtorch             | ios demo, [link](TODO)                                                                           |
| kunlun          | linux               | xpu runtime          | Kunlun XPU runtime                                                                               |
| libtorch        | linux, windows, mac | libtorch             | c++ build with libtorch                                                                          |
| ipex            | linux               | libtorch + ipex      | c++ build with libtorch and ipex optimization                                                                         |
| onnxrutnime     | linux, windows, mac | onnxruntime          | c++ build with onnxruntime                                                                       |
| openvino        | linux, windows, mac | openvino             | c++ build with openvino                                                                          |
| raspberrypi     | linux               | onnxruntime          | c++ build on raspberrypi with onnxruntime                                                        |
| web             | linux, windows, mac | libtorch             | web demo with gradio and python binding, [link]()                                                |

---

### Overview (`wenet/README.md`)

# Module Introduction

Here is a brief introduction of each module(directory).

* `bin`: training and recognition binaries
* `dataset`: IO design
* `utils`: common utils
* `transformer`: the core of `WeNet`, in which the standard transformer/conformer is implemented. It contains the common blocks(backbone) of speech transformers.
  * transformer/attention.py: Standard multi head attention
  * transformer/embedding.py: Standard position encoding
  * transformer/positionwise_feed_forward.py: Standard feed forward in transformer
  * transformer/convolution.py: ConvolutionModule in Conformer model
  * transformer/subsampling.py: Subsampling implementation for speech task
* `transducer`: transducer implementation
* `squeezeformer`: squeezeformer implementation, please refer [paper](https://arxiv.org/pdf/2206.00888.pdf)
* `efficient_conformer`: efficient conformer implementation, please refer [paper](https://arxiv.org/pdf/2109.01163.pdf)
* `paraformer`: paraformer implementation, please refer [paper](https://arxiv.org/pdf/1905.11235.pdf)
   * `paraformer/cif.py`: Continuous Integrate-and-Fire implemented, please refer [paper](https://arxiv.org/pdf/1905.11235.pdf)
* `branchformer`: branchformer implementation, please refer [paper](https://arxiv.org/abs/2207.02971)
* `whisper`: whisper implementation, please refer [paper](https://arxiv.org/abs/2212.04356)
* `ssl`: Self-supervised speech model implementation. e.g. wav2vec2, bestrq, w2vbert.
* `ctl_model`: Enhancing the Unified Streaming and Non-streaming Model with  with Contrastive Learning implementation [paper](https://arxiv.org/abs/2306.00755)

`transducer`, `squeezeformer`, `efficient_conformer`, `branchformer` and `cif` are all based on `transformer`,
they resue a lot of the common blocks of `tranformer`.

**If you want to contribute your own x-former, please reuse the current code as much as possible**.

---

### Overview (`runtime/android/README.md`)

# WeNet On-device ASR Android Demo

This Android demo shows we can run on-device streaming ASR with WeNet. You can download our prebuilt APK or build your APK from source code.

## Prebuilt APK

* [Chinese ASR Demo APK, with model trained on AIShell data](http://mobvoi-speech-public.ufile.ucloud.cn/public/wenet/aishell/20210202_app.apk)
* [English ASR Demo APK, with model trained on GigaSpeech data](http://mobvoi-speech-public.ufile.ucloud.cn/public/wenet/gigaspeech/20210823_app.apk)

## Build your APK from source code

### 1) Build model

You can use our pretrained model (click the following link to download):

[中文(WenetSpeech)](https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/wenetspeech/wenetspeech_u2pp_conformer_libtorch_quant.tar.gz)
| [English(GigaSpeech)](https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/gigaspeech/gigaspeech_u2pp_conformer_libtorch_quant.tar.gz)

Or you can train your own model using WeNet training pipeline on your data.

### 2) Build APK

When your model is ready, put `final.zip` and `units.txt` into Android assets (`app/src/main/assets`) folder,
then just build and run the APK. Here is a gif demo, which shows how our on-device streaming e2e ASR runs with low latency.
Please note the wifi and data has been disabled in the demo so there is no network connection ^\_^.



## Compute the RTF

Step 1, connect your Android phone, and use `adb push` command to push your model, wav scp, and waves to the sdcard.

Step 2, build the binary and the APK with Android Studio directly, or with the commands as follows:

``` sh
cd runtime/android
./gradlew build
```

Step 3, push your binary and the dynamic library to `/data/local/tmp` as follows:

``` sh
adb push app/.cxx/cmake/release/arm64-v8a/decoder_main /data/local/tmp
adb push app/build/pytorch_android-1.10.0.aar/jni/arm64-v8a/* /data/local/tmp
```

Step 4, change to the directory `/data/local/tmp` of your phone, and export the library path by:

``` sh
adb shell
cd /data/local/tmp
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:.
```

Step 5, execute the same command as the [x86 demo](../../../libtorch) to run the binary to decode and compute the RTF.

---

### Overview (`runtime/gpu/README.md`)

## Introduction
The below example shows how to deploy WeNet offline and online ASR models on GPUs.

## Instructions
* Step 1. Convert your model/pretrained model to onnx models. For example:

```bash
conda activate wenet
pip install onnxruntime-gpu onnxmltools
cd wenet/examples/aishell2 && . ./path.sh
model_dir=<absolute path to>/20211025_conformer_exp
onnx_model_dir=<absolute path>
mkdir $onnx_model_dir
python3 wenet/bin/export_onnx_gpu.py --config=$model_dir/train.yaml --checkpoint=$model_dir/final.pt --cmvn_file=$model_dir/global_cmvn --ctc_weight=0.5 --output_onnx_dir=$onnx_model_dir --fp16
cp $model_dir/words.txt $model_dir/train.yaml $onnx_model_dir/
```

If you want to export streaming model (u2/u2++) for streaming inference (inference by chunks) instead of offline inference (inference by audio segments/utterance), you should add `--streaming` option:

```
python3 wenet/bin/export_onnx_gpu.py --config=$model_dir/train.yaml --checkpoint=$model_dir/final.pt --cmvn_file=$model_dir/global_cmvn  --ctc_weight=0.1 --reverse_weight=0.4 --output_onnx_dir=$onnx_model_dir --fp16 --streaming
```

Congratulations! You've successully exported the onnx models and now you are able to deploy them. Please also ensure you have set up [NGC](https://catalog.ngc.nvidia.com/) account before the next step!

* Step 2. Build server docker and start server on one gpu:

```
docker build . -f Dockerfile/Dockerfile.server -t wenet_server:latest --network host
# offline model
docker run --gpus '"device=0"' -it -v $PWD/model_repo:/ws/model_repo -v $onnx_model_dir:/ws/onnx_model -p 8000:8000 -p 8001:8001 -p 8002:8002 --shm-size=1g --ulimit memlock=-1  wenet_server:latest /workspace/scripts/convert_start_server.sh
# streaming model
docker run --gpus '"device=0"' -it -v $PWD/model_repo_stateful:/ws/model_repo -v $onnx_model_dir:/ws/onnx_model -p 8000:8000 -p 8001:8001 -p 8002:8002 --shm-size=1g --ulimit memlock=-1  wenet_server:latest /workspace/scripts/convert_start_server.sh
```
Whenever there's something wrong starting the server, you may check the config.pbtxt files in every model in `model_repo` or `model_repo_stateful` to find if the settings are right.

* Step 3. Start client

```
docker build . -f Dockerfile/Dockerfile.client -t wenet_client:latest --network host

AUDIO_DATA=<path to your wav data>
docker run -ti --net host --name wenet_client -v $PWD/client:/ws/client -v $AUDIO_DATA:/ws/test_data wenet_client:latest
# In docker
# offline model test
cd /ws/client
# test one wav file
python3 client.py --audio_file=/ws/test_data/mid.wav --url=localhost:8001

# test a list of wav files & cer
python3 client.py --wavscp=/ws/dataset/test/wav.scp --data_dir=/ws/dataset/test/ --trans=/ws/dataset/test/text
```

Similarly, if your model is exported with `--streaming`, you should add this option when calling your streaming model.
For example,

```
python3 client.py --wavscp=/ws/test_data/data_aishell2/test/wav.scp --data_dir=/ws/test_data/data_aishell2/test/ --trans=/ws/test_data/data_aishell2/test/trans.txt --model_name=streaming_wenet --streaming
```

<img src="test.gif" alt="test" width="500"/>

## Precision Impact
Some of you may be worried about whether fp16 will affect the final accuracy. We did several experiments and we may find the accuracy is acceptable.
|Model | Dataset | Precision | CER |
|------------|-----------|-----------|----------|
|Aishell2-U2++ Conformer|Aishell2-TEST|FP16| 5.39%|
|Aishell2-U2++ Conformer|Aishell2-TEST|FP32| 5.38%|
|Wenetspeech Conformer| Wenetspeech-DEV|FP16| 8.61%|
|Wenetspeech Conformer| Wenetspeech-DEV|FP32| 8.61%|
|Wenetspeech Conformer | Wenetspeech-TestNet|FP16|9.07%|
|Wenetspeech Conformer| Wenetspeech-TestNet|FP32|9.06%|
|Wenetspeech Concofmer| Wenetspeech-TestMeeting|FP16|15.72%|
|Wenetspeech Conformer| Wenetspeech-TestMeeting|FP32|15.70%|


## Perf Test
We use the below command to do our testing and we run the below command several times to warm up:

```
cd /ws/client
# generate the test data, input to our feature extractor
# offline model
python3 generate_perf_input.py --audio_file=input.wav
# offline_input.json generated
perf_analyzer -m attention_rescoring -b 1 -p 20000 --concurrency-range 100:200:50 -i gRPC --input-data=offline_input.json  -u localhost:8001

# streaming input
python3 generate_perf_input.py --audio_file=input.wav --streaming
# online_input.json generated
perf_analyzer -u "localhost:8001" -i gRPC --streaming --input-data=online_input.json -m streaming_wenet -b 1 --concurrency-range 100:200:50
```
Where:
- `input.wav` the input test audio, we tested 5 seconds, 8 seconds, 10 seconds audio;
- `-m` option indicates the name of the served model;
- `-p` option is the mearsurement window, which indicates in what time duration to collect the metrics;
- `-v` option turns on the verbose model;
- `-i` option is for choosing the networking protocol, you can choose `HTTP` or `gRPC` here;
- `-u` option sets the url of the service in the form of `<IP Adrress>:<Port>`, but notice that port `8000` corresponds to HTTP protocol while port `8001` corresponds to gRPC protocol;
- `-b` option indicates the batch size of the input requests used fo testing; since we simulate individual users sending requests, we set batch size here to `1`;
- `--input-data` option points to the path of the json file containing the real input data
- `--concurrency-range` option is an important one, it indicates the concurrency of the requests which defines the pressure we will give to the server.
- You can also set `-f` option to set the path of testing result file;
- You can also set `--max-threads` option to set the number of threads used to send test request, it should be set to the number of CPU cores in your test machine.

### Tested ENV
* NVIDIA DRIVER: 470.57.02
* GPU: V100 & T4 & A30
* CUDA: 11.4
* Triton Inference Server: 22.03

### Offline Model Perf
Here are the wenetspeech conformer model and aishell2 u2++ perf on T4.
* Aishell2, FP16, onnx, [U2++ Conformer](http://mobvoi-speech-public.ufile.ucloud.cn/public/wenet/aishell2/20210618_u2pp_conformer_exp.tar.gz)

|input length|Concurrency|RTF / GPU  |Throughput|Latency_p50 (ms)|Latency_p90 (ms)|Latency_p95 (ms)|Latency_p99 (ms)|
|------------|-----------|-----------|----------|----------------|----------------|----------------|----------------|
|5s          | 50        |0.0010     |204       |245.003         |277.086         |284.818         |295.777         |
|5s          | 100       |0.0009     |225       | 452.578        |492.355         |506.399         |533.325         |
|5s          | 150       |0.0009     |228       |657.478         |722.427         |747.493         |794.346         |
|5s          |200        |0.0009     |228       |875.721         |946.02          |975.703         |1020.615        |
|10s         |10         |0.0011     |88        |116.203         |113.929         |136.902         |149.18          |
|10s         |50         |0.0009     |108       |476.678         |522.65          |532.693         |562.097         |
|10s         |100        |0.0009     |110       |921.383         |1001.848        |1029.96         |1067.6          |

* Wenetspeech, FP16, onnx, [Conformer](http://mobvoi-speech-public.ufile.ucloud.cn/public/wenet/wenetspeech/20211025_conformer_exp.tar.gz)

|input length|Concurrency|RTF / GPU  |Throughput|Latency_p50 (ms)|Latency_p90 (ms)|Latency_p95 (ms)|Latency_p99 (ms)|
|------------|-----------|-----------|----------|----------------|----------------|----------------|----------------|
|5s          |50         |0.0018     |110       |464.18          |508.246         |517.967         |547.002         |
|5s          |100        |0.0018     |112       |891.173         |958.046         |1011.058        |1093.231        |
|10s         |5          |0.0020     |50        |100.35          |120.757         |122.148         |132.053         |
|10s         |10         |0.0020     |50        |201.551         |240.75          |252.026         |286.132         |
|10s         |50         |0.0019     |52        |986.52          |1030.635        |1051.282        |1101.016        |

* The offline model_repo's overview:



### Online Model Perf
* Aishell2, FP16, onnx, [U2++ Conformer](http://mobvoi-speech-public.ufile.ucloud.cn/public/wenet/aishell2/20210618_u2pp_conformer_exp.tar.gz)
Our chunksize is 16 * 4 * 10 = 640 ms, so we should care about the perf of latency less than 640ms so that it can be a realtime application.

<table border='0' cellpadding='0' cellspacing='0'>
<tr id='r0'>
<td>Model</td>
<td>Num Left Chunks</td>
<td>Concurrency</td>
<td>Avg Latency (ms)</td>
<td>Latency P95 (ms)</td>
<td>Latency P99 (ms)</td>
<td>Throughput (infer/s)</td>
<td>Platform</td>
 </tr>
 <tr id='r1'>
<td rowspan='2' class='x21'>ctc prefix beam search</td>
<td rowspan='4' class='x21'>5</td>
<td>50</td>
<td>78</td>
<td>110</td>
<td>134</td>
<td>644</td>
<td rowspan='4' class='x21'>onnx</td>
 </tr>
 <tr id='r2'>
<td>100</td>
<td>124</td>
<td>155</td>
<td>192</td>
<td>805</td>
 </tr>
 <tr id='r5'>
<td rowspan='2' class='x21'>attention rescoring</td>
<td>50</td>
<td>99</td>
<td>136</td>
<td>164</td>
<td>504</td>
 </tr>
 <tr id='r6'>
<td>100</td>
<td>172</td>
<td>227</td>
<td>272</td>
<td>585</td>
 </tr>
</table>

### Improve Accuracy

#### Add Lanaguage Model
* Add language model: set `--lm_path` in the `convert_start_server.sh`. Notice the path of your language model is the path in docker.
* You may refer to `wenet/bin/recognize_onnx.py` to run inference locally. If you want to add language model locally, you may refer to [here](https://github.com/Slyne/ctc_decoder/blob/master/README.md#usage)

#### Add Hotwords
* Add hotwords: If you use offline model, modify `hotwords_path` in `model_repo/scoring/config_template.pbtxt` (`None`->`/ws/model_repo/scoring/hotwords.yaml`). Otherwise, in streaming model, modify hotwords_path (`None`->`/ws/model_repo/wenet/hotwords.yaml`) in `model_repo_stateful/wenet/config_template.pbtxt`.
* Then follow the steps in Instructions to start the hotwords server.

We use `client.py` to test the effect of hotwords on AISHELL-1 Test dataset and AISHELL-1 hostwords sub-testsets.

[AISHELL-1 Test dataset](https://www.openslr.org/33/)
* Test set contains 7176 utterances (5 hours) from 20 speakers.

| model (FP16)                 | RTF     | CER    |
|------------------------------|---------|--------|
| offline model w/o hotwords   | 0.00437 | 4.6805 |
| offline model w/  hotwords   | 0.00428 | 4.5841 |
| streaming model w/o hotwords | 0.01231 | 5.2777 |
| streaming model w/  hotwords | 0.01195 | 5.1850 |

[AISHELL-1 hostwords sub-testsets](https://www.modelscope.cn/datasets/speech_asr/speech_asr_aishell1_hotwords_testsets/summary)

* Test set contains 235 utterances with 187 entities words.

| model (FP16)               | Latency (s) | CER   | Recall | Precision | F1-score |
|----------------------------|-------------|-------|--------|-----------|----------|
| offline model w/o hotwords | 5.8673      | 13.85 | 0.27   | 0.99      | 0.43     |
| offline model w/  hotwords | 5.6601      | 11.96 | 0.47   | 0.97      | 0.63     |

Decoding result

| Label                | hotwords  | pred w/o hotwords            | pred w/ hotwords             |
|----------------------|-----------|------------------------------|------------------------------|
| 以及拥有陈露的女单项目          | 陈露        | 以及拥有**陈鹭**的女单项目              | 以及拥有**陈露**的女单项目              |
| 庞清和佟健终于可以放心地考虑退役的事情了 | 庞清<br/>佟健 | **庞青**和**董建**终于可以放心地考虑退役的事情了 | **庞清**和**佟健**终于可以放心地考虑退役的事情了 |
| 赵继宏老板电器做厨电已经三十多年了    | 赵继宏       | **赵继红**老板电器做厨店已经三十多年了        | **赵继宏**老板电器做厨电已经三十多年了        |

##### Tested ENV
* CPU：40 Core, Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
* GPU：NVIDIA GeForce RTX 2080 Ti
* Acoustic model: [20210601_u2++_conformer_exp (AISHELL-1)](https://github.com/wenet-e2e/wenet/blob/main/docs/pretrained_models.md)

Refer to more results: https://huggingface.co/58AILab/wenet_u2pp_aishell1_with_hotwords/tree/main/results
##### Hotwords usage
Please refer to the following steps how to use hotwordsboosting.
* Step 1. Initialize HotWordsScorer
```
# if you don't want to use hotwords. set hotwords_scorer=None(default),
# vocab_list is Chinese characters.
hot_words = {'再接': 10, '再厉': -10, '好好学习': 100}
hotwords_scorer = HotWordsScorer(hot_words, vocab_list, is_character_based=True)
```
If you set is_character_based is True (default mode), the first step is to combine Chinese characters into words, if words in hotwords dictionary then add hotwords score. If you set is_character_based is False, all words in the fixed window will be enumerated.

* Step 2. Add hotwords_scorer when decoding
```
result = ctc_beam_search_decoder_batch(batch_chunk_log_prob_seq,
                                        batch_chunk_log_probs_idx,
                                        batch_root_trie,
                                        batch_start,
                                        beam_size, num_processes,
                                        blank_id, space_id,
                                        cutoff_prob, scorer, hotwords_scorer)
```
Please refer to [swig/test/test_zh.py](https://github.com/Slyne/ctc_decoder/blob/master/swig/test/test_zh.py#L108) for how to decode with hotwordsboosting.

##### Hotwords evaluation

Prepare decode result file `with_hotwords_ali.log` and label file `aishell1_text_hotwords`
```
# utt \t text
BAC009S0764W0179    国务院发展研究中心市场经济研究所副所长邓郁松认为
BAC009S0764W0205    本报记者王颖春国家发改委近日发出通知
```

Run the script for evaluation, with multiple result files separated by ';'.
```
cd runtime/gpu/scripts
python compute_hotwords_f1.py \
    --label="aishell1_text_hotwords" \
    --preds="with_hotwords_ali.log;data/without_hotwords_ali.log" \
    --hotword="../model_repo/scoring/hotwords.yaml"
```

Output ner file:
```
BAC009S0764W0179    国务院发展研究中心市场经济研究所副所长邓郁松认为    邓郁松
BAC009S0764W0205    本报记者王颖春国家发改委近日发出通知  王颖春
```

#### Dynamic Left Chunks
For online model, training with dynamic left chunk option on will help further improve the model accuracy.
Let's take a look at the below table.

**WenetSpeech**
chunksize = 16
|Scoring                | left-full | left-8 | left-4 | left-2 | left-1 | left-0 |
|-----------------------|-----------|--------|--------|--------|--------|--------|
| DEV |
|ctc greedy             | 9.16        | 9.19   |9.12      |9.05    |9.08      | 9.3    |
|ctc prefix beam          | 9.1          | 9.11     |9.04      |8.98       |9          |9.23    |
|attention rescoring      | 8.76      |    8.75     |8.69      |8.62       |8.61      |8.74    |
| Test Meeting |
| ctc greedy              |18.43        |18.47     |18.53      |18.81     |19.27      |20.37   |
| ctc prefix beam          |18.27        |18.29     |18.35      |18.65     |19.13      |20.22   |
| attention rescoring      |17.77        |17.81     |17.9      |18.19     |18.66      |19.68   |
| Test Net |
|ctc greedy                |10.91        |10.93     |10.98      |11.12     |11.29      |11.77   |
|ctc prefix beam          |10.86        |10.88     |10.93      |11.07     |11.24      |11.72   |
|attention rescoring      |10.13        |10.16     |10.2      |10.34     |10.51      |10.93   |

With dynamic left = False in training and chunk size=16 when inferencing, we take all the left chunks:
|Scoring            | DEV | Test Meeting | Test Net |
|-------------------|-----|--------------|----------|
|ctc greedy         | 9.32| 18.79        | 11.02    |
|ctc prefix beam    | 9.25| 18.62        | 10.96    |
|attention rescoring| 8.87| 18.11        | 10.22    |


### Warning
* We only tested mandarin models in `wenetspeech` and `aishell2` and haven't tested models built with English corpus.
* The bpe models currently are not supported.
* You need to add a VAD module to split your long audio into segments.
* Please pad your audio in order to leverage triton inference [dynamic batching](https://github.com/triton-inference-server/server/blob/main/docs/architecture.md#models-and-schedulers) feature. We suggest you to pad your audio to nearest length such as 2s, 5s, 10s seconds.
* We're still working on improving this solution. If you find any issue, please raise an issue.
* For streaming pipeline, we haven't implemented `endpoint` feature, which means you have to cut your audio manually.
* Please use `client.py` as a reference for building an intermediate server between the true clients (e.g., web application) and triton inference server. You may add the padding strategy (for offline model), buffering, vad, endpoint or other preprocessing (for streaming model) in this intermediate server.

More info can be found on Triton documents. For example, if you want to deploy your application on scale, you may refer to [triton k8s deploy](https://github.com/triton-inference-server/server/blob/main/deploy/gcp/README.md).


## Reference
* [Triton inference server](https://github.com/triton-inference-server/server) & [Client](https://github.com/triton-inference-server/client)

## Acknowledge
This part originates from NVIDIA CISI project. We also have TTS and NLP solutions deployed on triton inference server.
If you are interested, please contact us.

Thanks to @RiverLiu @Jiawei and @day9011 for great effort in testing.

---

### Overview (`runtime/horizonbpu/README.md`)

# WeNet & Horizon BPU (Cross Compile)

* Step 1. Setup environment (install horizon packages and cross compile tools) in the PC. (~10min)

```sh
# Conda env (This conda env is only used for converting bpu models, not for training torch models,
#   It's OK to install cpu-version pytorch)
conda create -n horizonbpu python=3.8
conda activate horizonbpu
git clone https://github.com/wenet-e2e/wenet.git
cd wenet/runtime/horizonbpu
pip install -r ../../requirements.txt -i https://mirrors.aliyun.com/pypi/simple
pip install torch==1.13.0 torchaudio==0.13.0 torchvision==0.14.0 onnx onnxruntime -i https://mirrors.aliyun.com/pypi/simple

# Horizon packages
wget https://gitee.com/xcsong-thu/toolchain_pkg/releases/download/resource/wheels.tar.gz
tar -xzf wheels.tar.gz
pip install wheels/* -i https://mirrors.aliyun.com/pypi/simple

# Cross compile tools
sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
```


* Step 2. Build decoder_main. It requires cmake 3.14 or above. and Send the binary/libraries to Horizon X3PI. (~20min)

``` sh
# Assume current dir is `wenet/runtime/horizonbpu`
cmake -B build -DBPU=ON -DONNX=OFF -DTORCH=OFF -DWEBSOCKET=OFF -DGRPC=OFF -DCMAKE_TOOLCHAIN_FILE=toolchains/aarch64-linux-gnu.toolchain.cmake
cmake --build build

# Send binary and libraries
export BPUIP=xxx.xxx.xxx.xxx
export DEMO_PATH_ON_BOARD=/path/to/demo
scp build/bin/decoder_main sunrise@$BPUIP:$DEMO_PATH_ON_BOARD
scp fc_base/easy_dnn-src/dnn/*j3*/*/*/lib/libdnn.so sunrise@$BPUIP:$DEMO_PATH_ON_BOARD
scp fc_base/easy_dnn-src/easy_dnn/*j3*/*/*/lib/libeasy_dnn.so sunrise@$BPUIP:$DEMO_PATH_ON_BOARD
scp fc_base/easy_dnn-src/hlog/*j3*/*/*/lib/libhlog.so sunrise@$BPUIP:$DEMO_PATH_ON_BOARD
```

* Step 3. Export model to ONNX and convert ONNX to Horizon .bin and Send the model/dict/test_wav to Horizon X3PI. (~40min)

```
/* Detailed source-code truncated for AI context efficiency. */
```

* Step 4. Testing on X3PI, the RTF(real time factor) is shown in Horizon X3PI's console. (~1min)

``` sh
cd /path/to/demo
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH \
  GLOG_logtostderr=1 GLOG_v=2 \
  ./decoder_main \
      --chunk_size 8 \
      --num_left_chunks 16 \
      --rescoring_weight 0.0 \
      --wav_path ./test_wav.wav \
      --bpu_model_dir ./ \
      --unit_path ./units.txt 2>&1 | tee log.txt
```

---

### Overview (`runtime/ios/README.md`)

# WeNet On-device ASR iOS Demo

## Build application from source code

### 1) Generate cmake project, install LibTorch pod and build static libraries

```
cd runtime/ios/build
cmake .. -G Xcode -DTORCH=ON -DONNX=OFF -DIOS=ON -DGRAPH_TOOLS=OFF -DBUILD_TESTING=OFF -DCMAKE_TOOLCHAIN_FILE=../toolchains/ios.toolchain.cmake -DPLATFORM=OS64 -DENABLE_BITCODE=FALSE
pod install

# Build debug version
cmake --build . --config Debug

# Build release version
cmake --build . --config Release
```

### 2) Build and run iOS application

You can use our pretrained model (click the following link to download):

[AISHELL-1](https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/aishell/20210601_u2%2B%2B_conformer_libtorch.tar.gz)
| [AISHELL-2](https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/aishell2/20210618_u2pp_conformer_libtorch.tar.gz)
| [GigaSpeech](https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/gigaspeech/20210728_u2pp_conformer_libtorch.tar.gz)
| [LibriSpeech](https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/librispeech/20210610_u2pp_conformer_libtorch.tar.gz)
| [Multi-CN](https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/multi_cn/20210815_unified_conformer_libtorch.tar.gz)


Or you can train your own model using WeNet training pipeline on your data.

When your model is ready, put `final.zip` and `units.txt` into model (`WenetDemo/WenetDemo/model`) folder.

Open WenetDemo.xcodeproj with Xcode, build and run on iOS device.

---

### Overview (`runtime/ipex/README.md`)

## WeNet Server (x86) ASR Demo With Intel® Extension for PyTorch\* Optimization

[Intel® Extension for PyTorch\*](https://github.com/intel/intel-extension-for-pytorch) (IPEX) extends [PyTorch\*](https://pytorch.org/) with up-to-date optimization features for extra performance boost on Intel hardware. The optimizations take advantage of AVX-512, Vector Neural Network Instructions (AVX512 VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel X<sup>e</sup> Matrix Extensions (XMX) AI engines on Intel discrete GPUs.

In the following we are introducing how to accelerate WeNet model inference performance on Intel® CPU machines with the adoption of Intel® Extension for PyTorch\*. The adoption mainly includes the export of pretrained models with IPEX optimization, as well as the buildup of WeNet runtime executables with IPEX C++ SDK. The buildup can be processed from local source code, or directly build and run a docker container in which the runtime binaries are ready.

We provide a shell script for WeNet runtime environment checking. Please run the script

``` sh
sh env_checking.sh
```

and see if the environment is suitable for leveraging resources to get optimal performance for WeNet runtime models.

### Run in Docker Build

We recommend using the docker environment to build the c++ binary to avoid
system or environment problems.

* Step 1. Build your docker image.

``` sh
cd docker
docker build --no-cache -t wenet-ipex:latest .
```

* Step 2. The pretrained PyTorch model checkpoint file (.pt) needs to be converted to TorchScript runtime model (.zip) via `export_ipex.py`. The detail usage for this model exporting process are introduced in [Run with Local Build](#run-with-local-build) section below. After model exporting, we can put all the resources, like runtime model, test wavs into a docker resource dir.

``` sh
mkdir -p docker_resource
cp -r <your_model_dir> docker_resource/model
cp <your_test_wav> docker_resource/test.wav
```

* Step 3. Start docker container.

``` sh
docker run --rm -v $PWD/docker_resource:/home/wenet/runtime/ipex/docker_resource -it wenet-ipex:latest bash
```

* Step 4. Test in docker container

```sh
cd /home/wenet/runtime/ipex
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=docker_resource/test.wav
model_dir=docker_resource/model
./build/bin/decoder_main \
    --chunk_size -1 \
    --wav_path $wav_path \
    --model_path $model_dir/<runtime_model_filename> \
    --unit_path $model_dir/units.txt 2>&1 | tee log.txt
```

### Run with Local Build

* Step 1. Environment Setup.

WeNet code cloning and default dependencies installation

``` sh
git clone https://github.com/wenet-e2e/wenet
cd wenet
pip install -r requirements.txt
```

Upgrading of PyTorch and TorchAudio, followed by the installation of IPEX

``` sh
pip install torch==2.3.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cpu --force-reinstall
pip install intel_extension_for_pytorch==2.3.0
```

Installation of related tools: Intel® OpenMP and TCMalloc

`pip install intel-openmp`

and

`yum -y install gperftools` or `apt-get install -y google-perftools`

based on the package manager of your system.

* Step 2. Download or prepare your pretrained model.

* Step 3. Export the pretrained model with IPEX optimization.

For exporting FP32 runtime model

``` sh
source examples/aishell/s0/path.sh
export OMP_NUM_THREADS=1
python wenet/bin/export_ipex.py \
    --config <model_config_yaml_filename> \
    --checkpoint <model_ckpt_filename> \
    --output_file <runtime_model_output_filename>
```

If you have an Intel® 4th Generation Xeon (Sapphire Rapids) server, you can export a BF16 runtime model and get better performance by virtue of [AMX instructions](https://en.wikipedia.org/wiki/Advanced_Matrix_Extensions)

``` sh
source examples/aishell/s0/path.sh
export OMP_NUM_THREADS=1
python wenet/bin/export_ipex.py \
    --config <model_config_yaml_filename> \
    --checkpoint <model_ckpt_filename> \
    --output_file <runtime_model_output_filename> \
    --dtype bf16
```

And for exporting int8 quantized runtime model

``` sh
source examples/aishell/s0/path.sh
export OMP_NUM_THREADS=1
python wenet/bin/export_ipex.py \
    --config <model_config_yaml_filename> \
    --checkpoint <model_ckpt_filename> \
    --output_quant_file <runtime_model_output_filename>
```

* Step 4. Build runtime executable binaries. The build requires cmake 3.14 or above. For building, please first change to `wenet/runtime/ipex` as your build directory, then type:

``` sh
mkdir build && cd build && cmake .. && cmake --build .
```

* Step 5. Testing, the RTF(real time factor) would be shown in the console.

``` sh
cd ..
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
model_dir=your_model_dir
ipexrun --no-python \
    ./build/bin/decoder_main \
        --chunk_size -1 \
        --wav_path $wav_path \
        --model_path $model_dir/<runtime_model_filename> \
        --unit_path $model_dir/units.txt 2>&1 | tee log.txt
```

NOTE: Please refer [IPEX Launch Script Usage Guide](https://intel.github.io/intel-extension-for-pytorch/cpu/2.3.0+cpu/tutorials/performance_tuning/launch_script.html) for usage of advanced features.

For advanced usage of WeNet, such as building Web/RPC/HTTP services, please refer [LibTorch Tutorial](../libtorch#advanced-usage). The difference is that the executables should be invoked via IPEX launch script `ipexrun`.

---

### Overview (`runtime/kunlun/README.md`)

# 在昆仑芯片上运行Wenet
## 介绍
下面的示例展示了如何在XPU上部署WeNet离线或在线的ASR模型。XPU是一种由昆仑芯100%自主研发的通用人工智能计算核心架构。

## 准备XPU运行环境

在开始之前，请确认您获得以下必须的环境。

    XRE(XPU Runtime Environment):昆仑芯片的基础运行环境，包括芯片驱动程序、runtime api库、固件FW工具等功能模块。
    XDNN(XPU Deep Neural Network Library):加速深度神经网络的昆仑芯片库，提供应用程序中使用的高性能DNN功能库。

如果您需要任何帮助，或是想要进一步了解昆仑芯片，请通过官方网址联系我们：
https://www.kunlunxin.com.cn/

## 操作步骤
- 第一步：构建，需要cmake 3.14及以上版本

``` sh
export CXX=${your_g++_path}
export CC=${your_gcc_path}
export XPU_API_PATH=${your_api_path}

# -r : release version; -d : debug version
bash ./compile.sh -r
```

- 第二步：测试，测试结果将在控制台输出

``` sh
## set KUNLUN XPU visible device
export XPU_VISIBLE_DEVICES=0
export XPUSIM_DEVICE_MODEL=KUNLUN2
## set logging level
export GLOG_logtostderr=1
export GLOG_v=3
## set speech wav and model/weight path
wav_path=${your_test_wav_path}
xpu_model_dir=${your_xpu_weight_dir}
units=${your_units.txt}
## executive command
./build/bin/decoder_main \
    --chunk_size -1 \
    --wav_path ${wav_path} \
    --xpu_model_dir ${xpu_model_di} \
    --unit_path ${units}   \
    --device_id 0           \
    --nbest  3  2>&1 | tee log.txt
```

单条语音执行结果如下所示:

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Overview (`runtime/libtorch/README.md`)

# WeNet Server (x86) ASR Demo

**[中文版:x86 平台上使用 WeNet 进行语音识别](./README_CN.md)**

## Run with Prebuilt Docker

* Step 1. Download the [pretrained models](../../docs/pretrained_models.en.md) or prepare your trained model.

* Step 2. Start docker websocket server. Here is a demo.

``` sh
model_dir=$PWD/aishell_u2pp_conformer_libtorch  # absolute path
docker run --rm -it -p 10086:10086 -v $model_dir:/home/wenet/model wenetorg/wenet-mini:latest bash /home/run.sh
```

* Step 3. Test with web browser. Open runtime/libtorch/web/templates/index.html in the browser directly, input your `WebSocket URL`, it will request some permissions, and start to record to test, as the following graph shows.



## Run in Docker Build

We recommend using the docker environment to build the c++ binary to avoid
system and environment problems.

* Step 1. Build your docker image.

``` sh
cd docker
docker build --no-cache -t wenet:latest .
```

* Step 2. Put all the resources, like model, test wavs into a docker resource dir.

``` sh
mkdir -p docker_resource
cp -r <your_model_dir> docker_resource/model
cp <your_test_wav> docker_resource/test.wav
```

* Step 3. Start docker container.
``` sh
docker run --rm -v $PWD/docker_resource:/home/wenet/runtime/libtorch/docker_resource -it wenet bash
```

* Step 4. Testing in docker container
```
cd /home/wenet/runtime/libtorch
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=docker_resource/test.wav
model_dir=docker_resource/model
./build/bin/decoder_main \
    --chunk_size -1 \
    --wav_path $wav_path \
    --model_path $model_dir/final.zip \
    --unit_path $model_dir/units.txt 2>&1 | tee log.txt
```

Or you can do the WebSocket server/client testing as described in the `WebSocket` section.

## Run with Local Build

* Step 1. Download or prepare your pretrained model.

* Step 2. Build. The build requires cmake 3.14 or above. For building, please first change to `wenet/runtime/libtorch` as your build directory, then type:

``` sh
mkdir build && cd build && cmake .. && cmake --build .
```

For building with GPU, you should turn on `GPU`:

``` sh
mkdir build && cd build && cmake -DGPU=ON .. && cmake --build .
```

* Step 3. Testing, the RTF(real time factor) is shown in the console.

``` sh
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
model_dir=your_model_dir
./build/bin/decoder_main \
    --chunk_size -1 \
    --wav_path $wav_path \
    --model_path $model_dir/final.zip \
    --unit_path $model_dir/units.txt 2>&1 | tee log.txt
```


## Advanced Usage

### WebSocket

* Step 1. Download or prepare your pretrained model.
* Step 2. Build as in `Run with Local Build`
* Step 3. Start WebSocket server.

``` sh
export GLOG_logtostderr=1
export GLOG_v=2
model_dir=your_model_dir
./build/bin/websocket_server_main \
    --port 10086 \
    --chunk_size 16 \
    --model_path $model_dir/final.zip \
    --unit_path $model_dir/units.txt 2>&1 | tee server.log
```
* Step 4. Start WebSocket client.

```sh
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
./build/bin/websocket_client_main \
    --hostname 127.0.0.1 --port 10086 \
    --wav_path $wav_path 2>&1 | tee client.log
```

You can also start WebSocket client by web browser as described before.

Here is a demo for command line based websocket server/client interaction.



### gRPC

Why grpc? You may find your answer in https://grpc.io/.
Please follow the following steps to try gRPC.

* Step 1. Download or prepare your pretrained model.
* Step 2. Build
``` sh
mkdir build && cd build && cmake -DGRPC=ON .. && cmake --build .
```
* Step 3. Start gRPC server

``` sh
export GLOG_logtostderr=1
export GLOG_v=2
model_dir=your_model_dir
./build/bin/grpc_server_main \
    --port 10086 \
    --workers 4 \
    --chunk_size 16 \
    --model_path $model_dir/final.zip \
    --unit_path $model_dir/units.txt 2>&1 | tee server.log
```

* Step 4. Start gRPC client.

```sh
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
./build/bin/grpc_client_main \
    --hostname 127.0.0.1 --port 10086 \
    --wav_path $wav_path 2>&1 | tee client.log
```

### http

* Step 1. Download or prepare your pretrained model.
* Step 2. Build
``` sh
mkdir build && cd build && cmake -DHTTP=ON .. && cmake --build .
```
* Step 3. Start http server

simply replace grpc_server_main with http_server_main of Step 3 in gRPC

* Step 4. Start http client.

simply replace grpc_client_main with http_client_main of Step 4 in gRPC

---

### Overview (`runtime/onnxruntime/README.md`)

# ONNX backend on WeNet

* Step 1. Export your experiment model to ONNX by https://github.com/wenet-e2e/wenet/blob/main/wenet/bin/export_onnx_cpu.py

``` sh
exp=exp  # Change it to your experiment dir
onnx_dir=onnx
python -m wenet.bin.export_onnx_cpu \
  --config $exp/train.yaml \
  --checkpoint $exp/final.pt \
  --chunk_size 16 \
  --output_dir $onnx_dir \
  --num_decoding_left_chunks -1

# When it finishes, you can find `encoder.onnx`, `ctc.onnx`, and `decoder.onnx` in the $onnx_dir respectively.
```

* Step 2. Build. The build requires cmake 3.14 or above.

``` sh
mkdir build && cd build
cmake -DONNX=ON -DTORCH=OFF -DWEBSOCKET=OFF -DGRPC=OFF ..
cmake --build .
```

* Step 3. Testing, the RTF(real time factor) is shown in the console.

``` sh
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
onnx_dir=your_model_dir
units=units.txt  # Change it to your model units path
# Make sure that the `chunk_size` and `num_left_chunks` variables are set to the corresponding values used when exporting the ONNX models.
./build/bin/decoder_main \
    --chunk_size 16 \
    --num_left_chunks -1 \
    --wav_path $wav_path \
    --onnx_dir $onnx_dir \
    --unit_path $units 2>&1 | tee log.txt
```

---

### Overview (`runtime/openvino/README.md`)

# OpenVINO™ backend on WeNet

* Step 1. Get ONNX model.

Use `wenet.bin.export_onnx_cpu` to export ONNX model. Follow this [guide](https://github.com/wenet-e2e/wenet/blob/main/runtime/onnxruntime/README.md)

Please note, 3 ONNX models(`encoder.onnx`, `ctc.onnx`, and `decoder.onnx`) will be generated.

* Step 2. Convert ONNX model to OpenVINO™ IR(Intermediate Representation).

Install OpenVINO™ runtime with Model Optimizer (MO) command tool for model conversion:

``` sh
mo --input_model onnx/encoder.onnx --input chunk[1,-1,80],att_cache[12,4,-1,128],cnn_cache[12,1,256,7] --output_dir openvino 
mo --input_model onnx/ctc.onnx --input_shape [1,-1,256] --output_dir openvino 
mo --input_model onnx/decoder.onnx --input hyps[-1,-1],hyps_lens[-1],encoder_out[1,-1,256]  --output_dir openvino

By using MO, user can specify shape of multiple inputs by `--input` option, or just directly provide `--input_shape` for single input model. To enable the dynamic shape support for inference, user can use `-1` or provide a range of input shape value, like `1..80`.Please refer the [usage guide of MO](https://docs.openvino.ai/latest/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html).


# When it finishes, you can find IR files(.xml and .bin) for encoder, ctc and decoder.
```

* Step 3. Build WeNet with OpenVINO™.

Please refer [system requirement](https://github.com/openvinotoolkit/openvino#system-requirements) to check if the hardware platform available by OpenVINO™.
The OpenVINO Linux installation package will not provide Intel TBB runtime library and pugixml library, please check if your system already installed:
``` sh
# For Ubuntu and Debian:
sudo apt install libtbb-dev libpugixml-dev
# For CentOS:
sudo yum install tbb-devel pugixml-devel
```
To build WeNet with OpenVINO:
``` sh
mkdir build && cd build
cmake -DOPENVINO=ON -DTORCH=OFF -DWEBSOCKET=OFF -DGRPC=OFF ..
make --jobs=$(nproc --all)
```

(Optional) Some users may cannot easily download OpenVINO™ binary package from server due to firewall or proxy issue. If you failed to download by CMake script, you can download OpenVINO™ package by your selves and put the package to below path:

``` sh
${wenet_path}/runtime/openvino/fc_base/openvino-subbuild/openvino-populate-prefix/src/l_openvino_toolkit_ubuntu20_2022.3.0.9052.9752fafe8eb_x86_64.tgz
```

* Step 4. Testing.

``` sh
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
openvino_dir=your_model_dir
units=units.txt  # Change it to your model units path
./build/bin/decoder_main \
    --chunk_size 16 \
    --wav_path $wav_path \
    --openvino_dir $openvino_dir \
    --unit_path $units 2>&1 | tee log.txt
```

---

### Overview (`runtime/raspberrypi/README.md`)

# WeNet & Raspberry PI (Cross Compile)

* Step 1. Install cross compile tools in the PC.

``` sh
sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
```

Or download, and install the binaries from: https://releases.linaro.org/components/toolchain/binaries/latest-7


* Step 2. Export your experiment model to ONNX by https://github.com/wenet-e2e/wenet/blob/main/wenet/bin/export_onnx_cpu.py

``` sh
exp=exp  # Change it to your experiment dir
onnx_dir=onnx
python -m wenet.bin.export_onnx_cpu \
  --config $exp/train.yaml \
  --checkpoint $exp/final.pt \
  --chunk_size 16 \
  --output_dir $onnx_dir \
  --num_decoding_left_chunks -1

# When it finishes, you can find `encoder.onnx(.quant)`, `ctc.onnx(.quant)`, and `decoder.onnx(.quant)` in the $onnx_dir respectively.
# We use the quantified to speed up the inference, so rename it without the suffix `.quant`
```

* Step 3. Build. The build requires cmake 3.14 or above. and Send the binary and libraries to Raspberry PI.

``` sh
cmake -B build -DONNX=ON -DTORCH=OFF -DWEBSOCKET=OFF -DGRPC=OFF -DCMAKE_TOOLCHAIN_FILE=toolchains/aarch64-linux-gnu.toolchain.cmake
cmake --build build
scp build/bin/decoder_main pi@xxx.xxx.xxx:/path/to/wenet
scp fc_base/onnxruntime-src/lib/libonnxruntime.so* pi@xxx.xxx.xxx:/path/to/wenet
```

* Step 4. Testing, the RTF(real time factor) is shown in Raspberry PI's console.

``` sh
cd /path/to/wenet
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=your_test_wav_path
onnx_dir=your_model_dir
units=units.txt  # Change it to your model units path
./build/bin/decoder_main \
    --chunk_size 16 \
    --wav_path $wav_path \
    --onnx_dir $onnx_dir \
    --unit_path $units 2>&1 | tee log.txt
```

---

### Overview (`runtime/web/README.md`)

## WeNet Web Demo

* How to install? `pip install -r requirements.txt`
* How to start?
  - Non-streaming: `python app.py`

---

### Overview (`examples/aishell/NST/README.md`)

# Recipe to run Noisy Student Training with LM filter in WeNet

Noisy Student Training (NST) has recently demonstrated extremely strong performance in Automatic Speech Recognition (ASR).

Here, we provide a recipe to run NST with `LM filter` strategy using AISHELL-1 as supervised data and WenetSpeech as unsupervised data from [this paper](https://arxiv.org/abs/2211.04717), where hypotheses with and without Language Model are generated and CER differences between them are utilized as a filter threshold to improve the ASR performances of non-target domain datas.

## Table of Contents

- [Guideline](#guideline)
  - [Data preparation](#data-preparation)
  - [Initial supervised teacher](#initial-supervised-teacher)
  - [Noisy student interations](#noisy-student-interations)
- [Performance Record](#performance-record)
  - [Supervised baseline and standard NST](##supervised-baseline-and-standard-nst)
  - [Supervised AISHELL-1 and unsupervised 1khr WenetSpeech](#supervised-aishell-1-and-unsupervised-1khr-wenetspeech)
  - [Supervised AISHELL-2 and unsupervised 4khr WenetSpeech](#supervised-aishell-2-and-unsupervised-4khr-wenetspeech)
- [Citations](#citations)

## Guideline


First, you have to prepare supervised and unsupervised data for NST. Then in stage 1 of `run.sh`, you will train an initial supervised teacher and generate pseudo labels for unsupervised data.
After that, you can run the noisy student training iteratively in stage 2. The whole pipeline is illustrated in the following picture.



### Data preparation

To run this recipe, you should follow the steps from [WeNet examples](https://github.com/wenet-e2e/wenet/tree/main/examples) to prepare [AISHELL1](https://github.com/wenet-e2e/wenet/tree/main/examples/aishell/s0) and [WenetSpeech](https://github.com/wenet-e2e/wenet/tree/main/examples/wenetspeech/s0) data.
We extract 1khr data from WenetSpeech and data should be prepared and stored in the following format:

```
data/
├── train/
├──── data_aishell.list
├──── wenet_1khr.list
├──── wav_dir/
├──── utter_time.json (optional)
├── dev/
└── test/

```
- Files `*.list` contain paths for all the data shards for training.
- A Json file containing the audio length should be prepared as `utter_time.json` if you want to apply the `speaking rate` filter.
- A wav_dir contains all the audio data (id.wav) and labels (id.txt which is optional) for unsupervised data.

### Initial supervised teacher

To train an initial supervised teacher model, run the following command:

```bash
bash run.sh --stage 1 --stop-stage 1
```

Full arguments are listed below, you can check `run.sh` and `run_nst.sh` for more information about steps in each stage and their arguments. We used `num_split = 60` and generate shards with different cpu for the experiments in our paper which saved us lots of inference time and data shards generation time.

```bash
bash run.sh --stage 1 --stop-stage 1 --dir exp/conformer_test_fully_supervised --supervised_data_list data_aishell.list --enable_nst 0 --num_split 1 --unsupervised_data_list wenet_1khr.list --dir_split wenet_split_60_test/ --job_num 0 --hypo_name hypothesis_nst0.txt --label 1 --wav_dir data/train/wenet_1k_untar/ --cer_hypo_dir wenet_cer_hypo --cer_label_dir wenet_cer_label --label_file label.txt --cer_hypo_threshold 10 --speak_rate_threshold 0 --utter_time_file utter_time.json --untar_dir data/train/wenet_1khr_untar/ --tar_dir data/train/wenet_1khr_tar/ --out_data_list data/train/wenet_1khr.list
```
- `dir` contains the training parameters.
- `data_list` contains paths for the training data list.
- `supervised_data_list` contains paths for supervised data shards.
- `unsupervised_data_list`contains paths for unsupervised data shards which is used for inference.
- `dir_split` is the directory stores split unsupervised data for parallel computing.
- `out_data_list` is the pseudo label data list file path.
- `enable_nst` indicates whether we train with pseudo label and split data, for initial teacher we set it to 0.
- This recipe uses the default `num_split=1` while we strongly recommend use larger number to decrease the inference and shards generation time.
> **HINTS** If num_split is set to N larger than 1, you need to modify the script in step 4-8 in run_nst.sh to submit N tasks into your own clusters (such as slurm,ngc etc..).
> We strongly recommend to do so since inference and pseudo-data generation is time-consuming.

### Noisy student interations

After finishing the initial fully supervised baseline, we now have the mixed list contains both supervised and pseudo data which is `wenet_1khr_nst0.list`.
We will use it as the `data_list` in the training step and the `data_list` for next NST iteration will be generated.

Here is an example command:

```bash
bash run.sh --stage 2 --stop-stage 2 --iter_num 2
```

Here we add extra argument `iter_num` for number of NST iterations. Intermediate files are named with `iter_num` as a suffix.
Please check the `run.sh` and `run_nst.sh` scripts for more information about each stage and their arguments.

## Performance Record

### Supervised baseline and standard NST
* Non-streaming conformer model with attention rescoring decoder.
* Without filter strategy, first iteration
* Feature info: using FBANK feature, dither, cmvn, online speed perturb
* Training info: lr 0.002, batch size 32, 8 gpu, acc_grad 4, 240 epochs, dither 0.1
* Decoding info: ctc_weight 0.3, average_num 30


| Supervised               | Unsupervised | Test CER |
|--------------------------|--------------|----------|
| AISHELL-1 Only           | ----         | 4.85     |
| AISHELL-1+WenetSpeech    | ----         | 3.54     |
| AISHELL-1+AISHELL-2      | ----         | 1.01     |
| AISHELL-1 (standard NST) | WenetSpeech  | 5.52     |



### Supervised AISHELL-1 and unsupervised 1khr WenetSpeech
* Non-streaming conformer model with attention rescoring decoder.
* Feature info: using FBANK feature
* Training info: lr=0.002, batch_size=32, 8 GPUs, acc_grad=4, 120 epochs, dither=0.1
* Decoding info: ctc_weight=0.3, average_num=30, pseudo_ratio=0.75

| # nst iteration | AISHELL-1 test CER | Pseudo CER| Filtered CER | Filtered hours |
|----------------|--------------------|-----------|--------------|----------------|
| 0 | 4.85             | 47.10     |   25.18           |     323           |
| 1 | 4.86             | 37.02     |   20.93           |     436           |
| 2 | 4.75             | 31.81     |   19.74           |     540           |
| 3 | 4.69             | 28.27     |   17.85           |     592           |
| 4 | 4.48             | 26.64     |   14.76           |     588           |
| 5 | 4.41             | 24.70     |   15.86           |     670           |
| 6 | 4.34             | 23.64     |   15.40           |     669           |
| 7 | 4.31             | 23.79     |   15.75           |     694           |

### Supervised AISHELL-2 and unsupervised 4khr WenetSpeech
* Non-streaming conformer model with attention rescoring decoder.
* Feature info: using FBANK feature
* Training info: lr=0.002, batch_size=32, 8 GPUs, acc_grad=4, 120 epochs, dither=0.1
* Decoding info: ctc_weight=0.3, average_num=30, pseudo_ratio=0.75

| # nst iteration | AISHELL-2 test CER | Pseudo CER | Filtered CER | Filtered hours |
|----------------|--------------------|------------|--------------|----------------|
| 0 | 5.48               | 30.10      | 11.73        | 1637           |
| 1 | 5.09               | 28.31      | 9.39         | 2016           |
| 2 | 4.88               | 25.38      | 9.99         | 2186           |
| 3 | 4.74               | 22.47      | 10.66        | 2528           |
| 4 | 4.73               | 22.23      | 10.43        | 2734           |



## Citations

``` bibtex

@article{chen2022NST,
  title={Improving Noisy Student Training on Non-target Domain Data for Automatic Speech Recognition},
  author={Chen, Yu and Wen, Ding and Lai, Junjie},
  journal={arXiv preprint arXiv:2203.15455},
  year={2022}
}

---

### Overview (`examples/aishell/paraformer/README.md`)

# Preliminary
1. Run below command to convert funasr-style  ckpt to wenet-style ckpt:
```sh
output_dir=exp/paraformer/large
mkdir -p ${output_dir}
. ./path.sh && python wenet/paraformer/convert_paraformer_to_wenet_config_and_ckpt.py \
  --output_dir ${output_dir}
# init ctc and embed(used in sampler)
python local/modify_ckpt.py \
  --add_list "{\"ctc.ctc_lo.weight\": \"embed.weight\"}" \
  --input_ckpt exp/paraformer/large/wenet_paraformer.pt \
  --output_ckpt exp/paraformer/large/wenet_paraformer.init-ctc.init-embed.pt
```

# Performance Record

## Paraformer (original) Result

| decoding mode             |  full | 16  |
|---------------------------|-------|-----|
| paraformer greedy search  | 1.95  | N/A |

## Paraformer (full-parameter tuning) Result

* Training info: torch_ddp fp32, batch size 28, ctc_weight: 0.3, acc_grad 1, 8 * 3090 gpu, 60 epochs (about 8h)
* Decoding info: ctc_weight 0.3, average_num 5
* Git hash: TBD

| decoding mode             | full  | 16  |
|---------------------------|-------|-----|
| ctc greedy search         | 3.45 % N=104765 C=101244 S=3406 D=115 I=91  | N/A |
| ctc prefix beam search    | 3.44 % N=104765 C=101247 S=3407 D=111 I=83  | N/A |
| paraformer greedy search  | 2.19 % N=104765 C=102643 S=1959 D=163 I=172 | N/A |

## Paraformer-dynamic training (full-parameter tuning) Result

* Training info: torch_ddp fp32, batch size 28, ctc_weight: 0.3, acc_grad 1, 8 * 3090 gpu, 60 epochs (about 8h)
* Decoding info: ctc_weight 0.3, average_num 5
* Git hash: TBD

| decoding mode             | full   | 16   |
|---------------------------|--------|------|
| ctc greedy search         | 3.46 % N=104765 C=101235 S=3409 D=121 I=98   | 4.18 % N=104765 C=100495 S=4149 D=121 I=107 |
| ctc prefix beam search    | 3.45 % N=104765 C=101239 S=3413 D=113 I=91   | 4.17 % N=104765 C=100500 S=4150 D=115 I=103 |
| paraformer greedy search  | 2.15 % N=104765 C=102640 S=1977 D=148 I=132  | 2.40 % N=104765 C=102409 S=2220 D=136 I=161 |

---

### Overview (`examples/aishell/rnnt/README.md`)

# Performance Record

## Conformer Result

* Feature info: using fbank feature, dither, cmvn, online speed perturb
* Training info: lr 0.001, batch size 8, 8 gpu, acc_grad 1, 100 epochs, dither 0.1
* Training weight info: transducer_weight 0.75, ctc_weight 0.1, attention_weight 0.15, average_num 10
* Predictor type: lstm

| decoding mode             | CER   |
|---------------------------|-------|
| rnnt greedy search        | 5.24  |

* after 165 epochs and avg 30

| decoding mode             | CER   |
|---------------------------|-------|
| rnnt greedy search        | 5.02  |
| ctc prefix beam search    | 5.17  |
| ctc prefix beam + rescore | 4.48  |

## Conformer Result

* Feature info: using fbank feature, dither, cmvn, online speed perturb
* Training info: lr 0.001, batch size 20, 8 gpu, acc_grad 1, 140 epochs, dither 0.1
* Training weight info: transducer_weight 0.4, ctc_weight 0.2, attention_weight 0.4, average_num 10
* Predictor type: lstm
* Model link: https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/aishell/20220728_conformer_rnnt_exp.tar.gz

| decoding mode                         | CER   |
|---------------------------------------|-------|
| rnnt greedy search                    | 4.88  |
| rnnt beam search                      | 4.67  |
| ctc prefix beam search                | 5.02  |
| ctc prefix beam + rescore             | 4.51  |
| ctc prefix beam + rnnt&attn rescore   | 4.45  |
| rnnt prefix beam + rnnt&attn rescore  | 4.49  |


## U2++ Conformer Result

* Feature info: using fbank feature, dither, cmvn, oneline speed perturb
* Training info: lr 0.001, batch size 4, 32 gpu, acc_grad 1, 360 epochs
* Training weight info: transducer_weight 0.75,  ctc_weight 0.1, reverse_weight 0.15  average_num 30
* Predictor type: lstm

| decoding mode/chunk size  | full  | 16    |
|---------------------------|-------|-------|
| rnnt greedy search        | 5.68  | 6.26  |

## Pretrain
* Pretrain model: https://wenet-1256283475.cos.ap-shanghai.myqcloud.com/models/aishell/20210601_u2%2B%2B_conformer_exp.tar.gz
* Feature info: using fbank feature, dither, cmvn, oneline speed perturb
* Training info: lr 0.001, batch size 8, 8 gpu, acc_grad 1, 140 epochs
* Training weight info: transducer_weight 0.4,  ctc_weight 0.2 , attention_weight 0.4, reverse_weight 0.3  average_num 30
* Predictor type: lstm

| decoding mode/chunk size    | full  | 16     |
|-----------------------------|-------|--------|
| rnnt greedy search          | 5.21  | 5.73   |
| rnnt prefix beam            | 5.14  | 5.63   |
| rnnt prefix beam + rescore  | 4.73  | 5.095  |


## Training loss ablation study

note:

- If rnnt is checked, greedy means rnnt  greedy search; so is beam

- if rnnt is checked, rescoring means rnnt beam & attention rescoring

- if only 'ctc & att' is checked, greedy means ctc gredy search; so is beam

- if only  'ctc & att' (AED)  is checked, rescoring means ctc beam & attention rescoring

- what if rnnt model do search of wenet's style, comming soon

| rnnt | ctc | att | greedy | beam | rescoring | fusion |
|------|-----|-----|--------|------|-----------|--------|
| ✔    | ✔   | ✔   |   4.88 | 4.67 |      4.45 |   4.49 |
| ✔    | ✔   |     |   5.56 | 5.46 |       /   |   5.40 |
| ✔    |     | ✔   |   5.03 | 4.94 |      4.87 |    /   |
| ✔    |     |     |   5.64 | 5.59 |       /   |    /   |
|      | ✔   | ✔   |   4.94 | 4.94 |      4.61 |    /   |

---

### Overview (`examples/aishell/s0/README.md`)

# Performance Record

## Conformer Result

* Feature info: using fbank feature, dither, cmvn, online speed perturb
* Training info: lr 0.002, batch size 18, 4 gpu, acc_grad 4, 240 epochs, dither 0.1
* Decoding info: ctc_weight 0.5, average_num 20
* Git hash: 919f07c4887ac500168ba84b39b535fd8e58918a

| decoding mode             | CER   |
|---------------------------|-------|
| attention decoder         | 5.18  |
| ctc greedy search         | 4.94  |
| ctc prefix beam search    | 4.94  |
| attention rescoring       | 4.61  |
| LM + attention rescoring  | 4.36  |

## U2++ Conformer Result

* Feature info: using fbank feature, dither=1.0, cmvn, oneline speed perturb
* Training info: lr 0.001, batch size 16, 8 gpu, acc_grad 1, 360 epochs
* Decoding info: ctc_weight 0.3, reverse_weight 0.5  average_num 30, lm_scale 0.7, decoder_scale 0.1, r_decoder_scale 0.7
* Git hash: 5a1342312668e7a5abb83aed1e53256819cebf95

| decoding mode/chunk size  | full  | 16    |
|---------------------------|-------|-------|
| ctc greedy search         | 5.19  | 5.81  |
| ctc prefix beam search    | 5.17  | 5.81  |
| attention rescoring       | 4.63  | 5.05  |
| LM + attention rescoring  | 4.40  | 4.75  |
| HLG(k2 LM)                | 4.81  | 5.27  |
| HLG(k2 LM)  + attention rescoring | 4.32  | 4.70  |
| HLG(k2 LM)  + attention rescoring + LFMMI | 4.11  | 4.47  |

## U2++ lite Conformer Result (uio shard)

* Feature info: using fbank feature, dither=1.0, cmvn, oneline speed perturb
* Training info: lr 0.001, batch size 16, 8 gpu, acc_grad 1, load a well trained model and continue training 80 epochs with u2++ lite config
* Decoding info: ctc_weight 0.3, reverse_weight 0.5  average_num 30
* Git hash: 73185808fa1463b0163a922dc722513b7baabe9e

| decoding mode/chunk size  | full  | 16    |
|---------------------------|-------|-------|
| ctc greedy search         | 5.21  | 5.91  |
| ctc prefix beam search    | 5.20  | 5.91  |
| attention rescoring       | 4.67  | 5.10  |

## Unified Conformer Result

* Feature info: using fbank feature, dither=0, cmvn, oneline speed perturb
* Training info: lr 0.001, batch size 16, 8 gpu, acc_grad 1, 180 epochs, dither 0.0
* Decoding info: ctc_weight 0.5, average_num 20
* Git hash: 919f07c4887ac500168ba84b39b535fd8e58918a

| decoding mode/chunk size  | full  | 16    | 8     | 4     |
|---------------------------|-------|-------|-------|-------|
| attention decoder         | 5.40  | 5.60  | 5.74  | 5.86  |
| ctc greedy search         | 5.56  | 6.29  | 6.68  | 7.10  |
| ctc prefix beam search    | 5.57  | 6.30  | 6.67  | 7.10  |
| attention rescoring       | 5.05  | 5.45  | 5.69  | 5.91  |
| LM + attention rescoring  | 4.73  | 5.08  | 5.22  | 5.38  |

## U2++ Transformer Result

* Feature info: using fbank feature, dither, cmvn, online speed perturb.
* Training info: lr 0.001, batch size 26, 8 gpu, acc_grad 1, 360 epochs, dither 0.1
* Decoding info: ctc_weight 0.2, reverse_weight 0.5, average_num 30
* Git hash: 65270043fc8c2476d1ab95e7c39f730017a670e0

| decoding mode/chunk size  | full  | 16    |
|---------------------------|-------|-------|
| ctc greedy search         | 6.05  | 6.92  |
| ctc prefix beam search    | 6.05  | 6.90  |
| attention rescoring       | 5.11  | 5.63  |
| LM + attention rescoring  | 4.82  | 5.24  |

## Transformer Result

* Feature info: using fbank feature, dither, with cmvn, online speed perturb.
* Training info: lr 0.002, batch size 26, 4 gpu, acc_grad 4, 240 epochs, dither 0.1
* Decoding info: ctc_weight 0.5, average_num 20
* Git hash: 919f07c4887ac500168ba84b39b535fd8e58918a

| decoding mode             | CER   |
|---------------------------|-------|
| attention decoder         | 5.69  |
| ctc greedy search         | 5.92  |
| ctc prefix beam search    | 5.91  |
| attention rescoring       | 5.30  |
| LM + attention rescoring  | 5.04  |

## Unified Transformer Result

* Feature info: using fbank feature, dither=0, with cmvn, online speed perturb.
* Training info: lr 0.002, batch size 16, 4 gpu, acc_grad 1, 240 epochs, dither 0.1
* Decoding info: ctc_weight 0.5, average_num 20
* Git hash: 919f07c4887ac500168ba84b39b535fd8e58918a

| decoding mode/chunk size  | full  | 16    | 8     | 4     |
|---------------------------|-------|-------|-------|-------|
| attention decoder         | 6.04  | 6.35  | 6.45  | 6.70  |
| ctc greedy search         | 6.28  | 6.99  | 7.39  | 7.89  |
| ctc prefix beam search    | 6.28  | 6.98  | 7.40  | 7.89  |
| attention rescoring       | 5.52  | 6.05  | 6.28  | 6.62  |
| LM + attention rescoring  | 5.11  | 5.59  | 5.86  | 6.17  |

## AMP Training Transformer Result

* Feature info: using fbank feature, dither, cmvn, online speed perturb
* Training info: lr 0.002, batch size, 4 gpus, acc_grad 4, 240 epochs, dither 0.1, warm up steps 25000
* Decoding info: ctc_weight 0.5, average_num 20
* Git hash: 1bb4e5a269c535340fae5b0739482fa47733d2c1

| decoding mode          | CER  |
|------------------------|------|
| attention decoder      | 5.73 |
| ctc greedy search      | 5.92 |
| ctc prefix beam search | 5.92 |
| attention rescoring    | 5.31 |


## Muilti-machines Training Conformer Result

* Feature info: using fbank feature, dither, cmvn, online speed perturb
* Training info: lr 0.004, batch size 16, 2 machines, 8\*2=16 gpus, acc_grad 4, 240 epochs, dither 0.1, warm up steps 10000
* Decoding info: ctc_weight 0.5, average_num 20
* Git hash: f6b1409023440da1998d31abbcc3826dd40aaf35

| decoding mode          | CER  |
|------------------------|------|
| attention decoder      | 4.90 |
| ctc greedy search      | 5.07 |
| ctc prefix beam search | 5.06 |
| attention rescoring    | 4.65 |


## Conformer with/without Position Encoding Result

* Feature info: using fbank feature, dither, cmvn, online speed perturb
* Training info: lr 0.002, batch size 16, 8 gpu, acc_grad 4, 240 epochs, dither 0.1
* Decoding info: ctc_weight 0.5, average_num 20

| decoding mode          | with PE | without PE |
|------------------------|---------|------------|
| attention decoder      | 5.18    | 5.73       |
| ctc greedy search      | 4.94    | 4.97       |
| ctc prefix beam search | 4.94    | 4.97       |
| attention rescoring    | 4.61    | 4.69       |


## Efficient Conformer v1 Result

* Feature info:
    * using fbank feature, cmvn, speed perturb, dither
* Training info:
    * train_u2++_efficonformer_v1.yaml
    * 8 gpu, batch size 16, acc_grad 1, 200 epochs
    * lr 0.001, warmup_steps 25000
* Model info:
    * Model Params: 48,488,347
    * Downsample rate: 1/4 (conv2d) * 1/2 (efficonformer block)
    * encoder_dim 256, output_size 256, head 8, linear_units 2048
    * num_blocks 12, cnn_module_kernel 15, group_size 3
* Decoding info:
    * ctc_weight 0.5, reverse_weight 0.3, average_num 20
* Model Download: [wenet_efficient_conformer_aishell_v1](https://huggingface.co/58AILab/wenet_efficient_conformer_aishell_v1)

| decoding mode          | full | 18   | 16   |
|------------------------|------|------|------|
| attention decoder      | 4.99 | 5.13 | 5.16 |
| ctc prefix beam search | 4.98 | 5.23 | 5.23 |
| attention rescoring    | 4.64 | 4.86 | 4.85 |


## Efficient Conformer v2 Result

* Feature info:
    * using fbank feature, cmvn, speed perturb, dither
* Training info:
    * train_u2++_efficonformer_v2.yaml
    * 8 gpu, batch size 16, acc_grad 1, 200 epochs
    * lr 0.001, warmup_steps 25000
* Model info:
    * Model Params: 49,354,651
    * Downsample rate: 1/2 (conv2d2) * 1/4 (efficonformer block)
    * encoder_dim 256, output_size 256, head 8, linear_units 2048
    * num_blocks 12, cnn_module_kernel 15, group_size 3
* Decoding info:
    * ctc_weight 0.5, reverse_weight 0.3, average_num 20
* Model Download: [wenet_efficient_conformer_aishell_v2](https://huggingface.co/58AILab/wenet_efficient_conformer_aishell_v2)

| decoding mode          | full | 18   | 16   |
|------------------------|------|------|------|
| attention decoder      | 4.87 | 5.03 | 5.07 |
| ctc prefix beam search | 4.97 | 5.18 | 5.20 |
| attention rescoring    | 4.56 | 4.75 | 4.77 |


## U2++ Branchformer Result

* Feature info: using fbank feature, dither=1.0, cmvn, oneline speed perturb
* * Model info:
    * Model Params: 48,384,667
    * Num Encoder Layer: 24
    * CNN Kernel Size: 63
    * Merge Method: concat
* Training info: lr 0.001, weight_decay: 0.000001, batch size 16, 3 gpu, acc_grad 1, 360 epochs
* Decoding info: ctc_weight 0.3, reverse_weight 0.5  average_num 30, lm_scale 0.7, decoder_scale 0.1, r_decoder_scale 0.7
* Git hash: 5a1342312668e7a5abb83aed1e53256819cebf95

| decoding mode             | CER   |
|---------------------------|-------|
| ctc greedy search         | 5.28  |
| ctc prefix beam search    | 5.28  |
| attention decoder         | 5.12  |
| attention rescoring       | 4.81  |
| LM + attention rescoring  | 4.46  |

## E-Branchformer Result

* Feature info: using fbank feature, dither=1.0, cmvn, online speed perturb
* * Model info:
    * Model Params: 47,570,132
    * Num Encoder Layer: 17
    * CNN Kernel Size: 31
* Training info: lr 0.001, weight_decay: 0.000001, batch size 16, 4 gpu, acc_grad 1, 240 epochs
* Decoding info: ctc_weight 0.3, average_num 30
* Git hash: 89962d1dcae18dd3a281782a40e74dd2721ae8fe

| decoding mode          | CER  |
| ---------------------- | ---- |
| attention decoder      | 4.73 |
| ctc greedy search      | 4.77 |
| ctc prefix beam search | 4.77 |
| attention rescoring    | 4.39 |
| LM + attention rescoring  | 4.22  |

---

### Overview (`examples/aishell/sensevoice/README.md`)

# Performance Record

## SenseVoice-small (original) Result

| decoding mode          | full |
|------------------------|------|
| ctc greedy search      | 2.92 % N=104765 C=101787 S=2884 D=94 I=77 |
| ctc prefix beam search | 2.92 % N=104765 C=101786 S=2884 D=95 I=82 |

## SenseVoice-small (full-parameter tuning) Result

* Training info: torch_ddp fp32, ctc_weight: 1.0, acc_grad 1, 8 * 3090 gpu, 49 epochs
* Decoding info: ctc_weight 1.0, average_num 5
* Git hash: TBD

| decoding mode          | full |
|------------------------|------|
| ctc greedy search      | 2.19 % N=104765 C=102543 S=2144 D=78 I=70 |
| ctc prefix beam search | 2.19 % N=104765 C=102543 S=2145 D=77 I=70 |

---

### Overview (`examples/aishell/whisper/README.md`)

# Preliminary
1. Download whisper ckpt from this [link](https://github.com/openai/whisper/blob/main/whisper/__init__.py#L17-L30)

2. We assume you have run stage0~stage3 using `aishell/s0/run.sh` and here we are simply creating a symbolic link
```sh
ln -s ../s0/data .
```

3. Run below command to convert openai-style ckpt to wenet-style ckpt:
```sh
mkdir -p exp
mkdir -p exp/whisper
mkdir -p exp/whisper/large-v3
. ./path.sh && python wenet/whisper/convert_whisper_to_wenet_config_and_ckpt.py \
  --whisper_ckpt downloaded-large-v3.pt \
  --output_dir exp/whisper/large-v3
# remove conv1d2
python local/modify_ckpt.py \
  --remove_list "encoder.embed.conv" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.remove-subsample.pt
# init ctc
python local/modify_ckpt.py \
  --add_list "{\"ctc.ctc_lo.weight\": \"decoder.embed.0.weight\"}" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.init-ctc.pt
# remove conv1d2 and init ctc
python local/modify_ckpt.py \
  --remove_list "encoder.embed.conv" \
  --add_list "{\"ctc.ctc_lo.weight\": \"decoder.embed.0.weight\"}" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.remove-subsample.init-ctc.pt
```

# Performance Record

## Whisper-largev2 (original) Result

| decoding mode             |  CER  |
|---------------------------|-------|
| attention decoder         | 8.47  |
| ctc greedy search         |  N/A  |
| ctc prefix beam search    |  N/A  |
| attention rescoring       |  N/A  |

## Whisper-largev3 (conv1d2, full-parameter tuning) Result

* Feature info: using log_mel_spectrogram feature, no cmvn
* Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 1, 8 * 3090 gpu, 30 epochs (about 16 hours), conf/finetune_whisper_largev3_onlyattn.yaml
* Decoding info: ctc_weight 0.0, average_num 2
* Git hash: TBD

| decoding mode             | CER   |
|---------------------------|-------|
| attention decoder         | 2.57 % N=104765 C=102142 S=2529 D=94 I=74 |
| ctc greedy search         | N/A |
| ctc prefix beam search    | N/A |
| attention rescoring       | N/A |

* Feature info: using log_mel_spectrogram feature, no cmvn, no speed perturb
* Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 1, 8 * 3090 gpu, 80 epochs (about 43 hours), conf/finetune_whisper_largev3.yaml
* Decoding info: ctc_weight 0.3, average_num 5
* Git hash: TBD

| decoding mode             | CER   |
|---------------------------|-------|
| attention decoder         | 2.78 % N=104765 C=101943 S=2711 D=111 I=87  |
| ctc greedy search         | 6.89 % N=104765 C=98386 S=6210 D=169 I=839  |
| ctc prefix beam search    | 6.86 % N=104765 C=98410 S=6194 D=161 I=830  |
| attention rescoring       | 5.00 % N=104765 C=99771 S=4874 D=120 I=245  |

## Whisper-largev3 (conv2d4, full-parameter tuning) Result

* Feature info: using log_mel_spectrogram feature, no cmvn
* Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 1, 8 * 3090 gpu, 30 epochs (about 14 hours), conf/finetune_whisper_largev3_conv2d4_onlyattn.yaml
* Decoding info: ctc_weight 0.0, average_num 2
* Git hash: TBD

| decoding mode             | CER   |
|---------------------------|-------|
| attention decoder         | 2.63 % N=104765 C=102088 S=2579 D=98 I=79  |
| ctc greedy search         | N/A |
| ctc prefix beam search    | N/A |
| attention rescoring       | N/A |

* Feature info: using log_mel_spectrogram feature, no cmvn, no speed perturb
* Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 1, 8 * 3090 gpu, 80 epochs (about 36 hours), conf/finetune_whisper_largev3_conv2d4.yaml
* Decoding info: ctc_weight 0.3, average_num 5
* Git hash: TBD

| decoding mode             | CER   |
|---------------------------|-------|
| attention decoder         | 3.38 % N=104765 C=101336 S=3305 D=124 I=111 |
| ctc greedy search         | 7.43 % N=104765 C=97759 S=6813 D=193 I=779  |
| ctc prefix beam search    | 7.42 % N=104765 C=97767 S=6806 D=192 I=777  |
| attention rescoring       | 5.65 % N=104765 C=99100 S=5538 D=127 I=259  |

# Frequently Asked Questions

- Q: Why are there so many insertion errors in the decoding results of CTC?
- A: Because Chinese characters are composed of multiple bytes, in Whisper's tokenizer, one Chinese character might be represented by multiple tokens (for example, 3 tokens). During the CTC decoding process, it's possible that only two of these tokens are decoded. This not only causes garbled text (see https://github.com/wenet-e2e/wenet/issues/2308 ) but also leads to insertion errors.

---

### Overview (`examples/aishell2/rnnt/README.md`)

# Performance Record

## U2++ Conformer Result

* Feature info: using fbank feature, dither, cmvn, oneline speed perturb
* Training info: lr 0.001, dynamic batch with max_frames_in_batch 15000, 4 gpu, acc_grad 1, 130 epochs
* Training weight info: transducer_weight 0.75,  ctc_weight 0.1, reverse_weight 0.30, average_num 30
* Predictor type: lstm

| decoding mode/chunk size  | full  | 16    |
|---------------------------|-------|-------|
| rnnt greedy search        | 6.44  | 7.09  |

---

### Overview (`examples/aishell2/s0/README.md`)

# Performance Record

## U2++ Conformer Result

* Feature info: using fbank feature, with cmvn, no speed perturb, dither
* Training info: lr 0.001, batch size 32, 8 gpus, acc_grad 1, 240 epochs, dither 1.0
* Decoding info: ctc_weight 0.1, reverse_weight 0.4, average_num 30
* Git hash: 5a1342312668e7a5abb83aed1e53256819cebf95

| decoding mode/chunk size  | full  | 16    |
|---------------------------|-------|-------|
| ctc greedy search         | 6.18  | 6.79  |
| ctc prefix beam search    | 6.20  | 6.80  |
| attention rescoring       | 5.39  | 5.78  |
| LM + attention rescoring  | 5.35  | 5.73  |

## U2++ Transformer Result

* Feature info: using fbank feature, with cmvn, no speed perturb
* Training info: lr 0.002, batch size 22, 8 gpus, acc_grad 1, 240 epochs, dither 0.0
* Decoding info: ctc_weight 0.1, reverse_weight 0.5, average_num 30
* Git hash: 5a1342312668e7a5abb83aed1e53256819cebf95

| decoding mode/chunk size  | full  | 16    |
|---------------------------|-------|-------|
| ctc greedy search         | 7.35  | 8.23  |
| ctc prefix beam search    | 7.36  | 8.23  |
| attention rescoring       | 6.09  | 6.70  |
| LM + attention rescoring  | 6.07  | 6.55  |

## Unified Conformer Result

* Feature info: using fbank feature, with cmvn, no speed perturb.
* Training info: lr 0.002, batch size 16, 8 gpus, acc_grad 1, 120 epochs, dither 1.0
* Decoding info: ctc_weight 0.5, average_num 20
* Git hash: 14d38085a8d966cf9e9577ffafc51d578dce954f

| decoding mode/chunk size  | full  | 16    | 8     | 4     |
|---------------------------|-------|-------|-------|-------|
| attention decoder         | 6.23  | 6.42  | 6.58  | 7.20  |
| ctc greedy search         | 6.98  | 7.75  | 8.21  | 9.91  |
| ctc prefix beam search    | 7.02  | 7.76  | 8.21  | 9.93  |
| attention rescoring       | 6.08  | 6.46  | 6.72  | 7.79  |
| LM + attention rescoring  | 5.87  | 6.37  | 6.47  | 6.61  |

## Unified Transformer Result

* Feature info: using fbank feature, with cmvn, no speed perturb.
* Training info: lr 0.002, batch size 22, 8 gpus, acc_grad 1, 180 epochs, dither 0.0
* Decoding info: ctc_weight 0.5, average_num 30
* Git hash: 14d38085a8d966cf9e9577ffafc51d578dce954f

| decoding mode/chunk size  | full  | 16    | 8     | 4     |
|---------------------------|-------|-------|-------|-------|
| attention decoder         | 6.71  | 7.08  | 7.17  | 7.40  |
| ctc greedy search         | 7.84  | 8.68  | 8.98  | 9.46  |
| ctc prefix beam search    | 7.86  | 8.68  | 8.98  | 9.45  |
| attention rescoring       | 6.71  | 7.31  | 7.51  | 7.85  |
| LM + attention rescoring  | 6.35  | 7.02  | 7.24  | 7.52  |

---

### Overview (`examples/aishell4/s0/README.md`)

# Performance Record

## Conformer Result

* Feature info: using fbank feature, cmvn, without speed perturb (not supported segments yet)
* Training info: lr 0.001, max_frames_in_batch 15000, 8 gpu, acc_grad 4, 100 epochs
* Decoding info: ctc_weight 0.5, average_num 30


| decoding mode       | Test WER |
|---------------------|----------|
| attention rescoring |  32.58%  |

## U2pp **Conformer Result

* Feature info: using fbank feature, cmvn
* Training info: lr 0.001, batchsize static16, 8 * 3090 gpu, gradient_checkpoint, 100 epochs, about 8 hours
* Decoding info: ctc_weight 0.5, average_num 30


### Aishell4-test

| decoding mode/chunk size  | full  | 16    |
|---------------------------|-------|-------|
| ctc greedy search         | 32.04 % N=180315 C=128615 S=39491 D=12209 I=6072  | 33.81 % N=180315 C=125431 S=42075 D=12809 I=6083  |
| ctc prefix beam search    | 32.00 % N=180315 C=128931 S=39643 D=11741 I=6318  | 33.77 % N=180315 C=125743 S=42266 D=12306 I=6316  |
| attention                 | 33.41 % N=180315 C=129609 S=39233 D=11473 I=9534  | 35.05 % N=180315 C=127206 S=41063 D=12046 I=10084 |
| attention rescoring       | 31.18 % N=180315 C=130334 S=38618 D=11363 I=6236  | 32.90 % N=180315 C=127223 S=41102 D=11990 I=6223  |

### WenetSpeech-test_meeting

| decoding mode/chunk size  | full  | 16    |
|---------------------------|-------|-------|
| ctc greedy search         | 22.15 % N=220358 C=174426 S=38162 D=7770 I=2872  | 24.13 % N=220358 C=170114 S=41554 D=8690 I=2934  |
| ctc prefix beam search    | 22.10 % N=220358 C=174708 S=38193 D=7457 I=3052  | 24.09 % N=220358 C=170391 S=41622 D=8345 I=3126  |
| attention                 | 22.55 % N=220358 C=175320 S=35853 D=9185 I=4653  | 24.15 % N=220358 C=172277 S=38164 D=9917 I=5142  |
| attention rescoring       | 21.35 % N=220358 C=176208 S=36911 D=7239 I=2886  | 23.27 % N=220358 C=172045 S=40211 D=8102 I=2965  |

---

### Overview (`examples/chime4/s0/README.md`)

# Performance Record

## Conformer Result

* Feature info: dither + specaug + speed perturb
* Training info: lr 0.0005, batch size 8, 1 gpu, acc_grad 4, 80 epochs
* Decoding info: average_num 10

|      decoding mode     | dt05_real_1ch | dt05_simu_1ch | et05_real_1ch | et05_simu_1ch |
|:----------------------:|:-------------:|:-------------:|:-------------:|:-------------:|
| ctc_prefix_beam_search |   19.06%      |   21.17%      |   28.39%      |    29.16%     |
|  attention_rescoring   |   17.92%      |   20.22%      |   27.40%      |    28.25%     |

---

### Overview (`examples/commonvoice/fr/README.md`)

# Performance Record
# Should be installed ffmpeg , pandas !!!
## Conformer Result

* Feature info: dither + specaug + speed perturb
* Training info: lr 0.0005, warmup_steps 20000 batch size 8, 3 gpu, 30 epochs
* Decoding info: average_num 20



|     decoding mode      | test (wer) |
| :--------------------: | :---------: |
|   ctc_greedy_search    |   16.12%    |
| ctc_prefix_beam_search |   16.07%    |
|       attention        |   13.56%    |
|  attention_rescoring   |   14.01%    |

---

### Overview (`examples/csj/s0/README.md`)

# Performance Record

## Conformer Result Bidecoder (large)


## Conformer Result

* Feature info: using fbank feature, cmvn, dither, online speed perturb
* Training info: train_conformer.yaml, kernel size 15, lr 0.004, batch size 12, 8 gpu, acc_grad 1, 50 epochs, dither 0.0
* Decoding info: ctc_weight 0.5, average_num 10


| decoding mode                    | test1      | test2      | test3      |
|----------------------------------|------------|------------|------------|
| ctc greedy search                | 7.94       | 5.29       | 6.10       |
| ctc prefix beam search           | 7.83+      | 5.28       | 6.08       |
| attention decoder                | 7.83       | 5.63       | 6.37       |
| attention rescoring              | 7.28+      | 4.81       | 5.44       |

note that "+" means we removed two <0.1s wav files in test1 before decoding.




## Conformer U2++ Result


## Conformer U2 Result

---

### Overview (`examples/gigaspeech/s0/README.md`)

# GigaSpeech
A Large, modern and evolving dataset for automatic speech recognition. More details about GigaSpeech can be found:  https://github.com/SpeechColab/GigaSpeech

# Performance Record

## Conformer bidecoder Result

* Feature info: using fbank feature, dither 1.0, cmvn, 16k
* Training info: conf/train_conformer_bidecoder.yaml, subsample 4, kernel size 31, lr 0.001, batch size 24, 8 gpu, acc_grad 4, 40 epochs
* Decoding info: ctc_weight 0.3, reverse_weight 0.5, average_num 10
* Git hash: 9a0c270f9f976d7e887f777690e6c358a45a1c27

### test set gigaspeech scoring

| SPKR      | # Snt |  # Wrd | Corr | Sub | Del | Ins | Err  | S.Err |
|-----------|-------|--------|------|-----|-----|-----|------|-------|
| Sum/Avg   | 19928 | 390656 | 91.4 | 6.4 | 2.2 | 2.0 | 10.6 | 63.1  |
|  Mean     | 152.1 | 2982.1 | 91.4 | 6.3 | 2.3 | 1.7 | 10.3 | 63.7  |
|  S.D.     | 142.2 | 2838.1 |  5.5 | 4.1 | 1.6 | 1.3 |  6.4 | 16.9  |
| Median    | 108.0 | 2000.0 | 93.0 | 5.1 | 2.0 | 1.3 |  8.4 | 64.6  |

### dev set gigaspeech scoring

| SPKR      | # Snt |  # Wrd | Corr | Sub | Del | Ins | Err  | S.Err |
|-----------|-------|--------|------|-----|-----|-----|------|-------|
| Sum/Avg   | 5715  | 127790 | 92.1 | 5.8 | 2.1 | 2.8 | 10.7 |  69.9 |
|  Mean     | 204.1 | 4563.9 | 92.9 | 5.2 | 1.9 | 2.0 |  9.1 |  69.4 |
|  S.D.     | 269.7 | 4551.6 |  3.4 | 2.7 | 0.9 | 1.7 |  4.6 |  15.9 |
| Median    | 151.5 | 3314.0 | 93.8 | 4.4 | 1.6 | 1.7 |  7.9 |  71.6 |

## Conformer U2++ Result

* Feature info: using fbank feature, dither 1.0, cmvn, 16k
* Training info: conf/train_u2++_conformer.yaml, subsample 6, kernel size 31, lr 0.001, batch size 28, 8 gpu, acc_grad 1, 50 epochs
* Decoding info: ctc_weight 0.3, reverse_weight 0.5, average_num 10
* Git hash: 9a0c270f9f976d7e887f777690e6c358a45a1c27

### test set gigaspeech scoring, full chunk (non-streaming)

| SPKR      | # Snt |  # Wrd | Corr | Sub | Del | Ins | Err  | S.Err |
|-----------|-------|--------|------|-----|-----|-----|------|-------|
| Sum/Avg   | 19928 | 390656 | 90.7 | 6.8 | 2.6 | 2.0 | 11.3 |  66.9 |
|  Mean     | 152.1 | 2982.1 | 90.6 | 6.8 | 2.7 | 1.6 | 11.1 |  67.1 |
|  S.D.     | 142.2 | 2838.1 |  5.8 | 4.3 | 1.9 | 1.2 |  6.7 |  16.5 |
| Median    | 108.0 | 2000.0 | 92.1 | 5.7 | 2.2 | 1.3 |  9.0 |  68.9 |

### test set gigaspeech scoring, chunk 8 (latency range from 0 to 480ms)

| SPKR      | # Snt |  # Wrd | Corr | Sub | Del | Ins | Err  | S.Err |
|-----------|-------|--------|------|-----|-----|-----|------|-------|
| Sum/Avg   | 19928 | 390656 | 89.6 | 7.5 | 2.9 | 2.0 | 12.5 |  70.1 |
|  Mean     | 152.1 | 2982.1 | 89.3 | 7.6 | 3.1 | 1.7 | 12.4 |  70.6 |
|  S.D.     | 142.2 | 2838.1 |  6.5 | 4.9 | 2.1 | 1.2 |  7.3 |  15.8 |
| Median    | 108.0 | 2000.0 | 91.1 | 6.3 | 2.5 | 1.4 | 10.2 |  72.2 |

## Conformer Result

* Feature info: using fbank feature, dither 1.0, no cmvn, 48k
* Training info: conf/train_conformer.yaml, kernel size 31, lr 0.001, batch size 24, 8 gpu, acc_grad 4, 30 epochs
* Decoding info: ctc_weight 0.5, average_num 5
* Git hash: 9a0c270f9f976d7e887f777690e6c358a45a1c27

### test set gigaspeech scoring

| SPKR          | # Snt |  # Wrd | Corr | Sub | Del | Ins | Err  | S.Err |
|---------------|-------|--------|------|-----|-----|-----|------|-------|
| Sum/Avg       | 19930 | 390744 | 90.8 | 6.9 | 2.3 | 2.0 | 11.2 | 65.1  |
| Mean          | 152.1 | 2982.8 | 90.6 | 6.9 | 2.5 | 1.7 | 11.1 | 65.7  |
| S.D.          | 142.3 | 2839.0 |  5.8 | 4.3 | 1.7 | 1.2 |  6.7 | 16.6  |
| Median        | 108.0 | 2000.0 | 92.5 | 5.6 | 2.1 | 1.3 |  9.1 | 65.9  |

---

### Overview (`examples/hkust/s0/README.md`)

# Performance Record

## Conformer Result (Old IO)

* Feature info: using fbank feature, with cmvn, with speed perturb.
* Training info: lr 0.002, batch size 16, 1 machines, 1*4 = 4 gpu, acc_grad 4, 240 epochs, dither 0.1
* Decoding info: ctc_weight 0.5, average_num 30

| decoding mode            |       |
|--------------------------|-------|
| attention decoder        | 21.9  |
| ctc greedy search        | 21.15 |
| ctc prefix beam search   | 21.13 |
| attention rescoring      | 20.47 |

## Conformer Result (New IO)

* Feature info: using fbank feature, with cmvn, with speed perturb.
* Training info: lr 0.002, batch size 16, 1 machines, 1*4 = 4 gpu, acc_grad 4, 133 epochs, dither 0.1
* Decoding info: ctc_weight 0.5, average_num 30

| decoding mode            |       |
|--------------------------|-------|
| attention decoder        | 21.42 |
| ctc greedy search        | 21.16 |
| ctc prefix beam search   | 21.18 |
| attention rescoring      | 20.42 |

---

### Overview (`examples/librispeech/rnnt/README.md`)

# Performance Record

## Conformer Bidecoder Transducer Result

* Feature info: using fbank feature, dither, cmvn, online speed perturb
* Training info: lr 0.001, dynamic batch with max_frames_in_batch 4000, 8 gpu, acc_grad 1, 60 epochs
* Training weight info: transducer_weight 0.75,  ctc_weight 0.1, reverse_weight 0.30, average_num 10
* Predictor type: lstm

| decoding mode         | dev_clean  | dev_other | test_clean | test_other |
|-----------------------|------------|-----------|------------|------------|
| rnnt_greedy_search    | 3.42%      | 8.99%     |    3.56%   |   9.15%    |
| rnnt_beam_search      | 3.35%      | 8.77%     |    3.45%   |   8.78%    |
| rnnt_beam_att_rescore | 3.25%      | 8.66%     |    3.41%   |   8.68%    |

Pretrained model: https://huggingface.co/yuekai/wenet-asr-librispeech-conformer-transducer-mtl/blob/main/exp/conformer_transducer/avg_10.pt

---

### Overview (`examples/librispeech/s0/README.md`)

# Performance Record

## Conformer Result Bidecoder (large)

* Encoder FLOPs(30s): 96,238,430,720, params: 85,709,704
* Feature info: using fbank feature, cmvn, dither, online speed perturb
* Training info: train_conformer_bidecoder_large.yaml, kernel size 31, lr 0.002, batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 1.0
* Decoding info: ctc_weight 0.3, reverse weight 0.5, average_num 30
* Git hash: 65270043fc8c2476d1ab95e7c39f730017a670e0
* LM-tgmed: [3-gram.pruned.1e-7.arpa.gz](http://www.openslr.org/resources/11/3-gram.pruned.1e-7.arpa.gz)
* LM-tglarge: [3-gram.arpa.gz](http://www.openslr.org/resources/11/3-gram.arpa.gz)
* LM-fglarge: [4-gram.arpa.gz](http://www.openslr.org/resources/11/4-gram.arpa.gz)

| decoding mode                    | test clean | test other |
|----------------------------------|------------|------------|
| ctc prefix beam search           | 2.96       | 7.14       |
| attention rescoring              | 2.66       | 6.53       |
| LM-tgmed + attention rescoring   | 2.78       | 6.32       |
| LM-tglarge + attention rescoring | 2.68       | 6.10       |
| LM-fglarge + attention rescoring | 2.65       | 5.98       |

## SqueezeFormer Result (U2++, FFN:2048)

* Encoder info:
    * SM12, reduce_idx 5, recover_idx 11, conv1d, batch_norm, syncbn
    * encoder_dim 512, output_size 512, head 8, ffn_dim 512*4=2048
    * Encoder FLOPs(30s): 82,283,704,832, params: 85,984,648
* Feature info:
    * using fbank feature, cmvn, dither, online speed perturb, spec_aug
* Training info:
    * train_squeezeformer_bidecoder_large.yaml, kernel size 31
    * batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 1.0
    * adamw, lr 8e-4, NoamHold, warmup 0.2, hold 0.3, lr_decay 1.0
* Decoding info:
    * ctc_weight 0.3, reverse weight 0.5, average_num 30

| decoding mode                    | dev clean | dev other | test clean | test other |
|----------------------------------|-----------|-----------|------------|------------|
| ctc greedy search                | 2.55      | 6.62      | 2.73       | 6.59       |
| ctc prefix beam search           | 2.53      | 6.60      | 2.72       | 6.52       |
| attention decoder                | 2.93      | 6.56      | 3.31       | 6.47       |
| attention rescoring              | 2.19      | 6.06      | 2.45       | 5.85       |

## Conformer Result

* Encoder FLOPs(30s): 34,085,088,512, params: 34,761,608
* Feature info: using fbank feature, cmvn, dither, online speed perturb
* Training info: train_conformer.yaml, kernel size 31, lr 0.004, batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 0.1
* Decoding info: ctc_weight 0.5, average_num 30
* Git hash: 90d9a559840e765e82119ab72a11a1f7c1a01b78
* LM-fglarge: [4-gram.arpa.gz](http://www.openslr.org/resources/11/4-gram.arpa.gz)

| decoding mode                    | test clean | test other |
|----------------------------------|------------|------------|
| ctc greedy search                | 3.51       | 9.57       |
| ctc prefix beam search           | 3.51       | 9.56       |
| attention decoder                | 3.05       | 8.36       |
| attention rescoring              | 3.18       | 8.72       |
| attention rescoring (beam 50)    | 3.12       | 8.55       |
| LM-fglarge + attention rescoring | 3.09       | 7.40       |

## Conformer Result (12 layers, FFN:2048)
* Encoder FLOPs(30s): 34,085,088,512, params: 34,761,608
* Feature info: using fbank feature, cmvn, dither, online speed perturb
* Training info: train_squeezeformer.yaml, kernel size 31,
* batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 0.1
* AdamW, lr 1e-3, NoamHold, warmup 0.2, hold 0.3, lr_decay 1.0
* Decoding info: ctc_weight 0.3, reverse weight 0.5, average_num 30

| decoding mode                    | dev clean | dev other | test clean | test other |
|----------------------------------|-----------|-----------|------------|------------|
| ctc greedy search                | 3.49      | 9.59      | 3.66       | 9.59       |
| ctc prefix beam search           | 3.49      | 9.61      | 3.66       | 9.55       |
| attention decoder                | 3.52      | 9.04      | 3.85       | 8.97       |
| attention rescoring              | 3.10      | 8.91      | 3.29       | 8.81       |

## SqueezeFormer Result (SM12, FFN:1024)
* Encoder info:
    * SM12, reduce_idx 5, recover_idx 11, conv2d, w/o syncbn
    * encoder_dim 256, output_size 256, head 4, ffn_dim 256*4=1024
    * Encoder FLOPs(30s): 21,158,877,440, params: 22,219,912
* Feature info:
    * using fbank feature, cmvn, dither, online speed perturb
* Training info:
    * train_squeezeformer.yaml, kernel size 31,
    * batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 0.1
    * adamw, lr=1e-3, noamhold, warmup=0.2, hold=0.3, lr_decay=1.0
* Decoding info: ctc_weight 0.3, reverse weight 0.5, average_num 30

| decoding mode                    | dev clean | dev other | test clean | test other |
|----------------------------------|-----------|-----------|------------|------------|
| ctc greedy search                | 3.49      | 9.24      | 3.51       | 9.28       |
| ctc prefix beam search           | 3.44      | 9.23      | 3.51       | 9.25       |
| attention decoder                | 3.59      | 8.74      | 3.75       | 8.70       |
| attention rescoring              | 2.97      | 8.48      | 3.07       | 8.44       |

## SqueezeFormer Result (SM12, FFN:2048)
* Encoder info:
    * SM12, reduce_idx 5, recover_idx 11, conv2d, w/o syncbn
    * encoder_dim 256, output_size 256, head 4, ffn_dim 256*8=2048
    * encoder FLOPs(30s): 28,230,473,984, params: 34,827,400
* Feature info: using fbank feature, cmvn, dither, online speed perturb
* Training info:
    * train_squeezeformer.yaml, kernel size 31
    * batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 0.1
    * adamw, lr 1e-3, noamhold, warmup 0.2, hold 0.3, lr_decay 1.0
* Decoding info:
    * ctc_weight 0.3, reverse weight 0.5, average_num 30

| decoding mode                    | dev clean | dev other | test clean | test other |
|----------------------------------|-----------|-----------|------------|------------|
| ctc greedy search                | 3.34      | 9.01      | 3.47       | 8.85       |
| ctc prefix beam search           | 3.33      | 9.02      | 3.46       | 8.81       |
| attention decoder                | 3.64      | 8.62      | 3.91       | 8.33       |
| attention rescoring              | 2.89      | 8.34      | 3.10       | 8.03       |

## SqueezeFormer Result (SM12, FFN:1312)
* Encoder info:
    * SM12, reduce_idx 5, recover_idx 11, conv1d, w/o syncbn
    * encoder_dim 328, output_size 256, head 4, ffn_dim 328*4=1312
    * encoder FLOPs(30s): 34,103,960,008, params: 35,678,352
* Feature info:
    * using fbank feature, cmvn, dither, online speed perturb
* Training info:
    * train_squeezeformer.yaml, kernel size 31,
    * batch size 12, 8 gpu, acc_grad 4, 120 epochs, dither 1.0
    * adamw, lr 1e-3, noamhold, warmup 0.2, hold 0.3, lr_decay 1.0
* Decoding info:
    * ctc_weight 0.3, reverse weight 0.5, average_num 30

| decoding mode                    | dev clean | dev other | test clean | test other |
|----------------------------------|-----------|-----------|------------|------------|
| ctc greedy search                | 3.20      | 8.46      | 3.30       | 8.58       |
| ctc prefix beam search           | 3.18      | 8.44      | 3.30       | 8.55       |
| attention decoder                | 3.38      | 8.31      | 3.89       | 8.32       |
| attention rescoring              | 2.81      | 7.86      | 2.96       | 7.91       |

## Conformer U2++ Result

* Feature info: using fbank feature, cmvn, no speed perturb, dither
* Training info: train_u2++_conformer.yaml lr 0.001, batch size 24, 8 gpu, acc_grad 1, 120 epochs, dither 1.0
* Decoding info: ctc_weight 0.3,  reverse weight 0.5, average_num 30
* Git hash: 65270043fc8c2476d1ab95e7c39f730017a670e0

test clean

| decoding mode                  | full | 16   |
|--------------------------------|------|------|
| ctc prefix beam search         | 3.76 | 4.54 |
| attention rescoring            | 3.32 | 3.80 |

test other

| decoding mode                  | full  | 16    |
|--------------------------------|-------|-------|
| ctc prefix beam search         | 9.50  | 11.52 |
| attention rescoring            | 8.67  | 10.38 |

## SqueezeFormer Result (U2++, FFN:2048)

* Encoder info:
    * SM12, reduce_idx 5, recover_idx 11, conv1d, layer_norm
    * do_rel_shift false, warp_for_time, syncbn
    * encoder_dim 256, output_size 256, head 4, ffn_dim 256*8=2048
    * Encoder FLOPs(30s): 28,255,337,984, params: 34,893,704
* Feature info:
    * using fbank feature, cmvn, dither, online speed perturb
* Training info:
    * train_squeezeformer.yaml, kernel size 31
    * batch size 12, 8 gpu, acc_grad 2, 120 epochs, dither 1.0
    * adamw, lr 8e-4, NoamHold, warmup 0.2, hold 0.3, lr_decay 1.0
* Decoding info:
    * ctc_weight 0.3, reverse weight 0.5, average_num 30

test clean

| decoding mode                  | full | 16   |
|--------------------------------|------|------|
| ctc prefix beam search         | 3.45 | 4.34 |
| attention rescoring            | 3.07 | 3.71 |

test other

| decoding mode                  | full  | 16    |
|--------------------------------|-------|-------|
| ctc prefix beam search         | 8.29  | 10.60 |
| attention rescoring            | 7.58  | 9.60  |

## Branchformer U2++ Result

* Feature info: using fbank feature, cmvn, online speed perturb, dither
* Encoder info: layer num 24, cnn_kernel size 63, cgmlp linear units: 2048
* Training info: train_u2++_branchformer.yaml lr 0.001, batch size 16, 8 gpu, acc_grad 1, 120 epochs, dither 1.0
* Decoding info: ctc_weight 0.3,  reverse weight 0.5, average_num 30

test clean

| decoding mode                  | full | 16   |
|--------------------------------|------|------|
| ctc prefix beam search         | 3.78 | 4.60 |
| attention rescoring            | 3.33 | 3.83 |

test other

| decoding mode                  | full  | 16    |
|--------------------------------|-------|-------|
| ctc prefix beam search         | 9.51  | 11.50 |
| attention rescoring            | 8.76  | 10.34 |


## Conformer U2 Result

* Feature info: using fbank feature, cmvn, speed perturb, dither
* Training info: train_unified_conformer.yaml lr 0.001, batch size 10, 8 gpu, acc_grad 1, 120 epochs, dither 1.0
* Decoding info: ctc_weight 0.5, average_num 30
* Git hash: 90d9a559840e765e82119ab72a11a1f7c1a01b78
* LM-tgmed: [3-gram.pruned.1e-7.arpa.gz](http://www.openslr.org/resources/11/3-gram.pruned.1e-7.arpa.gz)
* LM-tglarge: [3-gram.arpa.gz](http://www.openslr.org/resources/11/3-gram.arpa.gz)
* LM-fglarge: [4-gram.arpa.gz](http://www.openslr.org/resources/11/4-gram.arpa.gz)

test clean

| decoding mode                    | full | 16   |
|----------------------------------|------|------|
| ctc prefix beam search           | 4.26 | 5.00 |
| attention decoder                | 3.05 | 3.44 |
| attention rescoring              | 3.72 | 4.10 |
| attention rescoring (beam 50)    | 3.57 | 3.95 |
| LM-tgmed + attention rescoring   | 3.56 | 4.02 |
| LM-tglarge + attention rescoring | 3.40 | 3.82 |
| LM-fglarge + attention rescoring | 3.38 | 3.74 |

test other

| decoding mode                    | full  | 16    |
|----------------------------------|-------|-------|
| ctc prefix beam search           | 10.87 | 12.87 |
| attention decoder                | 9.07  | 10.44 |
| attention rescoring              | 9.74  | 11.61 |
| attention rescoring (beam 50)    | 9.34  | 11.13 |
| LM-tgmed + attention rescoring   | 8.78  | 10.26 |
| LM-tglarge + attention rescoring | 8.34  | 9.74  |
| LM-fglarge + attention rescoring | 8.17  | 9.44  |


## Efficient Conformer V1 Result

* Feature info:
    * using fbank feature, cmvn, speed perturb, dither
* Training info:
    * train_u2++_efficonformer_v1.yaml
    * 8 gpu, batch size 16, acc_grad 1, 120 epochs
    * lr 0.001, warmup_steps 35000
* Model info:
    * Model Params: 49,474,974
    * Downsample rate: 1/4 (conv2d) * 1/2 (efficonformer block)
    * encoder_dim 256, output_size 256, head 8, linear_units 2048
    * num_blocks 12, cnn_module_kernel 15, group_size 3
* Decoding info:
    * ctc_weight 0.5, reverse_weight 0.3, average_num 20
* Model Download: [wenet_efficient_conformer_librispeech_v1](https://huggingface.co/58AILab/wenet_efficient_conformer_librispeech_v1)

test clean

| decoding mode          | full | 18   | 16   |
|------------------------|------|------|------|
| attention decoder      | 3.65 | 3.88 | 3.87 |
| ctc_greedy_search      | 3.46 | 3.79 | 3.77 |
| ctc prefix beam search | 3.44 | 3.75 | 3.74 |
| attention rescoring    | 3.17 | 3.44 | 3.41 |

test other

| decoding mode          | full | 18    | 16    |
|------------------------|------|-------|-------|
| attention decoder      | 8.51 | 9.24  | 9.25  |
| ctc_greedy_search      | 8.94 | 10.04 | 10.06 |
| ctc prefix beam search | 8.91 | 10    | 10.01 |
| attention rescoring    | 8.21 | 9.25  | 9.25  |


## Efficient Conformer V2 Result

* Feature info:
    * using fbank feature, cmvn, speed perturb, dither
* Training info:
    * train_u2++_efficonformer_v2.yaml
    * 8 gpu, batch size 16, acc_grad 1, 120 epochs
    * lr 0.001, warmup_steps 35000
* Model info:
    * Model Params: 50,341,278
    * Downsample rate: 1/2 (conv2d2) * 1/4 (efficonformer block)
    * encoder_dim 256, output_size 256, head 8, linear_units 2048
    * num_blocks 12, cnn_module_kernel 15, group_size 3
* Decoding info:
    * ctc_weight 0.5, reverse_weight 0.3, average_num 20
* Model Download: [wenet_efficient_conformer_librispeech_v2](https://huggingface.co/58AILab/wenet_efficient_conformer_librispeech_v2)

test clean

| decoding mode          | full | 18   | 16   |
|------------------------|------|------|------|
| attention decoder      | 3.49 | 3.71 | 3.72 |
| ctc_greedy_search      | 3.49 | 3.74 | 3.77 |
| ctc prefix beam search | 3.47 | 3.72 | 3.74 |
| attention rescoring    | 3.12 | 3.38 | 3.36 |

test other

| decoding mode          | full | 18   | 16   |
|------------------------|------|------|------|
| attention decoder      | 8.15 | 9.05 | 9.03 |
| ctc_greedy_search      | 8.73 | 9.82 | 9.83 |
| ctc prefix beam search | 8.70 | 9.81 | 9.79 |
| attention rescoring    | 8.05 | 9.08 | 9.10 |

---

### Overview (`examples/multi_cn/s0/README.md`)

# Performance Record

This is a Chinese speech recognition recipe that trains on all Chinese corpora including:

| Dataset    | Duration (Hours) |
|------------|------------------|
| Aidatatang | 140              |
| Aishell    | 151              |
| MagicData  | 712              |
| Primewords | 99               |
| ST-CMDS    | 110              |
| THCHS-30   | 26               |
| TAL-ASR    | 587              |
| AISHELL2   | 1000             |

## Unified Transformer Result

### Data info:

* Dataset: Aidatatang, Aishell, MagicData, Primewords, ST-CMDS, and THCHS-30.
* Feature info: using fbank feature, with cmvn, no speed perturb.
* Training info: lr 0.004, batch size 18, 3 machines, 3*8 = 24 GPUs, acc_grad 1, 220 epochs, dither 0.1
* Decoding info: ctc_weight 0.5, average_num 30
* Git hash: 013794572a55c7d0dbea23a66106ccf3e5d3b8d4

### WER

| Dataset    | chunk size | attention decoder | ctc greedy search | ctc prefix beam search | attention rescoring |
|------------|------------|-------------------|-------------------|------------------------|---------------------|
| Aidatatang | full       | 4.23              | 5.82              | 5.82                   | 4.71                |
|            | 16         | 4.59              | 6.99              | 6.99                   | 5.29                |
| Aishell    | full       | 4.69              | 5.80              | 5.80                   | 4.64                |
|            | 16         | 4.97              | 6.75              | 6.75                   | 5.37                |
| MagicData  | full       | 2.86              | 4.01              | 4.00                   | 3.07                |
|            | 16         | 3.10              | 5.02              | 5.02                   | 3.68                |
| THCHS-30   | full       | 16.68             | 15.46             | 15.46                  | 14.38               |
|            | 16         | 17.47             | 16.81             | 16.82                  | 15.63               |

## Unified Conformer Result

### Data info:

* Dataset: Aidatatang, Aishell, MagicData, Primewords, ST-CMDS, and THCHS-30.
* Feature info: using fbank feature, with cmvn, speed perturb.
* Training info: lr 0.001, batch size 8, 1 machines, 1*8 = 8 GPUs, acc_grad 12, 60 epochs
* Decoding info: ctc_weight 0.5, average_num 10
* Git hash: 5bdf436e671ef4c696d1b039f29cc33109e072fa

### WER

| Dataset    | chunk size | attention decoder | ctc greedy search | ctc prefix beam search | attention rescoring |
|------------|------------|-------------------|-------------------|------------------------|---------------------|
| Aidatatang | full       | 4.12              | 4.97              | 4.97                   | 4.22                |
|            | 16         | 4.45              | 5.73              | 5.73                   | 4.75                |
| Aishell    | full       | 4.49              | 5.07              | 5.05                   | 4.43                |
|            | 16         | 4.77              | 5.77              | 5.77                   | 4.85                |
| MagicData  | full       | 2.55              | 3.07              | 3.05                   | 2.59                |
|            | 16         | 2.81              | 3.88              | 3.86                   | 3.08                |
| THCHS-30   | full       | 13.55             | 13.75             | 13.76                  | 12.72               |
|            | 16         | 13.78             | 15.10             | 15.08                  | 13.90               |

## Unified Conformer Result

### Data info:

* Dataset: Aidatatang, Aishell, MagicData, Primewords, ST-CMDS, THCHS-30, TAL-ASR, and AISHELL2.
* Feature info: using fbank feature, dither=0, cmvn, speed perturb
* Training info: lr 0.001, batch size 22, 4 GPUs, acc_grad 4, 120 epochs, dither 0.1
* Decoding info: ctc_weight 0.5, average_num 10
* Git hash: 66f30c197d00c59fdeda3bc8ada801f867b73f78

### WER

| Dataset    | chunk size | attention decoder | ctc greedy search | ctc prefix beam search | attention rescoring |
|------------|------------|-------------------|-------------------|------------------------|---------------------|
| Aidatatang | full       | 3.22              | 4.00              | 4.01                   | 3.35                |
|            | 16         | 3.50              | 4.63              | 4.63                   | 3.79                |
| Aishell    | full       | 1.23              | 2.12              | 2.13                   | 1.42                |
|            | 16         | 1.33              | 2.72              | 2.72                   | 1.72                |
| MagicData  | full       | 2.38              | 3.07              | 3.05                   | 2.52                |
|            | 16         | 2.66              | 3.80              | 3.78                   | 2.94                |
| THCHS-30   | full       | 9.93              | 11.07             | 11.06                  | 10.16               |
|            | 16         | 10.28             | 11.85             | 11.85                  | 10.81               |
| AISHELL2   | full       | 5.25              | 5.81              | 5.79                   | 5.22                |
|            | 16         | 5.48              | 6.48              | 6.50                   | 5.61                |
| TAL-ASR    | full       | 9.54              | 10.35             | 10.28                  | 9.66                |
|            | 16         | 10.04             | 11.43             | 11.39                  | 10.55               |

---

### Overview (`examples/openasr2021/s0/README.md`)

# w2v-conformer based end-to-end model for Openasr2021 challenge

This is a example to use unsupervised pretrained w2v-conformer model to fintune [OpenASR2021](https://www.nist.gov/itl/iad/mig/openasr-challenge) constrained-plus tasks.

We pretrain conformer encoders using wav2vec 2.0 pre-training method , which we called ch-w2v-conformer. The original pre-training works take raw waveforms
as input. Unlike these works, we use MFCC features as inputs.

The ch-w2v-conformer model uses following datasets to pretrain:

ISML datasets (6 languages,70k hours): internal dataset contains 40k hours Chinese, Cantonese, Tibetan, Inner Mongolian, Inner Kazakh, Uighur.

Babel datasets (17 languages, 2k hours): Assamese, Bengali, Cantonese, Cebuano, Georgian, Haitian, Kazakh, Kurmanji, Lao, Pashto, Swahili, Tagalog, Tamil, Tok, Turkish, Vietnamese, Zulu

After pretraining, we build ASR system based on CTC-Attention structure. In very low resource task, we find that if too many initialization network structures are constructed in the upper layer of pre-training conformer encoder, the migration performance of the pre-training model will be destroyed, so we only build a single-layer transformer decoder for joint training.

pretrained model link: https://huggingface.co/emiyasstar/ch-w2v-conformer


## constrained-plus Task Performance

* Languages: Cantonese,mongolian,kazakh
* config: conf/train_conformer_large_10h.yaml
* Feature info: using mfcc feature, with dither 1.0, without cmvn
* Training info: lr 0.001, batch size 10, 4 gpus on V100, acc_grad 1, 80 epochs
* Decoding info: ctc_weight 0.5, average_num 35

dev set results trained only with 10 hours training set

## w2v-Conformer

|   decoding_method   | Cantonese(CER)  | mongolian(WER) |
|:-------------------:|:----:|:----:|
|  ctc_greedy_search  | 31.46 | 53.64 |
|  ctc_prefix_search |  31.47   | 53.50 |
| attention_rescoring | 31.45 |  52.96 |

## Conformer （train from scratch）


|   decoding_method   |  Cantonese(CER)  | mongolian(WER) |
|:-------------------:|----:|:----:|
|  ctc_greedy_search  | 61.43 | 89.38 |
|  ctc_prefix_search |  61.37   | 89.53|
| attention_rescoring | 60.61 | 89.60|

---

### Overview (`examples/swbd/s0/README.md`)

# Performance Record

## Conformer Result

* Feature info: dither + specaug + speed perturb
* Training info: lr 0.001, warmup_steps 25000, batch size 16, 1 gpu, acc_grad 4, 240 epochs
* Decoding info: average_num 10

|      decoding mode     |   eval2000 (wer) |
|:----------------------:|:----------------:|
|   ctc_greedy_search    |       32.39%     |
| ctc_prefix_beam_search |       32.39%     |
|         attention      |       31.28%     |
|  attention_rescoring   |       31.36%     |

---

### Overview (`examples/tedlium3/s0/README.md`)

# Performance Record

## Conformer Result

* Feature info: using fbank feature, dither, cmvn, without speed perturb (not supported segments yet)
* Training info: lr 0.001, batch size 20, 8 gpu, acc_grad 1, 240 epochs, dither 0.1
* Decoding info: ctc_weight 0.5, average_num 10


| decoding mode       | Dev WER | Test WER |
|---------------------|---------|----------|
| attention rescoring | 9.54%   | 8.66%    |

---

### Overview (`examples/timit/s0/README.md`)

# Performance Record

## Conformer Result

* Feature info: dither + specaug + speed perturb
* Training info: lr 0.002, warmup_steps 5000 batch size 16, 1 gpu, acc_grad 4, 120 epochs
* Decoding info: average_num 20
* trans_type: phn


|     decoding mode      | test (wer) |
| :--------------------: | :---------: |
|   ctc_greedy_search    |   16.70%    |
| ctc_prefix_beam_search |   16.60%    |
|       attention        |   22.37%    |
|  attention_rescoring   |   16.60%    |

## transformer Result

* Feature info: dither + specaug + speed perturb
* Training info: lr 0.002, warmup_steps 5000 batch size 16, 1 gpu, acc_grad 4, 120 epochs
* Decoding info: average_num 20
* trans_type: phn


|     decoding mode      | test (wer) |
| :--------------------: | :---------: |
|   ctc_greedy_search    |   17.78%    |
| ctc_prefix_beam_search |   17.46%    |
|       attention        |   21.77%    |
|  attention_rescoring   |   17.06%    |

---

### Overview (`examples/vkw2021/s0/README.md`)

# conformer based end-to-end model for VKW challenge

## Standard E2E Results

Conformer without speed perpurb and lm
* config: conf/train_train_vkw_bidirect_12conformer_hs2048_output256_att4_conv2d_char.yaml
* beam: 10
* num of gpu: 8
* num of averaged model: 5
* ctc weight (used for attention rescoring): 0.5

dev set results trained only with training set (785 keywords, 1505 hour train set)

| scenario | Precision | Recall   | F1     | ATWV   |
|----------|-----------|----------|--------|--------|
| lgv      | 0.9281    | 0.6420   | 0.7590 | 0.5183 |
| liv      | 0.8886    | 0.6515   | 0.7518 | 0.6050 |
| stv      | 0.9120    | 0.7471   | 0.8213 | 0.6256 |

dev set results trained with training set and finetune set (785 keywords, 1505 hour train set + 15 hour finetune set)

| scenario | Precision | Recall   | F1     | ATWV   |
|----------|-----------|----------|--------|--------|
| lgv      | 0.9478    | 0.7311   | 0.8255 | 0.6352 |
| liv      | 0.9177    | 0.8398   | 0.8770 | 0.7412 |
| stv      | 0.9320    | 0.8207   | 0.8729 | 0.7120 |

test set results trained only with training set (384 keywords, 1505 hour train set)

| scenario | Precision | Recall   | F1     | ATWV   |
|----------|-----------|----------|--------|--------|
| lgv      | 0.6262    | 0.5648   | 0.5939 | 0.5825 |
| liv      | 0.8797    | 0.6282   | 0.7330 | 0.6061 |
| stv      | 0.9102    | 0.7221   | 0.8053 | 0.6682 |

test set results trained with training set and finetune set (384 keywords, 1505 hour train set + 15 hour finetune set)

| scenario | Precision | Recall   | F1     | ATWV   |
|----------|-----------|----------|--------|--------|
| lgv      | 0.6469    | 0.6276   | 0.6371 | 0.6116 |
| liv      | 0.9278    | 0.7560   | 0.8331 | 0.6927 |
| stv      | 0.9434    | 0.8061   | 0.8693 | 0.7275 |

---

### Overview (`examples/wenetspeech/paraformer/README.md`)

# Preliminary
1. Run below command to convert funasr-style  ckpt to wenet-style ckpt:
```sh
output_dir=exp/paraformer/large
mkdir -p ${output_dir}
. ./path.sh && python wenet/paraformer/convert_paraformer_to_wenet_config_and_ckpt.py \
  --output_dir ${output_dir}
```

# Performance Record

## Paraformer (original) Result

| decoding mode             |  CER(DEV)  | CER(TEST\_NET) | CER(TEST\_MEETING) | CER(AISHELL4-TEST) |
|---------------------------|-------|-------|-------|-------|
| ctc greedy search         |  N/A  |  N/A  |  N/A  |  N/A  |
| ctc prefix beam search    |  N/A  |  N/A  |  N/A  |  N/A  |
| paraformer greedy search  | 3.38 % N=328207 C=319078 S=8045 D=1084 I=1959  | 6.74 % N=414285 C=388817 S=20119 D=5349 I=2444 | 6.95 % N=220358 C=206461 S=9090 D=4807 I=1422 | 18.37 % N=180315 C=156112 S=17714 D=6489 I=8923 |

## Paraformer-dynamic training (full-parameter tuning) Result

* Training info: lr 0.0005, batch size dynamic36000, gradient checkpointing, torch_ddp, acc_grad 4, 8 * 3090 gpus, 20 epochs, about 4 days
* Decoding info: ctc_weight 0.5, average_num 3, blank penalty 2.5 for dev and 0.0 for others
* Git hash: TBD

| decoding mode             |  CER(DEV)  | CER(TEST\_NET) | CER(TEST\_MEETING) | CER(AISHELL4-TEST) |
|---------------------------|-------|-------|-------|-------|
| ctc greedy search         | 7.44 % N=328207 C=307725 S=13700 D=6782 I=3925 | 10.53 % N=414285 C=372693 S=28860 D=12732 I=2013 | 9.60 % N=220358 C=200727 S=12091 D=7540 I=1527 | 20.24 % N=180315 C=149290 S=18264 D=12761 I=5467  |
| ctc prefix beam search    | 7.18 % N=328207 C=308288 S=13896 D=6023 I=3636 | 10.34 % N=414285 C=373390 S=29103 D=11792 I=1929 | 9.26 % N=220358 C=201542 S=12250 D=6566 I=1587 | 19.83 % N=180315 C=150352 S=18592 D=11371 I=5796  |
| paraformer greedy search  | 6.63 % N=328207 C=308518 S=8688 D=11001 I=2077 | 7.71 % N=414285 C=384845 S=20587 D=8853 I=2483 | 7.34 % N=220358 C=205924 S=9283 D=5151 I=1751 | 18.04 % N=180315 C=155176 S=17420 D=7719 I=7386 |


### NOTE

This is our first attempt at fine-tuning the paraformer-large to enable stream inference through a wenet-like chunk method.
Although the non-streaming results deteriorated after fine-tuning compared to before, we believe there is still significant room for improvement for paraformer-large when fine-tuned within wenet, considering this is a very initial result.

Additionally, on the same training set (wenetspeech+aishell4), we trained a conformer-large model from scratch (see experimental results in examples/aishell/s0). Comparing it with the fine-tuned results of paraformer-large, we found that the CTC results of paraformer-large consistently outperformed those of conformer-large, and the NAR results of paraformer-large were always better than the rescore of conformer-large. This is mainly due to paraformer-large having been pre-trained on 6wh industrial data, giving the model a better initialization.

---

### Overview (`examples/wenetspeech/s0/README.md`)

# Performance Record

## Conformer

* Feature info: using fbank feature, with dither 1.0, with cmvn
* Training info: lr 0.001, batch size 32, 24 gpus on V100, acc_grad 16, 26 epochs
* Decoding info: ctc_weight 0.5, average_num 10

|   decoding_method   |  Dev | Test\_Net | Test\_Meeting |
|:-------------------:|:----:|:---------:|:-------------:|
|  ctc_greedy_search  | 8.88 |   10.29   |     15.96     |
|      attention      | 9.38 |   10.12   |     17.28     |
| attention_rescoring | 8.69 |    9.7    |     15.59     |

## Conformer bidecoder

* Feature info: using fbank feature, with dither 1.0, with cmvn
* Training info: lr 0.001, batch size 32, 24 gpus on V100, acc_grad 16, 26 epochs
* Decoding info: ctc_weight 0.5, average_num 10

|   decoding_method   |  Dev | Test\_Net | Test\_Meeting |
|:-------------------:|:----:|:---------:|:-------------:|
|  ctc_greedy_search  | 8.98 |    9.55   |     16.48     |
|      attention      | 9.42 |   10.57   |     18.05     |
| attention_rescoring | 8.85 |    9.25   |     16.18     |

## U2++ conformer

* Feature info: using fbank feature, with dither 1.0, with cmvn
* Training info: lr 0.002, batch size dynamic24000, 24 gpus on 3090, acc_grad 16, 80 epochs, 4.5 days
* Decoding info: ctc_weight 0.5, reverse_weight 0.0, average_num 10, blank penalty 2.5, length penalty 8.5 for dev/testmeeting and 0.0 for testnet

| Decoding mode - Chunk size    | Dev  | Test\_Net | Test\_Meeting |
|:-----------------------------:|:----:|:---------:|:-------------:|
| ctc prefix beam search - full      | 7.21 % N=328207 C=309358 S=14175 D=4674 I=4801 | 9.46 % N=414285 C=381373 S=26013 D=6899 I=6295 | 14.02 % N=220358 C=195224 S=17266 D=7868 I=5754 |
| ctc prefix beam search - 16        | 7.93 % N=328207 C=307192 S=16529 D=4486 I=5000 | 11.14 % N=414285 C=374733 S=30241 D=9311 I=6596 | 16.37 % N=220358 C=191394 S=22435 D=6529 I=7116 |
| attention rescoring - full    | 7.10 % N=328207 C=308457 S=13215 D=6535 I=3537 | 8.83 % N=414285 C=381936 S=24808 D=7541 I=4215 | 13.64 % N=220358 C=194438 S=16238 D=9682 I=4133 |
| attention rescoring - 16      | 7.57 % N=328207 C=307065 S=15169 D=5973 I=3687 | 10.13 % N=414285 C=376854 S=28486 D=8945 I=4541 | 15.55 % N=220358 C=191270 S=21136 D=7952 I=5184 |
| attention - full    | 7.73 % N=328207 C=306688 S=13166 D=8353 I=3845 | 9.44 % N=414285 C=378096 S=24532 D=11657 I=2908 | 14.98 % N=220358 C=191881 S=15303 D=13174 I=4540 |

## U2++ conformer (text\_fixed, see https://github.com/wenet-e2e/WenetSpeech/discussions/54)

* Feature info: using fbank feature, with dither 1.0, with cmvn
* Training info: lr 0.001, batch size dynamic36000, 8 gpus on 3090, acc_grad 4, 130k steps, 4.6 days
* Decoding info: ctc_weight 0.5, reverse_weight 0.0, average_num 5, blank penalty 0.0, length penalty 0.0
* PR link: https://github.com/wenet-e2e/wenet/pull/2371

| Decoding mode - Chunk size    | Dev  | Test\_Net | Test\_Meeting |
|:-----------------------------:|:----:|:---------:|:-------------:|
| ctc prefix beam search - full      | 6.26 % N=328207 C=310671 S=15612 D=1924 I=3002 | 9.46 % N=414285 C=381373 S=26013 D=6899 I=6295 | 12.52 % N=220358 C=194801 S=19209 D=6348 I=2042 |
| attention rescoring - full    | 5.90 % N=328207 C=311721 S=14597 D=1889 I=2888 | 8.96 % N=414092 C=380232 S=27606 D=6254 I=3222 | 11.99 % N=220358 C=195808 S=18243 D=6307 I=1878 |
| attention - full    | 5.87 % N=328207 C=311922 S=14204 D=2081 I=2987 | 8.87 % N=414092 C=381014 S=27359 D=5719 I=3650 | 11.79 % N=220358 C=196484 S=17378 D=6496 I=2108 |

## U2++ conformer (wenetspeech plus aishell4)

* Feature info: using fbank feature, with dither 1.0, with cmvn
* Training info: lr 0.001, batch size dynamic36000, gradient checkpointing, torch_ddp, 8 * 3090 gpus, acc_grad 4, 60 epochs, about 8.5 days
* Decoding info: ctc_weight 0.5, reverse_weight 0.0, average_num 10, blank penalty 2.5 for dev and 0.0 for others

| Decoding mode - Chunk size    | Dev  | Test\_Net | Test\_Meeting |
|:-----------------------------:|:----:|:---------:|:-------------:|
| ctc prefix beam search - full      | 8.01 % N=328207 C=307477 S=15151 D=5579 I=5558 | 10.14 % N=414285 C=375271 S=27474 D=11540 I=2983 | 9.76 % N=220358 C=201205 S=13883 D=5270 I=2348 |
| attention rescoring - full    | 7.89 % N=328207 C=306307 S=13929 D=7971 I=3984 | 9.67 % N=414285 C=377058 S=25921 D=11306 I=2828 | 9.38 % N=220358 C=201833 S=13209 D=5316 I=2138 |

---

### Overview (`examples/wenetspeech/whisper/README.md`)

# Preliminary
1. Download whisper ckpt from this [link](https://github.com/openai/whisper/blob/main/whisper/__init__.py#L17-L30)

2. We assume you have run stage0~stage3 using `wenetspeech/s0/run.sh` and here we are simply creating a symbolic link
```sh
ln -s ../s0/data .
```

3. Run below command to convert openai-style ckpt to wenet-style ckpt:
```sh
mkdir -p exp
mkdir -p exp/whisper
mkdir -p exp/whisper/large-v3
. ./path.sh && python wenet/whisper/convert_whisper_to_wenet_config_and_ckpt.py \
  --whisper_ckpt downloaded-large-v3.pt \
  --output_dir exp/whisper/large-v3
# remove conv1d2
python local/modify_ckpt.py \
  --remove_list "encoder.embed.conv" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.remove-subsample.pt
# init ctc
python local/modify_ckpt.py \
  --add_list "{\"ctc.ctc_lo.weight\": \"decoder.embed.0.weight\"}" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.init-ctc.pt
# remove conv1d2 and init ctc
python local/modify_ckpt.py \
  --remove_list "encoder.embed.conv" \
  --add_list "{\"ctc.ctc_lo.weight\": \"decoder.embed.0.weight\"}" \
  --input_ckpt exp/whisper/large-v3/wenet_whisper.pt \
  --output_ckpt exp/whisper/large-v3/wenet_whisper.remove-subsample.init-ctc.pt
```

# Performance Record

## Whisper-largev2 (original) Result

|   decoding_method   |  Dev | Test\_Net | Test\_Meeting |
|:-------------------:|:----:|:---------:|:-------------:|
|  ctc_greedy_search  | N/A  |   N/A     |     N/A     |
|      attention      | N/A  |  10.99    |   26.32     |
| attention_rescoring | N/A  |    N/A    |     N/A     |

## Whisper-largev3 (conv1d2, full-parameter tuning) Result

* Feature info: using log_mel_spectrogram feature, no cmvn
* Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 8, 8 * 3090 gpu, 40k steps (about 5 days), conf/finetune_whisper_largev3.yaml
* Decoding info: ctc_weight 0.0, average_num 5
* PR link: https://github.com/wenet-e2e/wenet/pull/2356

|   decoding_method   |  Dev | Test\_Net | Test\_Meeting |
|:-------------------:|:----:|:---------:|:-------------:|
|  ctc_greedy_search  | 10.78 % N=328207 C=305525 S=18448 D=4234 I=12695  |   12.52 % N=414097 C=366245 S=34946 D=12906 I=4004     |    17.74 % N=220358 C=193382 S=20284 D=6692 I=12108     |
|      attention      | 7.27 % N=328207 C=308016 S=11392 D=8799 I=3672  |  7.90 % N=414097 C=383382 S=18954 D=11761 I=2018    |   13.00 % N=220358 C=194417 S=11788 D=14153 I=2705     |
| attention_rescoring | 8.95 % N=328207 C=305892 S=16696 D=5619 I=7056  |    10.83 % N=414097 C=371515 S=30229 D=12353 I=2269    |    15.64 % N=220358 C=193717 S=18669 D=7972 I=7812     |

## Whisper-largev3 (conv1d2, full-parameter tuning) Result (text\_fixed, see https://github.com/wenet-e2e/WenetSpeech/discussions/54)

* Feature info: using log_mel_spectrogram feature, no cmvn
* Training info: bf16, deepspeed stage1, activation checkpointing, batch dynamic12000, acc_grad 8, 8 * 3090 gpu, 48k steps (about 6 days), conf/finetune_whisper_largev3.yaml
* Decoding info: ctc_weight 0.0, average_num 5
* PR link: https://github.com/wenet-e2e/wenet/pull/2371

|   decoding_method   |  Dev | Test\_Net | Test\_Meeting |
|:-------------------:|:----:|:---------:|:-------------:|
|  ctc_greedy_search  | 7.09 % N=328207 C=308643 S=16976 D=2588 I=3709  | 10.98 % N=414092 C=373301 S=33375 D=7416 I=4697 | 12.84 % N=220358 C=194928 S=18398 D=7032 I=2862 |
|      attention      | 4.66 % N=328207 C=315591 S=10352 D=2264 I=2692  | 6.54 % N=414092 C=389523 S=19101 D=5468 I=2513 | 8.84 % N=220358 C=202722 S=11296 D=6340 I=1839  |
| attention_rescoring | 5.99 % N=328207 C=311106 S=14807 D=2294 I=2547  | 9.27 % N=414092 C=378406 S=28993 D=6693 I=2715 | 11.47 % N=220358 C=197013 S=16716 D=6629 I=1923 |

# Frequently Asked Questions

- Q: Why are there so many insertion errors in the decoding results of CTC and attention_rescoring?
- A: Because Chinese characters are composed of multiple bytes, in Whisper's tokenizer, one Chinese character might be represented by multiple tokens (for example, 3 tokens). During the CTC decoding process, it's possible that two or four tokens are decoded. This not only causes garbled text (see https://github.com/wenet-e2e/wenet/issues/2308 ) but also leads to insertion errors.

---

### Overview (`examples/wsj/s0/README.md`)

# Performance Record

## Conformer Result

* Feature info: dither + specaug + speed perturb
* Training info: lr 0.002, warmup_steps 20000 batch size 16, 1 gpu, acc_grad 4, 120 epochs
* Decoding info: average_num 20

|      decoding mode     |   dev93 (cer) |  dev93 (wer)  |
|:----------------------:|:-------------:|:-------------:|
|   ctc_greedy_search    |     5.25%     |    13.16%     |
| ctc_prefix_beam_search |     5.17%     |    13.10%     |
|  attention_rescoring   |     5.11%     |    12.17%     |

---

### Overview (`runtime/core/api/README.md`)

# WeNet API

We refer [vosk](https://github.com/alphacep/vosk-api/blob/master/src/vosk_api.h)
for the interface design.


We are going to implement the following interfaces:

- [x] non-streaming recognition
- [] streaming recognition
- [] nbest
- [] contextual biasing word
- [] alignment
- [] language support(post processor)
- [] label check

---

### Overview (`runtime/core/kaldi/README.md`)

We use Kaldi decoder to implement TLG based language model integration,
so we copied related files to this directory.
The main changes are:

1. To minimize the change, we use the same directories tree as Kaldi.

2. We replace Kaldi log system with glog in the following way.

``` c++
#define KALDI_WARN \
  google::LogMessage(__FILE__, __LINE__, google::GLOG_WARNING).stream()
#define KALDI_ERR \
  google::LogMessage(__FILE__, __LINE__, google::GLOG_ERROR).stream()
#define KALDI_INFO \
  google::LogMessage(__FILE__, __LINE__, google::GLOG_INFO).stream()
#define KALDI_VLOG(v) VLOG(v)

#define KALDI_ASSERT(condition) CHECK(condition)
```

3. We lint all the files to satisfy the lint in WeNet.

---

### Overview (`runtime/gpu/cuda_decoders/README.md`)

## Using CUDA based Decoders for Triton ASR Server
### Introduction
The triton model repository `model_repo_cuda_decoder` here, integrates the [CUDA WFST decoder](https://github.com/nvidia-riva/riva-asrlib-decoder) originally described in https://arxiv.org/abs/1910.10032. We take small conformer fp16 onnx inference for offline ASR as an example.

### Quick Start
```sh
# using docker image runtime/gpu/Dockerfile/Dockerfile.server
docker pull soar97/triton-wenet:22.12
docker run -it --rm --name "wenet_tlg_test" --gpus all --shm-size 1g --net host soar97/triton-wenet:22.12
# inside the docker container
git clone https://github.com/wenet-e2e/wenet.git
cd wenet/runtime/gpu/cuda_wfst_decoder
# Use pip3 install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple if you encounter network issue
pip3 install -r requirements.txt

# For offline TLG ASR
bash run.sh
# For streaming TLG ASR
bash run_streaming.sh
```

### Performance of Small Offline ASR Model using Different Decoders

Benchmark(small offline conformer onnx fp16 model trained on Aishell1) based on Aishell1 test set with V100, the total audio duration is 36108.919 seconds.

(Note: 80 concurrent tasks, service has been fully warm up.)
|Decoding Method | decoding time(s) | WER (%) |
|----------|--------------------|-------------|
| CTC Greedy Search             | 23s | 4.97  |
| CUDA TLG 1-best (3-gram LM)   | 31s | 4.58  |

---

### Overview (`runtime/gpu/tensorrt/README.md`)

### Using Tensorrt for Triton ASR Server

```sh
# using docker image runtime/gpu/Dockerfile/Dockerfile.server
docker pull soar97/triton-wenet:22.12
docker run -it --rm --name "wenet_trt_test" --gpus all --shm-size 1g --net host soar97/triton-wenet:22.12
# inside the docker container
git clone https://github.com/wenet-e2e/wenet.git
cd wenet/runtime/gpu/tensorrt
pip3 install nvidia-pyindex
# Use pip3 install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple if you encounter network issue
pip3 install -r requirements.txt

bash run_streaming_small_model.sh
```

#### Performance of Small u2pp Model for Streaming ASR

Benchmark(small u2pp onnx) based on Aishell1 test set with server-A10 (16vCPU 60GB Memory)/client(4vCPU 16GB Memory), the total audio duration is 36108.919 seconds.

(Note: using non-simulate-streaming mode)
|concurrent-tasks | processing time(s) |
|----------|--------------------|
| 20 (onnx fp16)                | 123.796 |
| 40 (onnx fp16)                | 84.557  |
| 60 (onnx fp16)                | 73.232  |
| 80 (onnx fp16)                | 66.862  |
| 20 (trt fp16+layernorm plugin)| 90.582  |
| 40 (trt fp16+layernorm plugin)| 75.411  |
| 60 (trt fp16+layernorm plugin)| 69.602  |
| 80 (trt fp16+layernorm plugin)| 65.603  |

---

### Overview (`runtime/gpu/tensorrt_fastertransformer/README.md`)

## Recipe for use of FasterTransformer accelerated Conformer model with Triton

```sh
# using docker image runtime/gpu/Dockerfile/Dockerfile.server
docker pull soar97/triton-wenet:22.12
docker run -it --rm --name "wenet_trt_test" --gpus all --net host --shm-size=1g soar97/triton-wenet:22.12
# inside the docker container
git clone https://github.com/wenet-e2e/wenet.git
cd wenet/runtime/gpu/tensorrt_fastertransformer
git submodule update --init
pip3 install nvidia-pyindex
# Use pip3 install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple if you encounter network issue
pip3 install -r requirements.txt
# Please modify the model parameters in run.sh. This recipe is for small conformer model.
bash run.sh
```
With help of FasterTransformer wenet tensorrt plugins, the overall throughput could get ~50% improvement comparing with onnx fp16 inference.

<p float="left">
  <img src="./encoder_plugin.JPG" width="45%" />
  <img src="./decoder_plugin.JPG" width="50%" />
</p>

#### Performance of Small Model for AIShell2

The following benchmakr shows the performance on T4 of a small Conformer model for AIShell2 case as defined in [WeNet AIShell2 Example](https://github.com/wenet-e2e/wenet/tree/main/examples/aishell2/s0).

<table>
 <col span='11'>
 <tr id='r0'>
<td>Backend</td>
<td>Precision</td>
<td>Input Length</td>
<td>concurrency</td>
<td>RTF</td>
<td>Throughput(infer/s)</td>
<td>Latency_p50(ms)</td>
<td>Latency_p90(ms)</td>
<td>Latency_p95(ms)</td>
<td>Latency_p99(ms)</td>
<td>Avg latency(ms)</td>
 </tr>
 <tr id='r1'>
<td rowspan='11' class='x21'>ONNX</td>
<td rowspan='11' class='x21'>FP16</td>
<td rowspan='6' class='x21'>5s</td>
<td style='text-align:right;'>10</td>
<td style='text-align:right;'>0.00082224</td>
<td style='text-align:right;'>243.238</td>
<td style='text-align:right;'>40.254</td>
<td style='text-align:right;'>49.884</td>
<td style='text-align:right;'>52.553</td>
<td style='text-align:right;'>57.191</td>
<td style='text-align:right;'>41.067</td>
 </tr>
 <tr id='r2'>
<td style='text-align:right;'>50</td>
<td style='text-align:right;'>0.000775588</td>
<td style='text-align:right;'>257.869</td>
<td style='text-align:right;'>200.534</td>
<td style='text-align:right;'>249.891</td>
<td style='text-align:right;'>263.713</td>
<td style='text-align:right;'>285.763</td>
<td style='text-align:right;'>193.721</td>
 </tr>
 <tr id='r3'>
<td style='text-align:right;'>100</td>
<td style='text-align:right;'>0.000777538</td>
<td style='text-align:right;'>257.222</td>
<td style='text-align:right;'>391.78</td>
<td style='text-align:right;'>479.738</td>
<td style='text-align:right;'>505.176</td>
<td style='text-align:right;'>551.676</td>
<td style='text-align:right;'>387.872</td>
 </tr>
 <tr id='r4'>
<td style='text-align:right;'>200</td>
<td style='text-align:right;'>0.000770553</td>
<td style='text-align:right;'>259.554</td>
<td style='text-align:right;'>757.215</td>
<td style='text-align:right;'>908.135</td>
<td style='text-align:right;'>962.108</td>
<td style='text-align:right;'>1122.423</td>
<td style='text-align:right;'>766.502</td>
 </tr>
 <tr id='r5'>
<td style='text-align:right;'>400</td>
<td style='text-align:right;'>0.000773168</td>
<td style='text-align:right;'>258.676</td>
<td style='text-align:right;'>1549.111</td>
<td style='text-align:right;'>1747.315</td>
<td style='text-align:right;'>1822.979</td>
<td style='text-align:right;'>1965.012</td>
<td style='text-align:right;'>1529.002</td>
 </tr>
 <tr id='r6'>
<td style='text-align:right;'>600</td>
<td style='text-align:right;'>0.000768832</td>
<td style='text-align:right;'>260.135</td>
<td style='text-align:right;'>2268.471</td>
<td style='text-align:right;'>2513.159</td>
<td style='text-align:right;'>2584.864</td>
<td style='text-align:right;'>2867.763</td>
<td style='text-align:right;'>2260.872</td>
 </tr>
 <tr id='r7'>
<td rowspan='5' class='x21'>8s</td>
<td style='text-align:right;'>10</td>
<td style='text-align:right;'>0.000749239</td>
<td style='text-align:right;'>166.836</td>
<td style='text-align:right;'>58.89</td>
<td style='text-align:right;'>76.627</td>
<td style='text-align:right;'>80.612</td>
<td style='text-align:right;'>87.494</td>
<td style='text-align:right;'>59.893</td>
 </tr>
 <tr id='r8'>
<td style='text-align:right;'>50</td>
<td style='text-align:right;'>0.000739693</td>
<td style='text-align:right;'>168.989</td>
<td style='text-align:right;'>307.598</td>
<td style='text-align:right;'>380.47</td>
<td style='text-align:right;'>398.13</td>
<td style='text-align:right;'>431.647</td>
<td style='text-align:right;'>295.728</td>
 </tr>
 <tr id='r9'>
<td style='text-align:right;'>100</td>
<td style='text-align:right;'>0.000740175</td>
<td style='text-align:right;'>168.879</td>
<td style='text-align:right;'>584.401</td>
<td style='text-align:right;'>722.173</td>
<td style='text-align:right;'>759.771</td>
<td style='text-align:right;'>820.324</td>
<td style='text-align:right;'>588.995</td>
 </tr>
 <tr id='r10'>
<td style='text-align:right;'>200</td>
<td style='text-align:right;'>0.000733417</td>
<td style='text-align:right;'>170.435</td>
<td style='text-align:right;'>1136.949</td>
<td style='text-align:right;'>1343.338</td>
<td style='text-align:right;'>1396.169</td>
<td style='text-align:right;'>1526.366</td>
<td style='text-align:right;'>1162.847</td>
 </tr>
 <tr id='r11'>
<td style='text-align:right;'>400</td>
<td style='text-align:right;'>0.000758132</td>
<td style='text-align:right;'>164.879</td>
<td style='text-align:right;'>2424.999</td>
<td style='text-align:right;'>2837.982</td>
<td style='text-align:right;'>2948.57</td>
<td style='text-align:right;'>3165.213</td>
<td style='text-align:right;'>2407.398</td>
 </tr>
 <tr id='r12'>
<td rowspan='12' class='x21'>TensorRT</td>
<td rowspan='12' class='x21'>FP16</td>
<td rowspan='6' class='x21'>5s</td>
<td style='text-align:right;'>10</td>
<td style='text-align:right;'>0.000806917</td>
<td style='text-align:right;'>247.857</td>
<td style='text-align:right;'>40.607</td>
<td style='text-align:right;'>55.071</td>
<td style='text-align:right;'>59.839</td>
<td style='text-align:right;'>70.496</td>
<td style='text-align:right;'>40.292</td>
 </tr>
 <tr id='r13'>
<td style='text-align:right;'>50</td>
<td style='text-align:right;'>0.000616084</td>
<td style='text-align:right;'>324.631</td>
<td style='text-align:right;'>155.034</td>
<td style='text-align:right;'>207.362</td>
<td style='text-align:right;'>224.129</td>
<td style='text-align:right;'>247.152</td>
<td style='text-align:right;'>153.683</td>
 </tr>
 <tr id='r14'>
<td style='text-align:right;'>100</td>
<td style='text-align:right;'>0.000539622</td>
<td style='text-align:right;'>370.63</td>
<td style='text-align:right;'>274.031</td>
<td style='text-align:right;'>319.398</td>
<td style='text-align:right;'>333.757</td>
<td style='text-align:right;'>353.26</td>
<td style='text-align:right;'>269.732</td>
 </tr>
 <tr id='r15'>
<td style='text-align:right;'>200</td>
<td style='text-align:right;'>0.000531885</td>
<td style='text-align:right;'>376.021</td>
<td style='text-align:right;'>515.018</td>
<td style='text-align:right;'>638.437</td>
<td style='text-align:right;'>657.918</td>
<td style='text-align:right;'>680.002</td>
<td style='text-align:right;'>592.341</td>
 </tr>
 <tr id='r16'>
<td style='text-align:right;'>400</td>
<td style='text-align:right;'>0.000536799</td>
<td style='text-align:right;'>372.579</td>
<td style='text-align:right;'>1040.958</td>
<td style='text-align:right;'>1173.062</td>
<td style='text-align:right;'>1187.594</td>
<td style='text-align:right;'>1237.658</td>
<td style='text-align:right;'>1064.299</td>
 </tr>
 <tr id='r17'>
<td style='text-align:right;'>800</td>
<td style='text-align:right;'>0.000561322</td>
<td style='text-align:right;'>356.302</td>
<td style='text-align:right;'>2210.466</td>
<td style='text-align:right;'>2319.285</td>
<td style='text-align:right;'>2343.111</td>
<td style='text-align:right;'>2893.016</td>
<td style='text-align:right;'>2207.461</td>
 </tr>
 <tr id='r18'>
<td rowspan='6' class='x21'>8s</td>
<td style='text-align:right;'>10</td>
<td style='text-align:right;'>0.00066929</td>
<td style='text-align:right;'>186.765</td>
<td style='text-align:right;'>53.462</td>
<td style='text-align:right;'>74.164</td>
<td style='text-align:right;'>81.128</td>
<td style='text-align:right;'>96.467</td>
<td style='text-align:right;'>53.454</td>
 </tr>
 <tr id='r19'>
<td style='text-align:right;'>50</td>
<td style='text-align:right;'>0.000550967</td>
<td style='text-align:right;'>226.874</td>
<td style='text-align:right;'>223.332</td>
<td style='text-align:right;'>282.558</td>
<td style='text-align:right;'>310.565</td>
<td style='text-align:right;'>355.387</td>
<td style='text-align:right;'>219.928</td>
 </tr>
 <tr id='r20'>
<td style='text-align:right;'>100</td>
<td style='text-align:right;'>0.000525813</td>
<td style='text-align:right;'>237.727</td>
<td style='text-align:right;'>423.628</td>
<td style='text-align:right;'>497.259</td>
<td style='text-align:right;'>517.581</td>
<td style='text-align:right;'>563.85</td>
<td style='text-align:right;'>418.962</td>
 </tr>
 <tr id='r21'>
<td style='text-align:right;'>200</td>
<td style='text-align:right;'>0.000513162</td>
<td style='text-align:right;'>243.588</td>
<td style='text-align:right;'>794.689</td>
<td style='text-align:right;'>911.775</td>
<td style='text-align:right;'>973.896</td>
<td style='text-align:right;'>1037.958</td>
<td style='text-align:right;'>815.351</td>
 </tr>
 <tr id='r22'>
<td style='text-align:right;'>400</td>
<td style='text-align:right;'>0.000531747</td>
<td style='text-align:right;'>235.074</td>
<td style='text-align:right;'>1683.981</td>
<td style='text-align:right;'>1823.173</td>
<td style='text-align:right;'>1852.701</td>
<td style='text-align:right;'>1910.553</td>
<td style='text-align:right;'>1685.217</td>
 </tr>
 <tr id='r23'>
<td style='text-align:right;'>600</td>
<td style='text-align:right;'>0.000585379</td>
<td style='text-align:right;'>213.537</td>
<td style='text-align:right;'>2782.118</td>
<td style='text-align:right;'>2939.944</td>
<td style='text-align:right;'>2998.534</td>
<td style='text-align:right;'>3192.138</td>
<td style='text-align:right;'>2751.356</td>
 </tr>
</table>

Note: Currently, encoder only supports offline conformer model. Decoder only supports transformer decoder.

For directly using FasterTransformer built wenet, you may refer to https://github.com/NVIDIA/FasterTransformer/tree/main/examples/cpp/wenet

---

### UIO (`docs/UIO.md`)

# Big Data Training: UIO

In order to support the model training of industrial tens of millions of hours of speech dataset, the data processing
method UIO (Unified IO) has been updated in WeNet. The document will introduce UIO from the following sections:
Necessity of upgrading IO mothod, System design of UIO, Validation experiments, Usage of UIO, Q&A.

## Necessity of upgrading IO mothod
The old IO method in WeNet is based on Pytorch's native Dataset. During training, it need to load all training audio
paths and correspondingly labels into the memory at one time, then randomly read data. In the case of industrial-grade
ultra-large-scale data (egs: more than 50,000 hours or 50 million or more audio), this method will cause the training
to fail to run for two reasons:
- Out of memory(OOM): The physical memory of the general machine is difficult to load the training data at one time.
- Slow down reading performance:  In the case that the large-scale data memory cannot be used as a file cache, the training
data reading speed is greatly reduced.

## System design of UIO
### Overall design
Inspired by the following industrial methods(egs: [webdataset](https://github.com/webdataset/webdataset), [TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord)),
WeNet redesigned the IO method. Its core idea is to make the audio and labels of multiple small data(such as 1000 pieces),
into compressed packets (tar) and read them based on the IterableDataset of Pytorch. The advantages of this method is：
- Only the index information of the compressed package needs to be maintained in memory, which greatly saves memory and
solves the problem of OOM.
- The on-the-fly decompression is performed in the memory, and the data in the same compressed package is read in
sequence, which solves the problem of slow random reading performance. Different compressed packets can be read randomly
to ensure the global randomness of data.

The new IO method takes into account both small datasets and large datasets， and provides two data reading methods.
We call it UIO. The overall design of UIO is shown in the figure below:



Some necessary explanations about the above figure:
- Small IO(raw) supports small datasets, which we call ``raw`` mode. This mode only supports local file reading.
The required documents must be sorted into Kaldi style file: wav.scp and text.(It's the same as before)
- Big IO(shared) supports large datasets, which we call ``shard`` mode. This mode can support both local file
reading and network cloud storage file reading. The required files must be sorted into compressed packages. Audio (wav)
and label (txt) are stored in a single compressed package in sequence.

### Chain IO
Inspired by TFRecord chain IO, UIO also adopts chain implementation. In practical use, chain IO is more flexible,
easier to expand and easier to debug. TFRecord IO example as follows,
```python
def read_dataset(filename, batch_size):
    dataset = tf.data.TFRecordDataset(filename)
    dataset = dataset.map(_parse_image_function, num_parallel_calls=tf.data.experimental.AUTOTUNE)
    dataset = dataset.shuffle(500)
    dataset = dataset.batch(batch_size, drop_remainder=True)
    dataset = dataset.repeat()
    dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
    return dataset
```
Refer to TFRecord IO, the UIO dataflow in WeNet is designed as the figure below:



It includes the following modules:
- tokenize module: convert the label into specify modeling unit(egs: char or BPE).

- filter module: filter out unqualified training data.

- resample module: optional resampling of training data.

- compute_fbank module: fbank feature extraction.

- spec_augmentation module: feature enhancement.

- shuffle module: disrupt local data.

- sort module: sort local data.

- batch module: organize multiple pieces of data into batch.

- padding module: padding data in the same batch.

what's more, There are several parameters to note. first, ``shuffle buffer`` and ``sort buffer`` in ``buffer size``:
* ``Shuffle buffer``: shuffle data. It is recommended that the size of this buffer be larger than
the number of data contained in a single shard. Each time it is equivalent to shuffling data between two shards,
which increases the randomness of the data.(egs: if each shard contains 1000, you can set shuffle buffer as 1500)
* ``Sort buffer``: sort the data according to the number of frames. This operation is very important and can greatly
improve the training speed.

then, ``Prefetch``:
``Prefetch`` is used in the Pytorch ``Dataloader`` to pre-read data. The granularity of prefetch is the batch of final training.
The default parameter is 2, that is, the data of two batches will be pre-read by default. In the design of the UIO,
due to the existence of the pre buffer, the pre-read data may already be in the buffer, so there is no real pre read.
Only when the data in the buffer is insufficient during the next training can the buffer be filled on the fly.
At this time, the training is blocked on the read data. In a word, when the prefetch is very small, the training will
block reading data in part of the time, because the previous data is still in cache. Therefore, you can set a large
prefetch to avoid this problem.


## Validation experiments
At present, we have verified the accuracy of UIO on aishell (200 hours) and wenetspeech (10000 hours) data respectively.
### Aishell(``raw`` vs ``shard``)

|IO|CER|
|:---|:---|
|Old|4.61|
|UIO(``Raw``)|4.63|
|UIO(``Shard``)|4.67|

### WenetSpeech(``shard``)



WeNet and ESPnet use similar model structure and parameter configuration, and they achieve similar recognition rate,
which shows the correctness of UIO in WeNet. And during the training, we observed that the overall utilization rate of
GPU of UIO is more than 80% - 90%, indicating that the overall IO reading efficiency is very high.

## Usage of UIO
For detailed usage of UIO, please refer to the aishell dataset example:
https://github.com/wenet-e2e/wenet/blob/main/examples/aishell/s0/run.sh
At present, all datasets in WeNet have used UIO as the default data preparation.

There are three parameters related to UIO in the training script train.py：
- ``train_data``(``cv_data``/``test_data``): data.list
- ``data_type``: raw/shard
- ``symbol_table``: specify modeling unit

For example:
```shell
python wenet/bin/train.py --gpu $gpu_id \
  --config $train_config \
  --data_type $data_type \
  --symbol_table $dict \
  --train_data $feat_dir/$train_set/data.list \
  --cv_data $feat_dir/dev/data.list \
  ...
```
If data_type is ``raw``, the format of data.list is as follows:
```
{"key": "BAC009S0002W0122", "wav": "/export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0122.wav", "txt": "而对楼市成交抑制作用最大的限购"}
{"key": "BAC009S0002W0123", "wav": "/export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0123.wav", "txt": "也成为地方政府的眼中钉"}
{"key": "BAC009S0002W0124", "wav": "/export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0124.wav", "txt": "自六月底呼和浩特市率先宣布取消限购后"}
```
Each line is a json serialized string, which contains three fields: ``key``, ``wav`` and ``txt``.

If data_type is ``shard``, the format of data.list is as follows:
```
# [option 1: local]
/export/maryland/binbinzhang/code/wenet/examples/aishell/s3/raw_wav/train/shards/shards_000000000.tar.gz
/export/maryland/binbinzhang/code/wenet/examples/aishell/s3/raw_wav/train/shards/shards_000000001.tar.gz
/export/maryland/binbinzhang/code/wenet/examples/aishell/s3/raw_wav/train/shards/shards_000000002.tar.gz

# [option 2: network(egs: OSS)]
https://examplebucket.oss-cn-hangzhou.aliyuncs.com/exampledir/1.tar.gz
https://examplebucket.oss-cn-hangzhou.aliyuncs.com/exampledir/2.tar.gz
```

## Q&A
Q1: How to operate distributed partition of training data?

A: According to rank and num_workers can segment the data. for example:
```python
class DistributedSampler:
    def __init__(self, shuffle=True, partition=True):
        self.epoch = -1
        self.update()
        self.shuffle = shuffle
        self.partition = partition

    def set_epoch(self, epoch):
        self.epoch = epoch

    def sample(self, data):
        data = list(range(len(data)))
        if self.partition:
            if self.shuffle:
                random.Random(self.epoch).shuffle(data)
            data = data[self.rank::self.world_size]
        data = data[self.worker_id::self.num_workers]
        return data
```

Q2: How to deal with unbalanced data?

A: Use model.join() to handle the imbalance of data allocated on each rank. Please refer [this](https://pytorch.org/tutorials/advanced/generic_join.html#how-does-join-work).

---

### Context (`docs/context.md`)

## Context Biasing

In the practical application of ASR, the recognition effect of commonly used words is better, but for some unique words, the recognition accuracy may be low. Contextual biasing is the problem of injecting prior knowledge into an ASR system during inference, for example a user’s favorite songs, contacts, apps or location. Conventional ASR systems perform contextual biasing by building an n-gram finite state transducer (FST) from a list of biasing phrases, which is composed on-the-fly with the decoder graph during decoding. This helps to bias the recognition result towards the n-grams contained in the contextual FST, and thus improves accuracy in certain scenarios.

In WeNet, we compute the biasing scores $P_C(\mathbf y)$, which are interpolated with the base model $P(\mathbf y|\mathbf x)$ using shallow-fusion during beam search, including CTC prefix beam search and CTC WFST beam search.

$$
\mathbf y^*=\mathrm{arg\,max\,log}P(\mathbf y|\mathbf x)+\lambda\,\mathrm{log}\,P_C(\mathbf y)
$$

where, $\lambda$ is a tunable hyperparameter controlling how much the contextual LM influences the overall model score during beam search.

### Context Graph

If we want to improve the score of the word "cat", and the biasing score $\lambda\,\mathrm{log}\,P_C(\mathbf y)$ of each character is 0.25. The context graph can be constructed as follow:



In the decoding process, when the corresponding prefix is matched, the corresponding score reward will be obtained. In order to avoid artificially boosting prefixes which match early on but do not match the entire phrase, we add a special failure arc which removes the boosted score.

WeNet records only one state for each prefix, to easily determine the boundary of the matched hot word. That is, only one hot word can be matched at the same time, and only after the hot word matching succeeds or fails can other hot words start matching.

``` c++
int ContextGraph::GetNextState(int cur_state, int word_id, float* score,
                               bool* is_start_boundary, bool* is_end_boundary) {
  int next_state = 0;
  // Traverse the arcs of current state.
  for (fst::ArcIterator<fst::StdFst> aiter(*graph_, cur_state); !aiter.Done();
       aiter.Next()) {
    const fst::StdArc& arc = aiter.Value();
    if (arc.ilabel == 0) {
      // Record the score of the backoff arc. It might will be covered.
      *score = arc.weight.Value();
    } else if (arc.ilabel == word_id) {
      // If they match, record the next state and the score.
      next_state = arc.nextstate;
      *score = arc.weight.Value();
      // Check whether is the boundary of the hot word.
      if (cur_state == 0) {
        *is_start_boundary = true;
      }
      if (graph_->Final(arc.nextstate) == fst::StdArc::Weight::One()) {
        *is_end_boundary = true;
      }
      break;
    }
  }
  return next_state;
}
```

### CTC Prefix Beam Search

In the process of CTC prefix beam search, each prefix needs to record the hot word matching information. After appending the current output character, if the prefix changes, call the above function `GetNextState` to update the state and score of the hot word. If it is the start or end of a hot word, it is also necessary to record the position, which are used to insert the start tag and end tag in the result, such as: "The \<context>cat\</context> is in the bag".

### CTC WFST Beam Search

WeNet adopts the Lattice Faster Online Decoder from Kaldi for WFST beam search. We have to modify the `lattice-faster-decoder.cc` to support context biasing.

WFST beam search decodes in the TLG graph according to the CTC outputs. If we bias the input label of the TLG, we need to compose the context graph with the Token graph. Finally, we decide to bias TLG's output towards the contextual fst. We need to modify the `ProcessEmitting` and `ProcessNonemitting` functions as follow:

```c++
Elem *e_next =
    FindOrAddToken(arc.nextstate, frame + 1, tot_cost, tok, NULL);
// NULL: no change indicator needed

// ========== Context code BEGIN ===========
bool is_start_boundary = false;
bool is_end_boundary = false;
float context_score = 0;
if (context_graph_) {
  if (arc.olabel == 0) {
    e_next->val->context_state = tok->context_state;
  } else {
    e_next->val->context_state = context_graph_->GetNextState(
      tok->context_state, arc.olabel, &context_score,
      &is_start_boundary, &is_end_boundary);
    graph_cost -= context_score;
  }
}
// ========== Context code END ==========

// Add ForwardLink from tok to next_tok (put on head of list
// tok->links)
tok->links = new ForwardLinkT(e_next->val, arc.ilabel, arc.olabel,
                              graph_cost, ac_cost, is_start_boundary,
                              is_end_boundary, tok->links);
tok->links->context_score = context_score;
```

### Pruning

The backoff arc will return the accumulated scores to a single ForwardLink. It leads to the cost of that ForwardLink is too large. We have to remove the cost returned by backoff arc before pruning.

```c++
void LatticeFasterDecoderTpl<FST, Token>::PruneForwardLinks(
    int32 frame_plus_one, bool *extra_costs_changed, bool *links_pruned,
    BaseFloat delta) {
  ...
  BaseFloat link_extra_cost =
            next_tok->extra_cost +
            ((tok->tot_cost + link->acoustic_cost + link->graph_cost) -
             next_tok->tot_cost);  // difference in brackets is >= 0
  // ========== Context code BEGIN ===========
  // graph_cost contains the score of hot word
  // link->context_score < 0 means the hot word of the link is returned from backoff arc
  if (link->context_score < 0) {
    link_extra_cost += link->context_score;
  }
  // ========== Context code END ==========
  // link_exta_cost is the difference in score between the best paths
  // through link source state and through link destination state
```

### Usage

1. Specify the `--context_path` to a text file.
   - Each line of the file contains a context.
   - Each context can be split into words with the symbol_table of the ASR model (It means there is no oov in the context).
2. Specify the `--context_score`, the reward of each word in the context.

```bash
cd /home/wenet/runtime/libtorch
export GLOG_logtostderr=1
export GLOG_v=2
wav_path=docker_resource/test.wav
context_path=docker_resource/context.txt
model_dir=docker_resource/model
./build/decoder_main \
    --chunk_size -1 \
    --wav_path $wav_path \
    --model_path $model_dir/final.zip \
    --context_path $context_path \
    --context_score 3 \
    --unit_path $model_dir/units.txt 2>&1 | tee log.txt
```

---

### Index (`docs/index.rst`)

.. wenet documentation master file, created by
   sphinx-quickstart on Thu Dec  3 11:43:53 2020.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to wenet's documentation!
=================================


wenet is an tansformer-based end-to-end ASR toolkit.

.. toctree::
   :maxdepth: 2
   :caption: Contents:

   ./python_package.md
   ./train.rst
   ./production.rst
   ./reference.rst

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

---

### Jit In Wenet (`docs/jit_in_wenet.md`)

# JIT in WeNet

We want that our PyTorch model can be directly exported by torch.jit.script method,
which is essential for deploying the model to production.

See the following resource for how to deploy PyTorch models in production in details.
- [INTRODUCTION TO TORCHSCRIPT](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html)
- [TORCHSCRIPT LANGUAGE REFERENCE](https://pytorch.org/docs/stable/jit_language_reference.html#language-reference)
- [LOADING A TORCHSCRIPT MODEL IN C++](https://pytorch.org/tutorials/advanced/cpp_export.html)
- [TorchScript and PyTorch JIT | Deep Dive](https://www.youtube.com/watch?v=2awmrMRf0dA&t=574s)
- [Research to Production: PyTorch JIT/TorchScript Updates](https://www.youtube.com/watch?v=St3gdHJzic0)

To ensure that, we will try to export the model before training stage.
If it fails, we should modify the training code to satisfy the export requirements.

``` python
# See in wenet/bin/train.py
script_model = torch.jit.script(model)
script_model.save(os.path.join(args.model_dir, 'init.zip'))
```

Two principles should be taken into consideration when we contribute our python code
to WeNet, especially for the subclass of torch.nn.Module, and for the forward function.

1. Know what is allowed and what is disallowed.
    - [Torch and Tensor Unsupported Attributes](https://pytorch.org/docs/master/jit_unsupported.html#jit-unsupported)
    - [Python Language Reference Coverage](https://pytorch.org/docs/master/jit_python_reference.html#python-language-reference)

2. Try to use explicit typing as much as possible. You can try to do type checking
forced by typeguard, see https://typeguard.readthedocs.io/en/latest/userguide.html for details.

---

### Lm (`docs/lm.md`)

# LM for WeNet

WeNet uses n-gram based statistical language model and the WFST framework to support the custom language model.
And LM is only supported in runtime of WeNet.

## Motivation

Why n-gram based LM? This may be the first question many people will ask.
Now that LM based on RNN and Transformer is in full swing, why does WeNet go backward?
The reason is simple, it is for productivity.
The n-gram-based language model has mature and complete training tools,
any amount of corpus can be trained, the training is very fast, the hotfix is easy,
and it has a wide range of mature applications in actual products.

Why WFST? It may be the second question many people will ask.
Since both industry and research have been working so hard to abandon traditional speech recognition,
especially the complex decoding technology. Why does WeNet back?
The reason is also very simple, it is for productivity.
WFST is a standard and powerful tool in traditional speech recognition.
And based on this solution, we have mature and complete bug fix solutions and product solutions,
such as that we can use the replace function in WFST for class-based personalization such as contact recognition.

Therefore, just like WeNet's design goal "Production first and Production Ready",
LM in WeNet also puts productivity as the first priority.
So it draws on many very productive tools and solutions accumulated in traditional speech recognition.
The difference to traditional speech recognition are:

1. The training in WeNet is pure end to end.
2. As described below, LM is optional in decoding, you can choose whether to use LM according to your needs and application scenarios.


## System Design

The whole system is shown in the bellowing picture. There are two ways to generate N-best.



1. Without LM, we use CTC prefix beam search to generate N-best.
2. With LM, we use CTC WFST search to generate N-best and CTC WFST search is the traditional WFST based decoder.

There are two main parts of the CTC WFST based search.

The first is building the decoding graph, which is to compose the model unit T, the lexicon L and the language model G into one unified graph TLG. And in which:
1. T is the model unit in E2E training. Typically it's char in Chinese, char or BPE in English.
2. L is the lexicon, the lexicon is very simple. What we need to do is just split a word into its modeling unit sequence.
For example, the word "我们" is split into two chars "我 们", and the word "APPLE" is split into five letters "A P P L E".
We can see there is no phonemes and there is no need to design pronunciation on purpose.
3. G is the language model, namely compiling the n-gram to standard WFST representation.

The second is the decoder, which is the same as the traditional decoder, which uses the standard Viterbi beam search algorithm in decoding.

## Implementation

WeNet draws on the decoder and related tools in Kaldi to support LM and WFST based decoding.
For ease of using and keeping independence, we directly migrated the code related to decoding in Kaldi to [this directory](https://github.com/wenet-e2e/wenet/tree/main/runtime/core/kaldi) in WeNet runtime.
And modify and organize according to the following principles:
1. To minimize changes, the migrated code remains the same directory structure as the original.
2. We use GLOG to replace the log system in Kaldi.
3. We modify the code format to meet the lint requirements of the code style in WeNet.

The core code is https://github.com/wenet-e2e/wenet/blob/main/runtime/core/decoder/ctc_wfst_beam_search.cc,
which wraps the LatticeFasterDecoder in Kaldi.
And we use blank frame skipping to speed up decoding.

In addition, WeNet also migrated related tools for building the decoding graph,
such as arpa2fst, fstdeterminizestar, fsttablecompose, fstminimizeencoded, and other tools.
So all the tools related to LM are built-in tools and can be used out of the box.


## Results

We get consistent gain (3%~10%) on different datasets,
including aishell, aishell2, and librispeech,
please go to the corresponding example dataset for the details.

## How to use?

Here is an example from aishell, which shows how to prepare the dictionary, how to train the LM,
how to build the graph, and how to decode with the runtime.

``` sh
# 7.1 Prepare dict
unit_file=$dict
mkdir -p data/local/dict
cp $unit_file data/local/dict/units.txt
tools/fst/prepare_dict.py $unit_file ${data}/resource_aishell/lexicon.txt \
    data/local/dict/lexicon.txt
# 7.2 Train lm
lm=data/local/lm
mkdir -p $lm
tools/filter_scp.pl data/train/text \
     $data/data_aishell/transcript/aishell_transcript_v0.8.txt > $lm/text
local/aishell_train_lms.sh
# 7.3 Build decoding TLG
tools/fst/compile_lexicon_token_fst.sh \
    data/local/dict data/local/tmp data/local/lang
tools/fst/make_tlg.sh data/local/lm data/local/lang data/lang_test || exit 1;
# 7.4 Decoding with runtime
./tools/decode.sh --nj 16 \
    --beam 15.0 --lattice_beam 7.5 --max_active 7000 \
    --blank_skip_thresh 0.98 --ctc_weight 0.5 --rescoring_weight 1.0 \
    --fst_path data/lang_test/TLG.fst \
    --dict_path data/lang_test/words.txt \
    data/test/wav.scp data/test/text $dir/final.zip \
    data/lang_test/units.txt $dir/lm_with_runtime
```

---

### Papers (`docs/papers.md`)

## Papers

* [WeNet: Production Oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://arxiv.org/pdf/2102.01547.pdf), accepted by InterSpeech 2021.
* [WeNet 2.0: More Productive End-to-End Speech Recognition Toolkit](https://arxiv.org/pdf/2203.15455.pdf), accepted by InterSpeech 2022.

---

### Pretrained Models (`docs/pretrained_models.md`)

# Pretrained Models in WeNet

## Model Types
We provide two types of pretrained model in WeNet to facilitate users with different requirements.

1. **Checkpoint Model**, with suffix **.pt**, the model trained and saved as checkpoint by WeNet python code, you can reproduce our published result with it, or you can use it as checkpoint to continue.

2. **Runtime Model**, with suffix **.zip**, you can directly use `runtime model` in our [x86](https://github.com/wenet-e2e/wenet/tree/main/runtime/libtorch) or [android](https://github.com/wenet-e2e/wenet/tree/main/runtime/android) runtime, the `runtime model` is export by Pytorch JIT on the `checkpoint model`. And the runtime models has been quantized to reduce the model size and network traffic.

## Model License

The pretrained model in WeNet follows the license of it's corresponding dataset.
For example, the pretrained model on LibriSpeech follows `CC BY 4.0`, since it is used as license of the LibriSpeech dataset, see http://openslr.org/12/.

## Model List

Here is a list of the pretrained models on different datasets. The model structure, model size, and download link are given.

| Datasets  | Languages     | Checkpoint Model  | Runtime Model     | Contributor |
|---    |---    |---    |---    |---    |
| [aishell](../examples/aishell/s0/README.md)   | CN    | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=aishell_u2pp_conformer_exp.tar.gz)  | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=aishell_u2pp_conformer_libtorch.tar.gz)     | <a href="https://www.chumenwenwen.com" target="_blank"><img src="https://raw.githubusercontent.com/wenet-e2e/wenet-contributors/main/companies/chumenwenwen.png" width="100px"></a> |
| [aishell2](../examples/aishell2/s0/README.md)     | CN    | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=aishell2_u2pp_conformer_exp.tar.gz)     | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=aishell2_u2pp_conformer_libtorch.tar.gz)    | <a href="https://www.chumenwenwen.com" target="_blank"><img src="https://raw.githubusercontent.com/wenet-e2e/wenet-contributors/main/companies/chumenwenwen.png" width="100px"></a> |
| [gigaspeech](../examples/gigaspeech/s0/README.md)     | EN    | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=gigaspeech_u2pp_conformer_exp.tar.gz)   | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=gigaspeech_u2pp_conformer_libtorch.tar.gz)  |  <a href="https://www.chumenwenwen.com" target="_blank"><img src="https://raw.githubusercontent.com/wenet-e2e/wenet-contributors/main/companies/chumenwenwen.png" width="100px"></a> |
| [librispeech](../examples/librispeech/s0/README.md)   | EN    | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=librispeech_u2pp_conformer_exp.tar.gz)  | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=librispeech_u2pp_conformer_libtorch.tar.gz)     |  <a href="https://www.chumenwenwen.com" target="_blank"><img src="https://raw.githubusercontent.com/wenet-e2e/wenet-contributors/main/companies/chumenwenwen.png" width="100px"></a> |
| [multi_cn](../examples/multi_cn/s0/README.md)     | CN    | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=multi_cn_unified_conformer_exp.tar.gz)  | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=multi_cn_unified_conformer_libtorch.tar.gz)     | <a href="https://www.jd.com" target="_blank"><img src="https://raw.githubusercontent.com/wenet-e2e/wenet-contributors/main/companies/jd.jpeg" width="100px"></a> |
| [wenetspeech](../examples/wenetspeech/s0/README.md)     | CN    | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=wenetspeech_u2pp_conformer_exp.tar.gz) | [Conformer](https://wenet.org.cn/downloads?models=wenet&version=wenetspeech_u2pp_conformer_libtorch.tar.gz)     | <a href="https://horizon.ai" target="_blank"><img src="https://raw.githubusercontent.com/wenet-e2e/wenet-contributors/main/companies/hobot.png" width="100px"></a> |
| paraformer    | CN&EN    | [Model](https://wenet.org.cn/downloads?models=wenet&version=paraformer.tar.gz) |  NA  | Alibaba |
| firered    | CN&EN    | [Model](https://wenet.org.cn/downloads?models=wenet&version=firered.tar.gz) |  NA  | FireRed |
| whisper-large-v3    |  multi-lingual  | [Model](https://wenet.org.cn/downloads?models=wenet&version=whisper-large-v3.tar.gz) |  NA  | OpenAI |
| whisper-large-v3-turbo    |  multi-lingual  | [Model](https://wenet.org.cn/downloads?models=wenet&version=whisper-large-v3-turbo.tar.gz) |  NA  | OpenAI |

---

### Production (`docs/production.rst`)

Production Runtime
==================

.. toctree::
   :maxdepth: 1
   :caption: Contents:

   ./lm.md
   ./context.md
   ./runtime.md
   ./jit_in_wenet.md

---

### Python Package (`docs/python_package.md`)

# Python Package


## Install

``` sh
pip install git+https://github.com/wenet-e2e/wenet.git
```

## Development Install

``` sh
git clone https://github.com/wenet-e2e/wenet.git
cd wenet
pip install -e .
```


## Command line Usage

``` sh
wenet --language chinese audio.wav
```

You can specify the following parameters.

* `-l` or `--language`: chinese/english are supported now.
* `-m` or `--model_dir`: your own model dir
* `-g` or `--gpu`: the device id of gpu, default value -1 represents for cpu.
* `-t` or `--show_tokens_info`: show the token level information such as timestamp, confidence, etc.
* `--align`: force align the input audio and transcript
* `--label`: the input label to align
* `--paraformer`: use the best Chinese model
* `--device`: specify the backend accelerator (cuda/npu/cpu)

## Python Programming Usage

``` python
import wenet

model = wenet.load_model('chinese')
# or model = wenet.load_model(model_dir='xxx')
result = model.transcribe('audio.wav')
print(result['text'])
```

## GPU/Hardware Notes

Measured on a Tesla T4 (16GB, Turing/sm_75), torch 2.4.0+cu121.

* **GPU is not the default.** `--device` (CLI) and `device` (`wenet.load_model`)
  both default to `'cpu'`. On a GPU host, `wenet -m <model> audio.wav` still
  runs on CPU unless you pass `--device cuda`. There is no warning when this
  happens. Measured on paraformer: CUDA is **2.84x** faster than CPU
  (0.205s vs 0.581s per utterance, cached model, T4).
* **No dtype flag on the pip-installed CLI.** The `wenet` console command
  (`wenet/cli/transcribe.py`) always runs fp32 and has no `--dtype`/`--fp16`
  flag. The training-recipe script `wenet/bin/recognize.py` does expose
  `--dtype {fp16,fp32,bf16}`, but that is a separate, more involved entry
  point (local recipe checkout, not the pip package).
* **bf16 can be slower than fp32 on Turing-class GPUs (e.g. T4).** T4 lacks
  bf16 tensor-core acceleration. On WeNet's own Conformer models (e.g.
  `wenetspeech`), casting to bf16 measured **~47% slower** than fp32
  (46.6ms vs 31.7ms per forward pass). fp16 does not crash but shows no
  speedup at batch size 1 with short audio (launch-overhead bound), though it
  does cut VRAM by about 46%.
* **`use_sdpa` is off by default in the shipped configs, and turning it on
  helps most on bf16.** WeNet's own Conformer implementation supports a
  `use_sdpa: true` toggle (`encoder_conf`/`decoder_conf`) that switches
  attention from a manual matmul+softmax to
  `torch.nn.functional.scaled_dot_product_attention`, using the same
  checkpoint with no retraining. The `train.yaml` shipped with pretrained
  models (e.g. `wenetspeech`) does not set this key, so it loads as `False`.
  Enabling it measured 7-29% faster on a T4, with identical output text:
  fp32 31.7ms -> 29.4ms (~7%), fp16 32.2ms -> 30.0ms (~7%), bf16
  46.6ms -> 33.1ms (~29%, cutting most of the bf16 slowdown above). This
  toggle only applies to WeNet's own Conformer models; Paraformer (SANM
  attention) does not have a `use_sdpa` code path.
* **int8 dynamic quantization is CPU-only.** `wenet/bin/export_jit.py` uses
  `torch.quantization.quantize_dynamic`, which works on CPU but raises
  `NotImplementedError: Could not run 'quantized::linear_dynamic' with
  arguments from the 'CUDA' backend` if applied to a CUDA model — this is a
  PyTorch backend limitation, not a WeNet bug. (`wenet/bin/export_onnx_cpu.py`
  quantizes separately via `onnxruntime.quantization.quantize_dynamic`, a
  different, ONNX-graph-level API that isn't affected by this PyTorch
  limitation.) For GPU-side reduced precision, see
  `wenet/bin/export_onnx_gpu.py --fp16` (ONNX export, a separate fp16-only
  workflow).

---

### Reference (`docs/reference.rst`)

Reference
=========

.. toctree::
   :maxdepth: 1
   :caption: Contents:

   ./papers.md
   ./python_api/modules.rst

---

### Requirements (`docs/requirements.txt`)

jinja2
nbsphinx
sphinx
recommonmark
sphinx-markdown-tables
sphinx-rtd-theme
torch
torchaudio
typeguard

---

### Runtime (`docs/runtime.md`)

# Runtime for WeNet

WeNet runtime uses [Unified Two Pass (U2)](https://arxiv.org/pdf/2102.01547.pdf) framework for inference. U2 has the following advantages:
* **Unified**: U2 unified the streaming and non-streaming model in a simple way, and our runtime is also unified. Therefore you can easily balance the latency and accuracy by changing chunk_size (described in the following section).
* **Accurate**: U2 achieves better accuracy by CTC joint training.
* **Fast**: Our runtime uses attention rescoring based decoding method described in U2, which is much faster than a traditional autoregressive beam search.
* **Other benefits**: In practice, we find U2 is more stable on long-form speech than standard transformer which usually fails or degrades a lot on long-form speech; and we can easily get the word-level time stamps by CTC spikes in U2. Both of these aspects are favored for industry adoption.

## Platforms Supported

The WeNet runtime supports the following platforms.

* Server
  * [x86](https://github.com/wenet-e2e/wenet/tree/main/runtime/libtorch)
* Device
  * [android](https://github.com/wenet-e2e/wenet/tree/main/runtime/android)

## Architecture and Implementation

### Architecture

The following picture shows how U2 works.



When input is not finished, the input frames $x_t$ are fed into the *Shared Encoder* module frame by frame to get the encoder output $e_t$, then $e_t$ is transformed by the *CTC Activation* module (typically, it's just a linear transform with a log_softmax) to get the CTC prob $y_t$ at current frame, and $y_t$ is further used by the *CTC prefix beam search* module to generate n-best results at current time $t$, and the best result is used as partial result of the U2 system.

When input is finished at time $T$, the n-best results from the *CTC prefix beam search* module and the encoder output $e_1, e_2, e_3, ..., e_T$ are fed into the *Attention Decoder* module, then the *Attention Decoder* module computes a score for every result. The result with the best score is selected as the final result of U2 system.

We can group $C$ continuous frames $x_t, x_{t+1}, x_{t+C}$ as one chunk for the *Shared Encoder* module, and $C$ is called chunk_size in the U2 framework. The chunk_size will affect the attention computation in the *Shared Encoder* module. When chunk_size is infinite, it is a non-streaming case. The system gives the best accuracy with infinite latency. When chunk_size is limited (typically less than 1s), it is a streaming case. The system has limited latency and also gives promising accuracy. So the developer can balance the accuracy and latency and setting a proper chunk_size.

### Interface Design

We use LibTorch to implement U2 runtime in WeNet, and we export several interfaces in PyTorch python code
by @torch.jit.export (see [asr_model.py](https://github.com/wenet-e2e/wenet/tree/main/wenet/transformer/asr_model.py)),
which are required and used in C++ runtime in [torch_asr_model.cc](https://github.com/wenet-e2e/wenet/tree/main/runtime/libtorch/decoder/torch_asr_model.cc).
Here we just list the interface and give a brief introduction.

| interface                        | description                             |
|----------------------------------|-----------------------------------------|
| subsampling_rate (args)          | get the subsampling rate of the model   |
| right_context (args)             | get the right context of the model      |
| sos_symbol (args)                | get the sos symbol id of the model      |
| eos_symbol (args)                | get the eos symbol id of the model      |
| forward_encoder_chunk (args)     | used for the *Shared Encoder* module    |
| ctc_activation (args)            | used for the *CTC Activation* module    |
| forward_attention_decoder (args) | used for the *Attention Decoder* module |

### Cache in Details

For streaming scenario, the *Shared Encoder* module works in an incremental way. The current chunk computation requries the inputs and outputs of all the history chunks. We implement the incremental computation by using caches. Overall, two types of cache are used in our runtime.

* att_cache: the attention cache of the *Shared Encoder*(Conformer/Transformer) module.
* cnn_cache: the cnn cache of the *Shared Encoder*, which caches the left context for causal CNN computation in Conformer.

Please see [encoder.py:forward_chunk()](https://github.com/wenet-e2e/wenet/tree/main/wenet/transformer/encoder.py) and [torch_asr_model.cc](https://github.com/wenet-e2e/wenet/tree/main/runtime/libtorch/decoder/torch_asr_model.cc) for details of the caches.

In practice, CNN is also used for subsampling, we should handle the CNN cache in subsampling.
However, there are different CNN layers in subsampling with different left contexts, right contexts and strides, which makes it tircky to directly implement the CNN cache in subsampling.
In our implementation, we simply overlap the input to avoid subsampling CNN cache.
It is simple and straightforward with negligible additional cost since subsampling CNN only costs a very small fraction of the whole computation.
The following picture shows how it works, where the blue color is for the overlap part of current inputs and previous inputs.



## References
1. [Sequence Modeling With CTC](https://distill.pub/2017/ctc/)
2. [First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs](https://arxiv.org/pdf/1408.2873.pdf)
3. [Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition](https://arxiv.org/pdf/2012.05481.pdf)

---

### Train (`docs/train.rst`)

How to train models?
====================

.. toctree::
   :maxdepth: 1
   :caption: Contents:

   ./tutorial_librispeech.md
   ./tutorial_aishell.md
   ./pretrained_models.md
   ./UIO.md

---

### Tutorial Aishell (`docs/tutorial_aishell.md`)

## Tutorial on AIShell

If you meet any problems when going through this tutorial, please feel free to ask in github [issues](https://github.com/mobvoi/wenet/issues). Thanks for any kind of feedback.


### First Experiment

We provide a recipe `example/aishell/s0/run.sh` on aishell-1 data.

The recipe is simple and we suggest you run each stage one by one manually and check the result to understand the whole process.

```
cd example/aishell/s0
bash run.sh --stage -1 --stop_stage -1
bash run.sh --stage 0 --stop_stage 0
bash run.sh --stage 1 --stop_stage 1
bash run.sh --stage 2 --stop_stage 2
bash run.sh --stage 3 --stop_stage 3
bash run.sh --stage 4 --stop_stage 4
bash run.sh --stage 5 --stop_stage 5
bash run.sh --stage 6 --stop_stage 6
```

You could also just run the whole script
```
bash run.sh --stage -1 --stop_stage 6
```


#### Stage -1: Download data

This stage downloads the aishell-1 data to the local path `$data`. This may take several hours. If you have already downloaded the data, please change the `$data` variable in `run.sh` and start from `--stage 0`.
Please set a **absolute path** for `$data`, e.g. `/home/username/asr-data/aishell/`

#### Stage 0: Prepare Training data

In this stage, `local/aishell_data_prep.sh` organizes the original aishell-1 data into two files:
* **wav.scp** each line records two tab-separated columns : `wav_id` and `wav_path`
* **text**  each line records two tab-separated columns :  `wav_id` and `text_label`

**wav.scp**
```
BAC009S0002W0122 /export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0122.wav
BAC009S0002W0123 /export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0123.wav
BAC009S0002W0124 /export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0124.wav
BAC009S0002W0125 /export/data/asr-data/OpenSLR/33/data_aishell/wav/train/S0002/BAC009S0002W0125.wav
...
```

**text**
```
BAC009S0002W0122 而对楼市成交抑制作用最大的限购
BAC009S0002W0123 也成为地方政府的眼中钉
BAC009S0002W0124 自六月底呼和浩特市率先宣布取消限购后
BAC009S0002W0125 各地政府便纷纷跟进
...
```

If you want to train using your customized data, just organize the data into two files `wav.scp` and `text`, and start from `stage 1`.


#### Stage 1: Extract optinal cmvn features

`example/aishell/s0` uses raw wav as input and and [TorchAudio](https://pytorch.org/audio/stable/index.html) to extract the features just-in-time in dataloader. So in this step we just copy the training wav.scp and text file into the `raw_wav/train/` dir.

`tools/compute_cmvn_stats.py` is used to extract global cmvn(cepstral mean and variance normalization) statistics. These statistics will be used to normalize the acoustic features. Setting `cmvn=false` will skip this step.

#### Stage 2: Generate label token dictionary

The dict is a map between label tokens (we use characters for Aishell-1) and
 the integer indices.

An example dict is as follows
```
<blank> 0
<unk> 1
一 2
丁 3
...
龚 4230
龟 4231
<sos/eos> 4232
```

* `<blank>` denotes the blank symbol for CTC.
* `<unk>` denotes the unknown token, any out-of-vocabulary tokens will be mapped into it.
* `<sos/eos>` denotes start-of-speech and end-of-speech symbols for attention based encoder decoder training, and they shares the same id.

#### Stage 3: Prepare WeNet data format

This stage generates the WeNet required format file `data.list`. Each line in `data.list` is in json format which contains the following fields.

1. `key`: key of the utterance
2. `wav`: audio file path of the utterance
3. `txt`: normalized transcription of the utterance, the transcription will be tokenized to the model units on-the-fly at the training stage.

Here is an example of the `data.list`, and please see the generated training feature file in `data/train/data.list`.

```
{"key": "BAC009S0002W0122", "wav": "/export/data/asr-data/OpenSLR/33//data_aishell/wav/train/S0002/BAC009S0002W0122.wav", "txt": "而对楼市成交抑制作用最大的限购"}
{"key": "BAC009S0002W0123", "wav": "/export/data/asr-data/OpenSLR/33//data_aishell/wav/train/S0002/BAC009S0002W0123.wav", "txt": "也成为地方政府的眼中钉"}
{"key": "BAC009S0002W0124", "wav": "/export/data/asr-data/OpenSLR/33//data_aishell/wav/train/S0002/BAC009S0002W0124.wav", "txt": "自六月底呼和浩特市率先宣布取消限购后"}
```

We aslo design another format for `data.list` named `shard` which is for big data training.
Please see [gigaspeech](https://github.com/wenet-e2e/wenet/tree/main/examples/gigaspeech/s0)(10k hours) or
[wenetspeech](https://github.com/wenet-e2e/wenet/tree/main/examples/wenetspeech/s0)(10k hours)
for how to use `shard` style `data.list` if you want to apply WeNet on big data set(more than 5k).

#### Stage 4: Neural Network training

The NN model is trained in this step.

- Multi-GPU mode

If using DDP mode for multi-GPU, we suggest using `dist_backend="nccl"`. If the NCCL does not work, try using `gloo` or use `torch==1.6.0`
Set the GPU ids in CUDA_VISIBLE_DEVICES. For example, set `export CUDA_VISIBLE_DEVICES="0,1,2,3,6,7"` to use card 0,1,2,3,6,7.

- Resume training

If your experiment is terminated after running several epochs for some reasons (e.g. the GPU is accidentally used by other people and is out-of-memory ), you could continue the training from a checkpoint model. Just find out the finished epoch in `exp/your_exp/`, set  `checkpoint=exp/your_exp/$n.pt` and run the `run.sh --stage 4`. Then the training will continue from the $n+1.pt

- Config

The config of neural network structure, optimization parameter, loss parameters, and dataset can be set in a YAML format file.

In `conf/`,  we provide several models like transformer and conformer. see `conf/train_conformer.yaml` for reference.

- Use Tensorboard

The training takes several hours. The actual time depends on the number and type of your GPU cards. In an 8-card 2080 Ti machine, it takes about less than one day for 50 epochs.
You could use tensorboard to monitor the loss.

```
tensorboard --logdir tensorboard/$your_exp_name/ --port 12598 --bind_all
```

#### Stage 5: Recognize wav using the trained model

This stage shows how to recognize a set of wavs into texts. It also shows how to do the model averaging.

- Average model

If `${average_checkpoint}` is set to `true`, the best `${average_num}` models on cross validation set will be averaged to generate a boosted model and used for recognition.

- Decoding

Recognition is also called decoding or inference. The function of the NN will be applied on the input acoustic feature sequence to output a sequence of text.

Four decoding methods are provided in WeNet:

* `ctc_greedy_search` : encoder + CTC greedy search
* `ctc_prefix_beam_search` :  encoder + CTC prefix beam search
* `attention` : encoder + attention-based decoder decoding
* `attention_rescoring` : rescoring the ctc candidates from ctc prefix beam search with encoder output on attention-based decoder.

In general, attention_rescoring is the best method. Please see [U2 paper](https://arxiv.org/pdf/2012.05481.pdf) for the details of these algorithms.

`--beam_size` is a tunable parameter, a large beam size may get better results but also cause higher computation cost.

`--batch_size` can be greater than 1 for "ctc_greedy_search" and "attention" decoding mode, and must be 1 for "ctc_prefix_beam_search" and "attention_rescoring" decoding mode.

- WER evaluation

`tools/compute-wer.py` will calculate the word (or char) error rate of the result. If you run the recipe without any change, you may get WER ~= 5%.


#### Stage 6: Export the trained model

`wenet/bin/export_jit.py` will export the trained model using Libtorch. The exported model files can be easily used for inference in other programming languages such as C++.

---

### Tutorial Librispeech (`docs/tutorial_librispeech.md`)

## Tutorial on LibriSpeech

If you meet any problems when going through this tutorial, please feel free to ask in github [issues](https://github.com/mobvoi/wenet/issues). Thanks for any kind of feedback.


### First Experiment

We provide a recipe `example/librispeech/s0/run.sh` on librispeech data.

The recipe is simple and we suggest you run each stage one by one manually and check the result to understand the whole process.

```
cd example/librispeech/s0
bash run.sh --stage -1 --stop_stage -1
bash run.sh --stage 0 --stop_stage 0
bash run.sh --stage 1 --stop_stage 1
bash run.sh --stage 2 --stop_stage 2
bash run.sh --stage 3 --stop_stage 3
bash run.sh --stage 4 --stop_stage 4
bash run.sh --stage 5 --stop_stage 5
bash run.sh --stage 6 --stop_stage 6
bash run.sh --stage 7 --stop_stage 7
```

You could also just run the whole script
```
bash run.sh --stage -1 --stop_stage 7
```


#### Stage -1: Download data

``` sh
data_url=www.openslr.org/resources/12
datadir=/export/data/en-asr-data/OpenSLR/
if [ ${stage} -le -1 ] && [ ${stop_stage} -ge -1 ]; then
  echo "stage -1: Data Download"
  for part in dev-clean test-clean dev-other test-other train-clean-100 train-clean-360 train-other-500; do
    local/download_and_untar.sh ${datadir} ${data_url} ${part}
  done
fi
```

This stage downloads the librispeech data to the local path `$data`. This may take several hours. If you have already downloaded the data, please change the `$data` variable in `run.sh` and start from `--stage 0`.

#### Stage 0: Prepare Training data

``` sh
wave_data=data
if [ ${stage} -le 0 ] && [ ${stop_stage} -ge 0 ]; then
  for part in dev-clean test-clean dev-other test-other train-clean-100 train-clean-360 train-other-500; do
    # use underscore-separated names in data directories.
    local/data_prep_torchaudio.sh ${datadir}/LibriSpeech/${part} $wave_data/${part//-/_}
  done
fi
```

In this stage, `local/data_prep_torchaudio.sh` organizes the original data into two files:
* **wav.scp** each line records two tab-separated columns : `wav_id` and `wav_path`
* **text**  each line records two tab-separated columns :  `wav_id` and `text_label`

**wav.scp**
```
1867-154075-0014 /export/data/en-asr-data/OpenSLR//LibriSpeech/train-clean-100/1867/154075/1867-154075-0014.flac
1970-26100-0022 /export/data/en-asr-data/OpenSLR//LibriSpeech/train-clean-100/1970/26100/1970-26100-0022.flac
...
```

**text**
```
1867-154075-0014 YOU SHOW HIM THAT IT IS POSSIBLE
1970-26100-0022 DID YOU SEE HIM AT THAT TIME
...
```

If you want to train using your customized data, just organize the data into two files `wav.scp` and `text`, and start from `stage 1`.


#### Stage 1: Extract optinal cmvn features

``` sh
if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
  ### Task dependent. You have to design training and dev sets by yourself.
  ### But you can utilize Kaldi recipes in most cases
  echo "stage 1: Feature Generation"
  mkdir -p $wave_data/train_960
  # merge total training data
  for set in train_clean_100 train_clean_360 train_other_500; do
    for f in `ls $wave_data/$set`; do
      cat $wave_data/$set/$f >> $wave_data/train_960/$f
    done
  done
  mkdir -p $wave_data/dev
  # merge total dev data
  for set in dev_clean dev_other; do
    for f in `ls $wave_data/$set`; do
      cat $wave_data/$set/$f >> $wave_data/dev/$f
    done
  done

  tools/compute_cmvn_stats.py --num_workers 16 --train_config $train_config \
    --in_scp $wave_data/$train_set/wav.scp \
    --out_cmvn $wave_data/$train_set/global_cmvn

fi
```

The librispeech corpus contains 3 subsets for training, namely `train_clean_100`, `train_clean_360`, and `train_other_500`,
so we first merge them to get our final training data.

`tools/compute_cmvn_stats.py` is used to extract global cmvn(cepstral mean and variance normalization) statistics. These statistics will be used to normalize the acoustic features. Setting `cmvn=false` will skip this step.

#### Stage 2: Generate label token dictionary

``` sh
dict=$wave_data/lang_char/${train_set}_${bpemode}${nbpe}_units.txt
bpemodel=$wave_data/lang_char/${train_set}_${bpemode}${nbpe}
echo "dictionary: ${dict}"
if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
  ### Task dependent. You have to check non-linguistic symbols used in the corpus.
  echo "stage 2: Dictionary and Json Data Preparation"
  mkdir -p data/lang_char/

  echo "<blank> 0" > ${dict} # 0 will be used for "blank" in CTC
  echo "<unk> 1" >> ${dict} # <unk> must be 1

  # we borrowed these code and scripts which are related bpe from ESPnet.
  cut -f 2- -d" " $wave_data/${train_set}/text > $wave_data/lang_char/input.txt
  tools/spm_train --input=$wave_data/lang_char/input.txt --vocab_size=${nbpe} --model_type=${bpemode} --model_prefix=${bpemodel} --input_sentence_size=100000000
  tools/spm_encode --model=${bpemodel}.model --output_format=piece < $wave_data/lang_char/input.txt | tr ' ' '\n' | sort | uniq | awk '{print $0 " " NR+1}' >> ${dict}
  num_token=$(cat $dict | wc -l)
  echo "<sos/eos> $num_token" >> $dict # <eos>
  wc -l ${dict}
fi
```

The model unit of English e2e speech recognition system could be char or BPE(byte-pair-encoding).
Typically, BPE shows better result. So here we use BPE as model unit,
and the BPE is trained by [sentencepiece](https://github.com/google/sentencepiece) tool on the librispeech training data.

The model unit is defined as a dict in WeNet, which maps the a BPE into integer index.
The librispeech dict is like:

```
<blank> 0
<unk> 1
' 2
▁ 3
A 4
▁A 5
AB 6
▁AB 7
▁YOU 4995
▁YOUNG 4996
▁YOUR 4997
▁YOUTH 4998
Z 4999
ZZ 5000
<sos/eos> 5001
```

* `<blank>` denotes the blank symbol for CTC.
* `<unk>` denotes the unknown token, any out-of-vocabulary tokens will be mapped into it.
* `<sos/eos>` denotes start-of-speech and end-of-speech symbols for attention based encoder decoder training, and they shares the same id.

#### Stage 3: Prepare WeNet data format

``` sh
if [ ${stage} -le 3 ] && [ ${stop_stage} -ge 3 ]; then
  # Prepare wenet required data
  echo "Prepare data, prepare required format"
  for x in dev ${recog_set} $train_set ; do
    tools/make_raw_list.py $wave_data/$x/wav.scp $wave_data/$x/text \
        $wave_data/$x/data.list
  done

fi
```

This stage generates the WeNet required format file `data.list`. Each line in `data.list` is in json format which contains the following fields.

1. `key`: key of the utterance
2. `wav`: audio file path of the utterance
3. `txt`: normalized transcription of the utterance, the transcription will be tokenized to the model units on-the-fly at the training stage.

Here is an example of the `data.list`, and please see the generated training feature file in `data/train/data.list`.

```
{"key": "1455-134435-0000", "wav": "/mnt/nfs/ptm1/open-data/LibriSpeech/train-clean-100/1455/134435/1455-134435-0000.flac", "txt": "THE GIRL WHO CAME INTO THE WORLD ON THAT NIGHT WHEN JESSE RAN THROUGH THE FIELDS CRYING TO GOD THAT HE BE GIVEN A SON HAD GROWN TO WOMANHOOD ON THE FARM"}
{"key": "1455-134435-0001", "wav": "/mnt/nfs/ptm1/open-data/LibriSpeech/train-clean-100/1455/134435/1455-134435-0001.flac", "txt": "AND WHEN NOT ANGRY SHE WAS OFTEN MOROSE AND SILENT IN WINESBURG IT WAS SAID THAT SHE DRANK HER HUSBAND THE BANKER"}
{"key": "1455-134435-0002", "wav": "/mnt/nfs/ptm1/open-data/LibriSpeech/train-clean-100/1455/134435/1455-134435-0002.flac", "txt": "BUT LOUISE COULD NOT BE MADE HAPPY SHE FLEW INTO HALF INSANE FITS OF TEMPER DURING WHICH SHE WAS SOMETIMES SILENT SOMETIMES NOISY AND QUARRELSOME SHE SWORE AND CRIED OUT IN HER ANGER SHE GOT A KNIFE FROM THE KITCHEN AND THREATENED HER HUSBAND'S LIFE"}
```

We aslo design another format for `data.list` named `shard` which is for big data training.
Please see [gigaspeech](https://github.com/wenet-e2e/wenet/tree/main/examples/gigaspeech/s0)(10k hours) or
[wenetspeech](https://github.com/wenet-e2e/wenet/tree/main/examples/wenetspeech/s0)(10k hours)
for how to use `shard` style `data.list` if you want to apply WeNet on big data set(more than 5k).

#### Stage 4: Neural Network training

``` sh
if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then
  # Training
  mkdir -p $dir
  INIT_FILE=$dir/ddp_init
  rm -f $INIT_FILE # delete old one before starting
  init_method=file://$(readlink -f $INIT_FILE)
  echo "$0: init method is $init_method"
  num_gpus=$(echo $CUDA_VISIBLE_DEVICES | awk -F "," '{print NF}')
  # Use "nccl" if it works, otherwise use "gloo"
  dist_backend="nccl"
  cmvn_opts=
  $cmvn && cmvn_opts="--cmvn $wave_data/${train_set}/global_cmvn"
  # train.py will write $train_config to $dir/train.yaml with model input
  # and output dimension, train.yaml will be used for inference or model
  # export later
  for ((i = 0; i < $num_gpus; ++i)); do
  {
    gpu_id=$(echo $CUDA_VISIBLE_DEVICES | cut -d',' -f$[$i+1])
    python wenet/bin/train.py --gpu $gpu_id \
      --config $train_config \
      --data_type raw \
      --symbol_table $dict \
      --train_data $wave_data/$train_set/data.list \
      --cv_data $wave_data/dev/data.list \
      ${checkpoint:+--checkpoint $checkpoint} \
      --model_dir $dir \
      --ddp.init_method $init_method \
      --ddp.world_size $num_gpus \
      --ddp.rank $i \
      --ddp.dist_backend $dist_backend \
      --num_workers 1 \
      $cmvn_opts \
      --pin_memory
  } &
  done
  wait
fi
```

The NN model is trained in this step.

- Multi-GPU mode

If using DDP mode for multi-GPU, we suggest using `dist_backend="nccl"`. If the NCCL does not work, try using `gloo` or use `torch==1.6.0`
Set the GPU ids in CUDA_VISIBLE_DEVICES. For example, set `export CUDA_VISIBLE_DEVICES="0,1,2,3,6,7"` to use card 0,1,2,3,6,7.

- Resume training

If your experiment is terminated after running several epochs for some reasons (e.g. the GPU is accidentally used by other people and is out-of-memory ), you could continue the training from a checkpoint model. Just find out the finished epoch in `exp/your_exp/`, set  `checkpoint=exp/your_exp/$n.pt` and run the `run.sh --stage 4`. Then the training will continue from the $n+1.pt

- Config

The config of neural network structure, optimization parameter, loss parameters, and dataset can be set in a YAML format file.

In `conf/`,  we provide several models like transformer and conformer. see `conf/train_conformer.yaml` for reference.

- Use Tensorboard

The training takes several hours. The actual time depends on the number and type of your GPU cards. In an 8-card 2080 Ti machine, it takes about less than one day for 50 epochs.
You could use tensorboard to monitor the loss.

```
tensorboard --logdir tensorboard/$your_exp_name/ --port 12598 --bind_all
```

#### Stage 5: Recognize wav using the trained model

```
/* Detailed source-code truncated for AI context efficiency. */
```

This stage shows how to recognize a set of wavs into texts. It also shows how to do the model averaging.

- Average model

If `${average_checkpoint}` is set to `true`, the best `${average_num}` models on cross validation set will be averaged to generate a boosted model and used for recognition.

- Decoding

Recognition is also called decoding or inference. The function of the NN will be applied on the input acoustic feature sequence to output a sequence of text.

Four decoding methods are provided in WeNet:

* `ctc_greedy_search` : encoder + CTC greedy search
* `ctc_prefix_beam_search` :  encoder + CTC prefix beam search
* `attention` : encoder + attention-based decoder decoding
* `attention_rescoring` : rescoring the ctc candidates from ctc prefix beam search with encoder output on attention-based decoder.

In general, attention_rescoring is the best method. Please see [U2 paper](https://arxiv.org/pdf/2012.05481.pdf) for the details of these algorithms.

`--beam_size` is a tunable parameter, a large beam size may get better results but also cause higher computation cost.

`--batch_size` can be greater than 1 for "ctc_greedy_search" and "attention" decoding mode, and must be 1 for "ctc_prefix_beam_search" and "attention_rescoring" decoding mode.

- WER evaluation

`tools/compute-wer.py` will calculate the word (or char) error rate of the result.


#### Stage 6(Optional): Export the trained model

``` sh
if [ ${stage} -le 6 ] && [ ${stop_stage} -ge 6 ]; then
  # Export the best model you want
  python wenet/bin/export_jit.py \
    --config $dir/train.yaml \
    --checkpoint $dir/avg_${average_num}.pt \
    --output_file $dir/final.zip
fi
```

`wenet/bin/export_jit.py` will export the trained model using Libtorch.
The exported model files can be easily used for C++ inference in our runtime.
It is required if you want to integrate language model(LM), as shown in Stage 7.


#### Stage 7(Optional): Add LM and test it with runtime



```
/* Detailed source-code truncated for AI context efficiency. */
```

LM is  only supported in runtime, you have to build the runtime as shown in [Installation](https://github.com/wenet-e2e/wenet#installation),
and please refer [LM for WeNet](https://wenet-e2e.github.io/wenet/lm.html) for the details of LM design.

---

--- METRICS ---
- Files Extracted: 60
- Estimated Token Budget: ~51430 tokens
- Canonical Reference: https://codewiki.google/github.com/wenet-e2e/wenet