## File: README.md # wav2letter++ [](https://app.circleci.com/pipelines/github/flashlight/wav2letter) [](https://gitter.im/wav2letter/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) ## Important Note: ### wav2letter has been moved and consolidated [into Flashlight](https://github.com/flashlight/flashlight) in the [ASR application](https://github.com/flashlight/flashlight/tree/master/flashlight/app/asr). Future wav2letter development will occur in Flashlight. *To build the old, pre-consolidation version of wav2letter*, checkout the [wav2letter v0.2](https://github.com/flashlight/wav2letter/releases/tag/v0.2) release, which depends on the old [Flashlight v0.2](https://github.com/flashlight/flashlight/releases/tag/v0.2) release. The [`wav2letter-lua`](https://github.com/flashlight/wav2letter/tree/wav2letter-lua) project can be found on the [`wav2letter-lua` branch](https://github.com/flashlight/wav2letter/tree/wav2letter-lua), accordingly. For more information on wav2letter++, see or cite [this arXiv paper](https://arxiv.org/abs/1812.07625). ## Recipes This repository includes recipes to reproduce the following research papers as well as *pre-trained* models. **All results reproduction must use Flashlight <= 0.3.2** for exact reproducability. Papers contained here include: - [Pratap et al. (2020): Scaling Online Speech Recognition Using ConvNets](recipes/streaming_convnets/) - [Synnaeve et al. (2020): End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](recipes/sota/2019) - [Kahn et al. (2020): Self-Training for End-to-End Speech Recognition](recipes/self_training) - [Likhomanenko et al. (2019): Who Needs Words? Lexicon-free Speech Recognition](recipes/lexicon_free/) - [Hannun et al. (2019): Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions](recipes/seq2seq_tds/) Data preparation for training and evaluation can be found in [data](data) directory. ### Building the Recipes First, install [Flashlight](https://github.com/flashlight/flashlight/tree/0.3) **(using the [0.3 branch](https://github.com/flashlight/flashlight/tree/0.3) is required)** with the [ASR application](https://github.com/flashlight/flashlight/tree/master/flashlight/app/asr). ```shell mkdir build && cd build cmake .. && make -j8 ``` If Flashlight or ArrayFire are installed in nonstandard paths via a custom `CMAKE_INSTALL_PREFIX`, they can be found by passing ```shell -Dflashlight_DIR=[PREFIX]/usr/share/flashlight/cmake/ -DArrayFire_DIR=[PREFIX]/usr/share/ArrayFire/cmake ``` when running `cmake`. ## Join the wav2letter community * Facebook page: https://www.facebook.com/groups/717232008481207/ * Google group: https://groups.google.com/forum/#!forum/wav2letter-users * Contact: vineelkpratap@fb.com, awni@fb.com, qiantong@fb.com, jacobkahn@fb.com, antares@fb.com, avidov@fb.com, gab@fb.com, vitaliy888@fb.com, locronan@fb.com ## License wav2letter++ is MIT-licensed, as found in the [LICENSE](LICENSE) file. --- ## File: data/mls/README.md # A Recipe for the MLS corpus. Multilingual LibriSpeech (MLS) dataset is a large multilingual corpus suitable for speech research. The dataset is derived from read audiobooks from LibriVox and consists of 8 languages - English, German, Dutch, Spanish, French, Italian, Portuguese, Polish. It is available at [OpenSLR](http://openslr.org/94). ## Steps to download and prepare the audio and text data First download the dataset and untar the file. Replace `[lang]` with appropriate language. ``` wget https://dl.fbaipublicfiles.com/mls/mls_[lang].tar.gz tar -I pigz -xf mls_[lang].tar.gz ``` Prepare train, dev and test sets as list files to be used for training with wav2letter. Replace `[...]` with appropriate paths ``` python prepare.py -indir [...]/mls_[lang] -outdir [...] ``` The following structure will be generated ``` > tree . └── lists ├── dev.lst ├── test.lst └── train.lst ``` --- ## File: data/librispeech/README.md # A Recipe for the Librispeech corpus. The Librispeech corpus consists of about 1000 hours of read English speech. The Librispeech corpus can be downloaded for free from [here](http://www.openslr.org/12) ## Wav2Letter models | DEV-CLEAN WER % / LER % | DEV-OTHER WER % / LER % | TEST-CLEAN WER % / LER % | TEST-OTHER WER % / LER % | MODEL | PAPER | |:-------------------------:|:-------------------------:|:--------------------------:|:--------------------------:|:------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------:| | - | 3.52 / - | - | 4.11 / - | [SOTA 2019](https://github.com/flashlight/wav2letter/tree/master/recipes/sota/2019)| [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://arxiv.org/abs/1911.08460)| | - | 7.27 / - | - | 7.95 / - | [streaming_convnets](https://github.com/flashlight/wav2letter/tree/master/recipes/streaming_convnets)|[Scaling Up Online Speech Recognition Using ConvNets](https://research.fb.com/publications/scaling-up-online-speech-recognition-using-convnets/)| | 3.01 / - | 8.86 / - | 3.28 / - | 9.84 / - | [seq2seq_tds](https://github.com/flashlight/wav2letter/tree/master/recipes/seq2seq_tds/librispeech)|[Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions](https://arxiv.org/abs/1904.02619)| | 4.6 / 2.3 | 13.8 / 9.0 | 4.8 / - | 14.5 / - | [conv_glu](https://github.com/flashlight/wav2letter/tree/master/recipes/conv_glu/librispeech) | [Wav2Letter: an End-to-End ConvNet-based Speech Recognition System](https://arxiv.org/pdf/1609.03193.pdf), [Letter-Based Speech Recognition with Gated ConvNets](https://arxiv.org/pdf/1712.09444.pdf) | ## Steps to download and prepare the audio and text data To download and prepare the audio and text for training/evaluation run (replace [...] with a path where the data should be stored) ``` python3 prepare.py --dst [...] ``` The following structure will be generated ``` tree -L 3 . ├── audio │ ├── dev-clean.tar │ ├── dev-other.tar │ ├── LibriSpeech │ │ ├── BOOKS.TXT │ │ ├── CHAPTERS.TXT │ │ ├── dev-clean │ │ ├── dev-other │ │ ├── LICENSE.TXT │ │ ├── README.TXT │ │ ├── SPEAKERS.TXT │ │ ├── test-clean │ │ ├── test-other │ │ ├── train-clean-100 │ │ ├── train-clean-360 │ │ └── train-other-500 │ ├── test-clean.tar │ ├── test-other.tar │ ├── train-clean-100.tar │ ├── train-clean-360.tar │ └── train-other-500.tar ├── lists │ ├── dev-clean.lst │ ├── dev-other.lst │ ├── test-clean.lst │ ├── test-other.lst │ ├── train-clean-100.lst │ ├── train-clean-360.lst │ └── train-other-500.lst └── text ├── dev-clean.txt ├── dev-other.txt ├── librispeech-lm-norm.txt ├── librispeech-lm-norm.txt.lower.shuffle ├── test-clean.txt ├── test-other.txt ├── train-clean-100.txt ├── train-clean-360.txt └── train-other-500.txt ``` --- ## File: data/ami/README.md # A Recipe for the AMI corpus. "The AMI Meeting Corpus consists of 100 hours of meeting recordings. The recordings use a range of signals synchronized to a common timeline. These include close-talking and far-field microphones, individual and room-view video cameras, and output from a slide projector and an electronic whiteboard. During the meetings, the participants also have unsynchronized pens available to them that record what is written. The meetings were recorded in English using three different rooms with different acoustic properties, and include mostly non-native speakers." See http://groups.inf.ed.ac.uk/ami/corpus/overview.shtml for more details. We use the individual headset microphone (IHM) setting for preparing train, dev and test sets. The recipe here is heavily inspired from the preprocessing scripts in Kaldi - https://github.com/kaldi-asr/kaldi/tree/master/egs/ami . ## Steps to download and prepare the audio and text data Prepare train, dev and test sets as list files to be used for training with wav2letter. Replace `[...]` with appropriate paths ``` python prepare.py -dst [...] ``` The above scripts download the AMI data, segments them into shorter `.flac` audio files based on word timestamps. Limited supervision training set for 10min, 1hr and 10hr will be generated as well. The following structure will be generated ``` >tree -L 4 . ├── audio │   ├── EN2001a │   │   ├── EN2001a.Headset-0.wav │   │   ├── ... │   │   └── EN2001a.Headset-4.wav │   ├── EN2001b │   ├── ... │   ├── ... │   ├── IS1009d │   │   ├── ... │   │   └── IS1009d.Headset-3.wav │   └── segments │ ├── ES2005a │ │ ├── ES2005a_H00_MEE018_0.75_1.61.flac │ │ ├── ES2005a_H00_MEE018_13.19_16.05.flac │ │ ├── ... │ │ └── ... │      ├── ... │      └── IS1009d │      ├── ... │ └── ... ├── lists │ ├── dev.lst │ ├── test.lst │ ├── train_10min_0.lst │ ├── train_10min_1.lst │ ├── train_10min_2.lst │ ├── train_10min_3.lst │ ├── train_10min_4.lst │ ├── train_10min_5.lst │ ├── train_9hr.lst │ └── train.lst │ └── text ├── ami_public_manual_1.6.1.zip └── annotations ├── 00README_MANUAL.txt ├── ... ├── transcripts0 ├── transcripts1 ├── transcripts2 ├── words └── youUsages ``` --- ## File: data/timit/README.md # A Recipe for the TIMIT corpus. The TIMIT corpus can be purchased from the LDC [LDC93S1](https://catalog.ldc.upenn.edu/LDC93S1). The data is mapped from 61 to 39 phonemes for training and testing. The map used here is taken from the [Kaldi TIMIT recipe](https://github.com/kaldi-asr/kaldi/blob/master/egs/timit/s5/conf/phones.60-48-39.map). ## Prerequisites - download the data from the LDC [LDC93S1](https://catalog.ldc.upenn.edu/LDC93S1). You will have `timit_LDC93S1.tar`. - unpack file ``` tar -xf timit_LDC93S1.tar ``` - install `sph2pipe` see https://www.ldc.upenn.edu/language-resources/tools/sphere-conversion-tools: ``` wget https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/ctools/sph2pipe_v2.5.tar.gz tar -xzf sph2pipe_v2.5.tar.gz && cd sph2pipe_v2.5 gcc -o sph2pipe *.c -lm ``` ## Preparation of audio and text data To prepare the audio and text data for training/evaluation run (set necessary paths instead of `[...]`) ``` python3 prepare.py --src [...]/timit --dst [...] --sph2pipe [...]/sph2pipe_v2.5/sph2pipe ``` The following structure will be generated ``` tree -L 2 . ├── audio │   ├── test │   ├── train │   └── valid ├── lists │   ├── test.lst │   ├── test.phn.lst │   ├── train.lst │   ├── train.phn.lst │   ├── valid.lst │   └── valid.phn.lst ├── text │   ├── test.phn.txt │   ├── test.txt │   ├── train.phn.txt │   ├── train.txt │   ├── valid.phn.txt │   └── valid.txt ├── timit │   ├── CONVERT │   ├── README.DOC │   ├── SPHERE │   └── TIMIT └── timit_LDC93S1.tar ``` --- ## File: data/wsj/README.md # A Recipe for the Wall Street Journal (WSJ) corpus. The WSJ corpus consists of about 80 hours of read sentences taken from the Wall Street Journal. The WSJ corpus can be purchased from the LDC: - as [LDC93S6B](https://catalog.ldc.upenn.edu/LDC93S6B) (WSJ0) and [LDC94S13B](https://catalog.ldc.upenn.edu/LDC94S13B) (WSJ1) - or the complete version of datasets: [LDC93S6A](https://catalog.ldc.upenn.edu/LDC93S6A) (WSJ0) and [LDC94S13A](https://catalog.ldc.upenn.edu/LDC94S13A) (WSJ1) In these experiments, we use three subsets following the [Kaldi WSJ recipe](https://github.com/kaldi-asr/kaldi/blob/master/egs/wsj/): - train: 37416 utterances, referred to as si284 in Kaldi - dev: 503 utterances, referred to as nov93dev in Kaldi - test: 333 utterances, referred to as nov92 in Kaldi ## Wav2Letter models | NOV93DEV WER % / LER % | NOV92 WER % / LER % | MODEL | PAPER | |-------------------------|-------------------------|------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------| | 9.8 / 7.2 | 5.6 / - | [conv_glu](https://github.com/flashlight/wav2letter/tree/master/recipes/conv_glu/wsj) | [Wav2Letter: an End-to-End ConvNet-based Speech Recognition System](https://arxiv.org/pdf/1609.03193.pdf), [Letter-Based Speech Recognition with Gated ConvNets](https://arxiv.org/pdf/1712.09444.pdf) | ## Prerequisites Here and later we assume that complete version of data is downloaded, but the same steps can be used for `B`-version of WSJ. - download the data from the LDC [WSJ0](https://catalog.ldc.upenn.edu/LDC93S6A) and [WSJ1](https://catalog.ldc.upenn.edu/LDC94S13A). You will have `csr_1_LDC93S6A.tar` and `csr_2_comp_LDC94S13A.tar`. - unpack files ``` tar -xf csr_1_LDC93S6A.tar tar -xf csr_2_comp_LDC94S13A.tar ``` - install `sph2pipe` see https://www.ldc.upenn.edu/language-resources/tools/sphere-conversion-tools: ``` wget https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/ctools/sph2pipe_v2.5.tar.gz tar -xzf sph2pipe_v2.5.tar.gz && cd sph2pipe_v2.5 gcc -o sph2pipe *.c -lm ``` ## Preparation of audio and text data To prepare the audio and text data for training/evaluation run (set necessary paths instead of `[...]`) (if you are using `B`-version of WSJ then call with `--wsj1_type LDC94S13B`) ``` python3 prepare.py --wsj0 [...]/csr_1 --wsj1 [...]/csr_2_comp --sph2pipe [...]/sph2pipe_v2.5/sph2pipe --dst [...] --wsj1_type LDC94S13A ``` The following structure will be generated ``` tree -L 2 . ├── audio │ ├── nov92 │ ├── nov92_5k │ ├── nov93 │ ├── nov93_5k │ ├── nov93dev │ ├── nov93dev_5k │ ├── si284 │ └── si84 ├── csr_1 │ ├── 11-10.1 │ ├── ... │ └── readme.txt ├── csr_1_LDC93S6A.tar ├── csr_2_comp │ ├── 13-10.1 │ ├── ... ├── csr_2_comp_LDC94S13A.tar ├── lists │ ├── nov92_5k.lst │ ├── nov92.lst │ ├── nov93_5k.lst │ ├── nov93dev_5k.lst │ ├── nov93dev.lst │ ├── nov93.lst │ ├── si284.lst │ └── si84.lst └── text ├── lm.txt ├── nov92_5k.txt ├── nov92.txt ├── nov93_5k.txt ├── nov93dev_5k.txt ├── nov93dev.txt ├── nov93.txt ├── si284.txt └── si84.txt ``` --- ## File: data/README.md # Data preparation This directory contains data processing scripts to perform speech recognition using wav2letter++ recipes on the popular datasets. ## Requirements to run scripts - python3 - `pip install sox tqdm numpy` --- ## File: recipes/conv_glu/librispeech/README.md # Steps to reproduce results on Librispeech Run data and auxiliary files (like lexicon, tokens set, etc.) preparation (set necessary paths instead of `[...]`: `data_dst` path to data to store, `model_dst` path to auxiliary path to store, `kenlm` path to prepare language model for decoding) ``` python3 prepare.py --data_dst [...] --model_dst [...] --kenlm [...] ``` Besides data the auxiliary files for acoustic and language models training/evaluation will be generated: ``` cd $MODEL_DST tree -L 2 . ├── am │ ├── lexicon_train+dev.txt │ └── tokens.txt └── decoder ├── 4-gram.arpa ├── 4-gram.bin └── lexicon.txt ``` To run training/decoding: - Fix the paths inside `*.cfg` - Run training with `train.cfg` - Run decoding with `decode*.cfg` --- ## File: recipes/conv_glu/wsj/README.md # Steps to reproduce results on WSJ Run data and auxiliary files (like lexicon, tokens set, etc.) preparation (set necessary paths instead of `[...]`: `data_dst` path to data to store, `model_dst` path to auxiliary path to store, `kenlm` path to prepare language model for decoding; if you are using `B`-version of WSJ then call with `--wsj1_type LDC94S13B`) ``` python3 prepare.py --wsj0 [...]/csr_1 --wsj1 [...]/csr_2_comp --wsj1_type LDC94S13A \ --data_dst [...] --model_dst [...] --sph2pipe [...]/sph2pipe_v2.5/sph2pipe --kenlm [...] ``` Besides data the auxiliary files for acoustic and language models training/evaluation will be generated: ``` cd $MODEL_DST tree -L 2 . ├── am │ ├── lexicon_si284+nov93dev.txt │ └── tokens.txt └── decoder ├── lexicon.txt # it is combination of words from si284 and lm text data ├── lm-4g.arpa └── lm-4g.bin ``` To run training/decoding: - Fix the paths inside `*.cfg` - Run training with `train.cfg` - Run decoding with `decode*.cfg` --- ## File: recipes/conv_glu/README.md # [Wav2Letter: an End-to-End ConvNet-based Speech Recognition System](https://arxiv.org/pdf/1609.03193.pdf) The architecture and flags files are provided for each dataset to reproduce results from the paper (training and decoding steps). Considered benchmarks: - Librispeech - WSJ