# Technical Documentation: k2-fsa/OmniVoice
> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/k2-fsa/OmniVoice)
### Overview (`README.md`)
# OmniVoice 🌍
OmniVoice is a state-of-the-art massively multilingual zero-shot text-to-speech (TTS) model supporting over 600 languages. Built on a novel diffusion language model-style architecture, it generates high-quality speech with superior inference speed, supporting voice cloning and voice design.
**Contents**: [Key Features](#key-features) | [Installation](#installation) | [Quick Start](#quick-start) | [Python API](#python-api) | [Command-Line Tools](#command-line-tools) | [Training & Evaluation](#training--evaluation) | [Discussion](#discussion--communication) | [Citation](#citation)
## Key Features
- **600+ Languages Supported**: The broadest language coverage among zero-shot TTS models ([full list](docs/languages.md)).
- **Voice Cloning**: State-of-the-art voice cloning quality.
- **Voice Design**: Control voices via assigned speaker attributes (gender, age, pitch, dialect/accent, whisper, etc.).
- **Fine-grained Control**: Non-verbal symbols (e.g., `[laughter]`) and pronunciation correction via pinyin or phonemes.
- **Fast Inference**: RTF as low as 0.025 (40x faster than real-time).
- **Diffusion Language Model-style Architecture**: A clean, streamlined, and scalable design that delivers both quality and speed.
---
## Installation
Choose **one** of the following methods: **pip** or **uv**.
### pip
> We recommend using a fresh virtual environment (e.g., `conda`, `venv`, etc.) to avoid conflicts.
**Step 1**: Install PyTorch
NVIDIA GPU
```bash
# Install pytorch with your CUDA version, e.g.
pip install torch==2.8.0+cu128 torchaudio==2.8.0+cu128 --extra-index-url https://download.pytorch.org/whl/cu128
```
> See [PyTorch official site](https://pytorch.org/get-started/locally/) for other versions installation.
Apple Silicon
```bash
pip install torch==2.8.0 torchaudio==2.8.0
```
Intel Arc GPU (XPU)
Intel Arc GPUs (Alchemist and Battlemage architectures) are supported via PyTorch's XPU backend.
1. Install the [Intel GPU drivers](https://dgpu-docs.intel.com/driver/installation.html) for your OS.
2. Install PyTorch with XPU support from Intel's wheel index:
```bash
pip install torch torchaudio --index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
```
> See [Intel's PyTorch XPU guide](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/) for version-specific instructions.
3. Verify the backend is working:
```bash
python -c "import torch; print(torch.xpu.is_available(), torch.xpu.device_count())"
```
**Notes**:
- `flash_attn` is not available on XPU; the model automatically falls back to SDPA.
- Training with packed sequences (`flex_attention`) has partial XPU support; single-GPU SDPA training should work.
- Tested on Arc A310 (Alchemist, 4 GB) and Arc Pro B50 (Battlemage, 16 GB).
**Step 2**: Install OmniVoice (choose one)
```bash
# From PyPI (stable release)
pip install omnivoice
# From the latest source on GitHub (no need to clone)
pip install git+https://github.com/k2-fsa/OmniVoice.git
# For development (clone first, editable install)
git clone https://github.com/k2-fsa/OmniVoice.git
cd OmniVoice
pip install -e .
```
### uv
Clone the repository and sync dependencies:
```bash
git clone https://github.com/k2-fsa/OmniVoice.git
cd OmniVoice
uv sync
```
> **Tip**: Can use mirror with `uv sync --default-index "https://mirrors.aliyun.com/pypi/simple"`
---
## Quick Start
Try OmniVoice without coding:
- Launch the local web UI: `omnivoice-demo --ip 0.0.0.0 --port 8001`
- Or try it directly on [HuggingFace Space](https://huggingface.co/spaces/k2-fsa/OmniVoice)
- Or run it in Google Colab: [](https://colab.research.google.com/github/k2-fsa/OmniVoice/blob/master/docs/OmniVoice.ipynb)
> If you have trouble connecting to HuggingFace when downloading the pre-trained models, set `export HF_ENDPOINT="https://hf-mirror.com"` before running.
For full usage, see the [Python API](#python-api) and [Command-Line Tools](#command-line-tools) sections below.
---
## Python API
OmniVoice supports three generation modes. All features in this section are also available via [command-line tools](#command-line-tools).
### Voice Cloning
Clone a voice from a short reference audio. Provide `ref_audio` and `ref_text`:
```python
from omnivoice import OmniVoice
import soundfile as sf
import torch
model = OmniVoice.from_pretrained(
"k2-fsa/OmniVoice",
device_map="cuda:0",
dtype=torch.float16
)
# Apple Silicon users: use device_map="mps" instead
# Intel Arc GPU users: use device_map="xpu" instead
audio = model.generate(
text="Hello, this is a test of zero-shot voice cloning.",
ref_audio="ref.wav",
ref_text="Transcription of the reference audio.",
) # audio is a list of `np.ndarray` with shape (T,) at 24 kHz.
# If you don't want to input `ref_text` manually, you can directly omit the `ref_text`.
# The model will use Whisper ASR to auto-transcribe it. To use a local copy (or
# a different Whisper model), pass `asr_model_name="..."` to `from_pretrained`.
# To control which device Whisper is loaded on (e.g. another GPU in multi-GPU
# setups, or the CPU), pass `asr_device="cuda:1"` (or `"cpu"`).
sf.write("out.wav", audio[0], 24000)
```
#### Reusing a cloned voice across sessions
Encode the reference audio once, save the resulting prompt, and skip the
audio loading / auto-transcription steps in later sessions:
```python
prompt = model.create_voice_clone_prompt(
ref_audio="ref.wav", ref_text="Transcription of the reference audio."
)
prompt.save("my_voice.pt")
# Later, in a new session:
from omnivoice import VoiceClonePrompt
prompt = VoiceClonePrompt.load("my_voice.pt")
audio = model.generate(text="Hello again!", voice_clone_prompt=prompt)
```
> **Tips**
>
> - Use a 3–10 seconds reference audio clip. Longer audio slows down inference and may degrade cloning quality.
> - For standard pronunciation, use a reference audio in the **same language** as the target speech. In cross-lingual voice cloning (i.e., the reference audio and target speech are in different languages), the generated speech will carry an accent from the reference audio's language.
> - For better results with Arabic numerals, normalize them to words first (e.g., "123" → "one hundred twenty-three"). You can pass `normalize_text=True` to `generate()` to do this automatically (opt-in; install the extra with `pip install "omnivoice[tn]"`, which pulls in [WeTextProcessing](https://github.com/wenet-e2e/WeTextProcessing)):
>
> ```python
> # "I have 2345 apples." is read correctly instead of digit-by-digit.
> audio = model.generate(text="I have 2345 apples.", normalize_text=True)
> ```
>
> Chinese and English use WeTextProcessing; other languages fall back to `num2words` for integers. Inline control syntax (`[laughter]`, `[B EY1 S]`, pinyin tone markers) is preserved. On macOS (Apple Silicon), `pynini` has no wheel — install it via `conda install -c conda-forge pynini` first.
>
> For more tips, see [docs/tips.md](docs/tips.md).
### Voice Design
Describe the desired voice with speaker attributes — no reference audio needed.
Supported attributes: **gender** (male/female), **age** (child to elderly),
**pitch** (very low to very high), **style** (whisper), **English accent**
(American, British, etc.), and **Chinese dialect** (四川话, 陕西话, etc.).
Attributes are comma-separated and freely combinable across categories.
```python
audio = model.generate(
text="Hello, this is a test of zero-shot voice design.",
instruct="female, low pitch, british accent",
)
```
> **Note**: The model is primarily trained on the voice cloning task, so voice cloning is the most stable mode. Voice design is trained on Chinese and English data only. It can generalize to other languages, but may produce unstable results for some low-resource languages or edge cases.
See [docs/voice-design.md](docs/voice-design.md) for the full attribute
reference, Chinese equivalents, and usage tips.
### Auto Voice
Let the model choose a voice automatically:
```python
audio = model.generate(text="This is a sentence without any voice prompt.")
```
### Generation Parameters
All above three modes share the same `model.generate()` API. You can further control the generation behavior via keyword arguments:
```python
audio = model.generate(
text="...",
num_step=32, # diffusion steps (or 16 for faster inference)
speed=1.0, # speed factor (>1.0 faster, <1.0 slower)
duration=10.0, # fixed output duration in seconds (overrides speed)
# ... more options
)
```
See more detailed control in [docs/generation-parameters.md](docs/generation-parameters.md).
### Non-Verbal & Pronunciation Control
OmniVoice supports inline **non-verbal symbols** and **pronunciation correction** within the input text.
**Non-verbal symbols**: Insert tags like `[laughter]` directly in the text to add expressive non-verbal sounds.
```python
audio = model.generate(text="[laughter] You really got me. I didn't see that coming at all.")
```
Supported tags: `[laughter]`, `[sigh]`, `[confirmation-en]`, `[question-en]`, `[question-ah]`, `[question-oh]`, `[question-ei]`, `[question-yi]`, `[surprise-ah]`, `[surprise-oh]`, `[surprise-wa]`, `[surprise-yo]`, `[dissatisfaction-hnn]`.
**Pronunciation control (Chinese)**: Use pinyin with tone numbers to correct specific character pronunciations.
```python
audio = model.generate(text="这批货物打ZHE2出售后他严重SHE2本了，再也经不起ZHE1腾了。")
```
**Pronunciation control (English)**: Use [CMU pronunciation dictionary](https://svn.code.sf.net/p/cmusphinx/code/trunk/cmudict/cmudict.0.7a) (uppercase, in brackets) to override default English pronunciations.
```python
audio = model.generate(text="He plays the [B EY1 S] guitar while catching a [B AE1 S] fish.")
```
---
## Command-Line Tools
Three CLI entry points are provided. The CLI tools support all features available in the Python API (voice cloning, voice design, auto voice, generation parameters, etc.) — all controlled via command-line arguments.
| Command | Description | Source |
|---|---|---|
| `omnivoice-demo` | Interactive Gradio web demo | [omnivoice/cli/demo.py](omnivoice/cli/demo.py) |
| `omnivoice-infer` | Single-item inference | [omnivoice/cli/infer.py](omnivoice/cli/infer.py) |
| `omnivoice-infer-batch` | Batch inference across multiple GPUs | [omnivoice/cli/infer_batch.py](omnivoice/cli/infer_batch.py) |
### Demo
```bash
omnivoice-demo --ip 0.0.0.0 --port 8001
```
Provides a web UI for voice cloning and voice design. See `omnivoice-demo --help` for all options.
### Single Inference
```bash
# Voice Cloning
# ref_text can be omitted (Whisper will auto-transcribe ref_audio to get it).
omnivoice-infer \
--model k2-fsa/OmniVoice \
--text "This is a test for text to speech." \
--ref_audio ref.wav \
--ref_text "Transcription of the reference audio." \
--output hello.wav
# Voice Design
omnivoice-infer --model k2-fsa/OmniVoice \
--text "This is a test for text to speech." \
--instruct "male, British accent" \
--output hello.wav
# Auto Voice
omnivoice-infer \
--model k2-fsa/OmniVoice \
--text "This is a test for text to speech."\
--output hello.wav
```
### Batch Inference
`omnivoice-infer-batch` can distribute batch inference across multiple GPUs, designed for large-scale TTS tasks.
```bash
omnivoice-infer-batch \
--model k2-fsa/OmniVoice \
--test_list test.jsonl \
--res_dir results/
```
The test list is a JSONL file where each line is a JSON object:
```json
{"id": "sample_001", "text": "Hello world", "ref_audio": "/path/to/ref.wav", "ref_text": "Reference transcript", "instruct": "female, british accent", "language_id": "en", "duration": 10.0, "speed": 1.0}
```
Only `id` and `text` are mandatory fields. `ref_audio` and `ref_text` are used in voice cloning mode. `instruct` is used in voice design mode. If no reference audio or instruct are provided, the model will generate text in a random voice.
`language_id`, `duration`, and `speed` are optional. `duration` (in seconds) fixes the output length; `speed` controls the speaking rate. If `duration` and `speed` are both provided, `speed` will be ignored.
### FlashInfer Acceleration
Inference can be accelerated ~2-2.9x losslessly with [FlashInfer](https://github.com/flashinfer-ai/flashinfer) kernels (sequence packing for the CFG cond/uncond pair, fused RMSNorm/RoPE/GEMM kernels, and optional CUDA graphs).
**Installation** (NVIDIA GPUs; pick the index matching your CUDA version, e.g. cu128 for PyTorch built with CUDA 12.8):
```bash
pip install flashinfer-python==0.6.15.post1 "flashinfer-jit-cache==0.6.15.post1+cu128" \
--extra-index-url https://flashinfer.ai/whl/cu128/
```
**Usage** with the batch inference CLI:
```bash
omnivoice-infer-batch \
--model k2-fsa/OmniVoice \
--test_list test.jsonl \
--res_dir results/ \
--batch_size 8 \
--enable_flashinfer true
```
or with the Python API:
```python
from omnivoice.models.omnivoice_flashinfer import apply_flashinfer
model = OmniVoice.from_pretrained("k2-fsa/OmniVoice", device_map="cuda", dtype=torch.float16)
apply_flashinfer(model) # throughput / batched inference
apply_flashinfer(model, enable_cuda_graph=True) # recommended for batch=1 (low latency)
```
CUDA graphs are recommended for single-stream (batch=1) usage, where kernel-launch overhead dominates; at batch >= 4 the plain FlashInfer path is already the fastest configuration.
**Benchmark** (seed-tts zh testset, 2020 samples / 3.3h audio, voice cloning, single H100, fp16, `num_step=32`; Average RTF as reported by `omnivoice-infer-batch`, outputs ASR-verified lossless):
| batch size | baseline | FlashInfer | speedup |
|---|---|---|---|
| 1 | 0.0899 | 0.0430 | 2.1x |
| 1 + CUDA graph | — | 0.0367 | 2.4x |
| 2 | 0.0480 | 0.0245 | 2.0x |
| 4 | 0.0331 | 0.0152 | 2.2x |
| 8 | 0.0298 | **0.0115** | **2.6x** |
---
## Training & Evaluation
See [examples/](examples/) for the complete pipeline — from data preparation to training, evaluation, and finetuning.
---
## Discussion & Communication
You can directly discuss on [GitHub Issues](https://github.com/k2-fsa/OmniVoice/issues).
You can also scan the QR code to join our wechat group or follow our wechat official account.
| Wechat Group | Wechat Official Account |
| ------------ | ----------------------- |
| | |
---
## Community Projects
OmniVoice is supported by a growing ecosystem of community projects.
Explore them in [Community Projects](docs/community-projects.md).
---
## Citation
```bibtex
@article{zhu2026omnivoice,
title={OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models},
author={Zhu, Han and Ye, Lingxuan and Kang, Wei and Yao, Zengwei and Guo, Liyong and Kuang, Fangjun and Han, Zhifeng and Zhuang, Weiji and Lin, Long and Povey, Daniel},
journal={arXiv preprint arXiv:2604.00688},
year={2026}
}
```
---
## Disclaimer
Users are strictly prohibited from using this model for unauthorized voice cloning, voice impersonation, fraud, scams, or any other illegal or unethical activities. All users shall ensure full compliance with applicable local laws, regulations, and ethical standards. The developers assume no liability for any misuse of this model and advocate for responsible AI development and use, encouraging the community to uphold safety and ethical principles in AI research and applications.
---
### Overview (`examples/README.md`)
# OmniVoice Examples
This directory contains scripts and configs for training, fine-tuning, and evaluating OmniVoice.
| Use Case | Script | Description |
|---|---|---|
| Training from scratch | [run_emilia.sh](run_emilia.sh) | Full pipeline on the Emilia dataset (data check, tokenization, training) |
| Fine-tuning | [run_finetune.sh](run_finetune.sh) | Fine-tune from a pretrained checkpoint using your own JSONL data |
| LoRA fine-tuning | [run_finetune_lora.sh](run_finetune_lora.sh) | Parameter-efficient finetuning via PEFT LoRA adapters — see [../docs/lora_finetuning.md](../docs/lora_finetuning.md) |
| Evaluation | [run_eval.sh](run_eval.sh) | Evaluate WER, speaker similarity, and UTMOS on standard test sets |
---
## Training from Scratch (Emilia)
[run_emilia.sh](run_emilia.sh) runs the full pipeline in 3 stages:
| Stage | What it does |
|---|---|
| 0 | Verify the Emilia dataset and JSONL manifests are in place |
| 1 | Tokenize audio into WebDataset shards |
| 2 | Launch multi-GPU training with `accelerate` |
**Prerequisites:**
1. Download the Emilia dataset from [OpenXLab](https://openxlab.org.cn/datasets/Amphion/Emilia) and place it under `download/`:
```
download/Amphion___Emilia
└── raw
├── EN
└── ZH
```
2. Obtain JSONL manifests and place them in `data/emilia/manifests/`:
- `emilia_en_train.jsonl`, `emilia_en_dev.jsonl`
- `emilia_zh_train.jsonl`, `emilia_zh_dev.jsonl`
You can generate them from the raw data, or download pre-processed manifests from [HuggingFace](https://huggingface.co/datasets/zhu-han/Emilia-Manifests).
**Run the full pipeline:**
```bash
bash examples/run_emilia.sh
```
Or run individual stages by setting `stage` and `stop_stage` at the top of the script (e.g. `stage=1`, `stop_stage=1` to only tokenize).
> See [docs/training.md](../docs/training.md) for config details, checkpoint resuming, and TensorBoard monitoring.
---
## Fine-tuning
[run_finetune.sh](run_finetune.sh) fine-tunes from a pretrained checkpoint on your own data.
### Step 1: Prepare Your Data
Create a JSONL manifest where each line describes one audio sample:
```jsonl
{"id": "sample_001", "audio_path": "/data/audio/001.wav", "text": "Hello world", "language_id": "en"}
{"id": "sample_002", "audio_path": "/data/audio/002.wav", "text": "你好世界", "language_id": "zh"}
```
`id`, `audio_path`, and `text` are mandatory. `language_id` is optional.
> See [docs/data_preparation.md](../docs/data_preparation.md) for the full data format specification.
### Step 2: Configure the Script
Edit the variables at the top of `run_finetune.sh`:
```bash
TRAIN_JSONL="data/my_data_train.jsonl" # path to training JSONL
DEV_JSONL="data/my_data_dev.jsonl" # path to dev JSONL
GPU_IDS="0,1" # GPUs to use
NUM_GPUS=2
OUTPUT_DIR="exp/omnivoice_finetune" # output directory
```
### Step 3: Run
```bash
bash examples/run_finetune.sh
```
The script will:
1. Tokenize your audio into WebDataset shards
2. Launch fine-tuning with `accelerate`
Main difference between fine-tuning config ([config/train_config_finetune.json](config/train_config_finetune.json)) and the Emilia training config ([config/train_config_emilia.json](config/train_config_emilia.json)) are:
| Parameter | Emilia (from scratch) | Fine-tune | Why |
|---|---|---|---|
| `init_from_checkpoint` | `null` | `"k2-fsa/OmniVoice"` | Load pretrained weights |
| `steps` | 300,000 | 5,000 | Fewer steps for fine-tuning, can be tuned according to your data/task. |
| `learning_rate` | 1e-4 | 5e-5 | Lower LR for fine-tuning, can be tuned according to your data/task |
To use a different pretrained checkpoint, modify `init_from_checkpoint` in the config file.
If you encounter issues with `flex_attention` on your GPU, use [config/train_config_finetune_sdpa.json](config/train_config_finetune_sdpa.json) instead, which uses SDPA attention for broader compatibility. See [docs/training.md](../docs/training.md#attention-implementation) for details.
---
## Evaluation
Install evaluation dependencies first:
```bash
pip install omnivoice[eval]
# or
uv sync --extra eval
```
Supported test sets: `librispeech_pc`, `seedtts_en`, `seedtts_zh`, `fleurs`, `minimax`.
```bash
bash examples/run_eval.sh
```
> See [docs/evaluation.md](../docs/evaluation.md) for metrics details, test set preparation, and running individual metrics.
---
### Community Projects (`docs/community-projects.md`)
# Community Projects
The following projects are built and maintained by the community. We appreciate all contributions! Note that these projects are not officially supported by the OmniVoice team.
If you have a project you'd like to add, please open a PR.
---
- **[ComfyUI-OmniVoice-TTS](https://github.com/Saganaki22/ComfyUI-OmniVoice-TTS)** —
ComfyUI custom node for OmniVoice text-to-speech generation.
- **[vLLM-Omni](https://github.com/vllm-project/vllm-omni)** —
A framework for efficient model inference with omni-modality model. Supports OmniVoice serving.
- **[audio.cpp](https://github.com/0xShug0/audio.cpp)** —
An all-in-one, pure C++ inference engine for audio models (TTS, ASR, music gen,etc), powered by GGML.
- **[pyVideoTrans](https://github.com/jianchang512/pyvideotrans)** —
Video translation tool with dubbing & subtitles. Supports OmniVoice as a TTS engine.
- **[MLX-Audio](https://github.com/Blaizzy/mlx-audio)** —
TTS, STT, and STS library built on Apple's MLX framework. Supports
OmniVoice among other models for efficient speech processing on Apple Silicon.
- **[OmniVoice-MLX](https://github.com/ailuntx/OmniVoice-MLX)** —
MLX inference backend and conversion/staging tools for running OmniVoice on
Apple Silicon, with community model weights hosted under
[mlx-community](https://huggingface.co/collections/mlx-community/omnivoice-6a06f610945a0b1d85b3b839).
- **[RealtimeTTS](https://github.com/KoljaB/RealtimeTTS)** —
Converts text to speech in realtime. Supports OmniVoice as a TTS engine.
- **[TTS-WebUI](https://github.com/rsxdalv/TTS-WebUI)** —
Gradio web UI for multiple TTS models. Supports OmniVoice as one of its backends.
- **[OmniVoice-Studio](https://github.com/debpalash/OmniVoice-Studio)** —
Desktop application for OmniVoice voice generation.
- **[omnivoice-server](https://github.com/maemreyo/omnivoice-server)** —
OpenAI-compatible HTTP server for serving OmniVoice via `/v1/audio/speech`.
Supports voice profiles for persistent cloning, sentence-level streaming,
and optional Bearer auth.
- **[omnivoice-rs](https://github.com/FerrisMind/omnivoice-rs)** —
GPU-first Rust workspace for OmniVoice inference, parity validation, CLI
execution, and an OpenAI-compatible HTTP server built with Candle.
- **[omnivoice-trtllm](https://github.com/tlitech/omnivoice-trtllm)** —
Deploy OmniVoice TTS model using TensorRT-LLM and Triton Inference Server
on Modal, faster than PyTorch.
- **[Auris](https://github.com/nikhilprasanth/Auris)** —
Offline audiobook reader for EPUB, PDF, and TXT with local OmniVoice TTS, character-aware voices, and per-book narrator control.
- **[LA Studio](https://github.com/dduongtrandai/LA-Studio)** —
Offline desktop AI audio workstation for local TTS, voice cloning, and voice design workflows with OmniVoice support.
- **[tts-audiobook-tool](https://github.com/zeropointnine/tts-audiobook-tool)** —
Audiobook creation tool supporting a dozen different TTS models including OmniVoice, Qwen3-TTS, VibeVoice, etc.,
focused on high-quality output
- **[LocalText2Voice](https://github.com/estebanstifli/LocalText2Voice)** —
User-friendly Windows desktop pipeline for long-form audiobook and podcast production with OmniVoice support, voice-library management, chapter-aware generation, Whisper-based segment review and retries, multi-voice markup, music/SFX mixing, and MCP automation.
---
### Data Preparation (`docs/data_preparation.md`)
# Data Preparation
OmniVoice trains on a custom WebDataset format where audio data is packed into **tar shards** with paired **JSONL metadata** files. Each tar shard contains hundreds to thousands of samples (as `.npy` audio token arrays), drastically reducing disk I/O during training. The separated jsonl file allows for easier modification of metadata. This document explains the data format in detail and walks through the preparation pipeline.
## 1. Input Format
Prepare a JSONL file where each line is a JSON object:
```jsonl
{"id": "sample_001", "audio_path": "/data/audio/001.wav", "text": "Hello world", "language_id": "en"}
{"id": "sample_002", "audio_path": "/data/audio/002.wav", "text": "你好世界", "language_id": "zh"}
```
Fields:
- `id` — unique sample identifier (used to match samples across shards and label files)
- `audio_path` — absolute path to the audio file (wav/flac/mp3, will be resampled to 24 kHz)
- `text` — transcript text
- `language_id` — (optional) language code, used for multilingual training, can be omitted
## 2. Processing
The tokenization script `extract_audio_tokens.py` converts audio into 8-layer discrete tokens and packs them into WebDataset shards.
```bash
export CUDA_VISIBLE_DEVICES="0,1,2,4" # GPUs used for token extraction
python -m omnivoice.scripts.extract_audio_tokens \
--input_jsonl data.jsonl \
--tar_output_pattern output/audios/shard-%06d.tar \
--jsonl_output_pattern output/txts/shard-%06d.jsonl \
--tokenizer_path eustlb/higgs-audio-v2-tokenizer \
--nj_per_gpu 3 \
--shuffle True
```
What it does:
1. Reads your JSONL manifest
2. Encodes each audio file into discrete tokens using audio tokenizer
3. Packs tokens into WebDataset tar shards with paired jsonl metadata files
4. Generates a `data.lst` manifest file
Alternative: WebDataset Input (if you already have raw-audio tar shards)
Pass the `data.lst` manifest instead of `--input_jsonl`:
```bash
export CUDA_VISIBLE_DEVICES="0,1,2,4" # GPUs used for token extraction
python -m omnivoice.scripts.extract_audio_tokens \
--input_manifest existing_data/data.lst \
--tar_output_pattern output/audios/shard-%06d.tar \
--jsonl_output_pattern output/txts/shard-%06d.jsonl \
--tokenizer_path eustlb/higgs-audio-v2-tokenizer \
--nj_per_gpu 3 \
--shuffle True
```
The existing_data/data.lst is generated with:
```bash
python -m omnivoice.scripts.jsonl_to_webdataset \
--input data.jsonl \
--output data/shards \
--sr 24000 \
--shard-size 1000
```
This resamples audio to the target sample rate and packs FLAC files into tar shards with paired jsonl metadata files.
### Explanation of the script's options:
| Option | Default | Description |
|---|---|---|
| `--input_manifest` | None | Path to input dataset manifest (`data.lst`), mutually exclusive with `--input_jsonl` |
| `--input_jsonl` | None | Path to raw JSONL file, mutually exclusive with `--input_manifest` |
| `--tar_output_pattern` | (required) | Tar shard output pattern, e.g. `output/audios/shard-%06d.tar` |
| `--jsonl_output_pattern` | (required) | JSONL shard output pattern, e.g. `output/txts/shard-%06d.jsonl` |
| `--tokenizer_path` | `eustlb/higgs-audio-v2-tokenizer` | HuggingFace tokenizer path or local path |
| `--nj_per_gpu` | 3 | Worker processes per GPU |
| `--loader_workers` | 24 | DataLoader workers for streaming `IterableDataset` |
| `--shuffle` | True | Shuffle samples before sharding |
| `--shuffle-seed` | 42 | Random seed for shuffling |
| `--samples_per_shard` | 1000 | Max samples per tar shard |
| `--min_num_shards` | 32 | Minimum number of output shards (ensures shard count >= num\_gpu × num\_workers) |
| `--min_length` | 0.0 | Skip audio shorter than this (seconds) |
| `--max_length` | inf | Skip audio longer than this (seconds) |
| `--skip_errors` | False | Continue on processing errors instead of aborting |
| `--num_machines` | 1 | Total number of machines for distributed runs |
| `--machine_index` | 0 | Zero-based machine index for distributed preprocessing |
### Output Structure
Output structure with the following output patterns
```bash
--tar_output_pattern output/audios/shard-%06d.tar \
--jsonl_output_pattern output/txts/shard-%06d.jsonl
```
will be:
```
output/
├── audios/ # WebDataset tar shards (audio tokens)
│ ├── shard-000000.tar # Each tar packs ~1000 samples
│ ├── shard-000001.tar
│ └── ...
├── txts/ # Per-shard companion JSONL labels
│ ├── shard-000000.jsonl # One JSON line per sample in the corresponding tar
│ ├── shard-000001.jsonl
│ └── ...
├── data.lst # Manifest linking tar ↔ jsonl shards
└── errors.jsonl # Samples that failed processing (if any)
```
`data.lst` and `errors.jsonl` are written to the **parent directory** of `audios/` and `txts/`.
### The `data.lst` manifest
Each line in `data.lst` describes one shard:
```
/path/to/shard-000000.tar /path/to/shard-000000.jsonl 1000 3600.500
/path/to/shard-000001.tar /path/to/shard-000001.jsonl 800 2880.200
```
Format: ` `
- Paths are **absolute**
- `.tar` file contains the audio tokens.
- `.jsonl` file contains the metadata in the original provided JSONL file, allows easier access and modification of metadata without decompressing the tar file.
- This manifest is what the training data config references.
### Inside a tar shard
Each `.tar` file packs **many samples** (default 1000 per shard) into a single archive. This is the key advantage of WebDataset: instead of reading thousands of tiny files, the dataloader reads sequentially from a few large tars, drastically reducing disk I/O pressure.
Each sample in the tar is a pair of files with matching keys:
```
shard-000000.tar:
sample_001.npy # Audio tokens: numpy array, shape [8, T], dtype int16
sample_002.npy
...
sample_1000.npy
```
## 3. Data Config for Training
After creating WebDataset shards, write a data config JSON that references them:
```json
{
"train": [
{
"language_id": "en",
"manifest_path": ["data/custom/tokens/train/data.lst"],
"repeat": 1
}
],
"dev": [
{
"language_id": "en",
"manifest_path": ["data/custom/tokens/dev/data.lst"],
"repeat": 1
}
]
}
```
- `manifest_path` — list of `data.lst` files (one per shard directory)
- `repeat` — how many times to repeat this dataset per epoch (useful for balancing languages)
- `language_id` is not used, just for a better data organization.
See [examples/config/](../examples/config/) for ready-to-use data config files.
> See [docs/data_preparation_advanced.md](../docs/data_preparation_advanced.md) for denoising and noise augmentation.
---
### Data Preparation Advanced (`docs/data_preparation_advanced.md`)
# Advanced Data Preparation
The advanced pipeline adds **denoising** and **prompt noise augmentation** on top of the basic tokenization workflow. Each stage is optional.
## Prerequisites
- **Denoising**: Sidon model checkpoints (`feature_extractor_cuda.pt`, `decoder_cuda.pt`) from https://huggingface.co/sarulab-speech/sidon-v0.1/tree/main.
- **Noise augmentation**: noise + RIR tar shards with `data.lst` manifests
## Pipeline Overview
```
Step 1 (optional): Denoise
Raw audio → Sidon denoiser → clean audio
Step 2: Tokenize (with optional noise augmentation)
Clean audio + noise augment on prefix → audio tokenizer → tokens
```
## Denoise
Use the [Sidon](https://github.com/sarulab-speech/Sidon) speech enhancement model to remove background noise from raw audio.
```bash
export CUDA_VISIBLE_DEVICES="0,1,2,3"
python -m omnivoice.scripts.denoise_audio \
--input_jsonl data.jsonl \
--tar_output_pattern data/denoised/audios/shard-%06d.tar \
--jsonl_output_pattern data/denoised/txts/shard-%06d.jsonl \
--feature_extractor_path /path/to/sidon_feature_extractor_cuda.pt \
--decoder_path /path/to/sidon_decoder_cuda.pt \
--target_sample_rate 24000 \
--batch_duration 200.0
```
What it does:
1. Reads your JSONL manifest
2. Runs Sidon denoiser on each audio file
3. Outputs denoised audio as custom WebDataset tar/jsonl shards
4. Generates a `data.lst` manifest in `data/denoised/`
> You can also pass `--input_manifest /path/to/data.lst` if you already have a custom webdataset format dataset.
> The next step would be passing the generated `data.lst` file with `--input_manifest` to `omnivoice.scripts.extract_audio_tokens` for tokens extraction.
### Tokenize with noise augmentation
Adds environmental noise and room reverb to **prompt audio** during tokenization, making the model robust to noisy reference audio at inference time. Note that in our model, we only add noise augmentation for a small proportion of data, making sure the model can also generate good audio with clean reference audio.
You need two additional datasets in WebDataset format:
- **Noise recordings**: environmental noise tar shards with a `data.lst` manifest
- **Room impulse responses (RIR)**: RIR tar shards with a `data.lst` manifest
```bash
export CUDA_VISIBLE_DEVICES="0,1,2,4"
python -m omnivoice.scripts.extract_audio_tokens_add_noise \
--input_jsonl data.jsonl \
--tar_output_pattern data/tokens/shard-%06d.tar \
--jsonl_output_pattern data/txts/shard-%06d.jsonl \
--tokenizer_path eustlb/higgs-audio-v2-tokenizer \
--noise_manifest data/noise_shards/data.lst \
--rir_manifest data/rir_shards/data.lst \
--nj_per_gpu 3
```
> You can also pass `--input_manifest /path/to/data.lst` if you already have a custom webdataset format dataset.
---
### Evaluation (`docs/evaluation.md`)
# Evaluation
Evaluate OmniVoice models with standard TTS metrics: WER (intelligibility), SIM-o (speaker similarity), and UTMOS (naturalness).
## Supported Test Sets
| Test Set | Languages | WER Module | Metrics |
|---|---|---|---|
| **LibriSpeech-PC** | English | HuBERT WER | WER + Speaker Sim + MOS |
| **Seed-TTS (en)** | English | Whisper WER | WER + MOS |
| **Seed-TTS (zh)** | Chinese | Paraformer WER | WER + MOS |
| **FLEURS** | 102 languages | Omnilingual-ASR WER | WER (per-language + macro-avg) |
| **MiniMax Multilingual** | 24 languages | Whisper + Paraformer | WER + MOS |
## Prerequisites
```bash
pip install omnivoice[eval]
# or
uv sync --extra eval
```
## Quick Start
```bash
cd examples
bash run_eval.sh
# run_eval.sh will
# (1) download all required test sets and test models;
# (2) inference and evaluation for each test set.
```
## Metrics Explained
### WER (Word Error Rate)
Measures how intelligible the generated speech is by transcribing it with an ASR model and comparing to the reference text. Lower is better. Note that some languages actually use CER (Character Error Rate).
- **LibriSpeech-PC**: HuBERT-based ASR
- **Seed-TTS**: Whisper (en) or Paraformer (zh)
- **MiniMax**: Whisper for non-Chinese, Paraformer for Chinese
- **FLEURS**: Omnilingual-ASR multilingual model
### Speaker Similarity
Cosine similarity between speaker embeddings (ECAPA-TDNN + WavLM) of the reference and generated audio. Higher is better.
### UTMOS (Predicted MOS)
Neural network that predicts Mean Opinion Score from audio. Higher is better.
---
### Generation Parameters (`docs/generation-parameters.md`)
# Generation Parameters
Parameters can be passed as keyword arguments to `model.generate(...)` or via the `OmniVoiceGenerationConfig` dataclass. See below for the full list and which category each belongs to.
```python
# 1) Direct keyword arguments
audio = model.generate(text="Hello world", num_step=32, guidance_scale=2.0)
# 2) Via OmniVoiceGenerationConfig dataclass
from omnivoice import OmniVoiceGenerationConfig
config = OmniVoiceGenerationConfig(num_step=32, guidance_scale=2.0)
audio = model.generate(text="Hello world", generation_config=config)
```
## Decoding
| Parameter | Type | Default | Description |
|---|---|---|---|
| `num_step` | int | 32 | Number of iterative unmasking steps. Higher values improve quality but slow down generation. Use 16 for faster inference. |
| `denoise` | bool | True | Prepend the `<|denoise|>` token to the input, which signals the model to produce cleaner speech. |
| `guidance_scale` | float | 2.0 | Classifier-free guidance scale.|
| `t_shift` | float | 0.1 | Time-step shift for the noise schedule. Smaller values emphasise earlier steps in decoding. |
## Sampling
| Parameter | Type | Default | Description |
|---|---|---|---|
| `position_temperature` | float | 5.0 | Temperature for mask-position selection. 0 = greedy (deterministic). Higher values increase randomness. |
| `class_temperature` | float | 0.0 | Temperature for token sampling at each step. 0 = greedy (deterministic). Higher values increase randomness. |
| `layer_penalty_factor` | float | 5.0 | Penalty applied to deeper codebook layers, encouraging earlier (lower) layers to unmask first. |
## Duration & Speed
These accept a single value applied to all items, or a per-item list (useful in batch mode):
```python
# Fixed 10-second output
audio = model.generate(text="Hello, this is a test of duration control", duration=10.0)
# Faster speech (1.2x faster than estimated)
audio = model.generate(text="Hello, this is a test of duration control", speed=1.2)
```
| Parameter | Type | Default | Description |
|---|---|---|---|
| `duration` | float or list[float \| None] | None | Fixed output duration in seconds. Overrides `speed` when set. |
| `speed` | float or list[float \| None] | None | Speed factor. Values > 1.0 produce shorter audio (faster); values < 1.0 produce longer audio (slower). Ignored when `duration` is set. Defaults to 1.0 when both are None. |
Priority: `duration` > `speed`.
> **Note:** When using `duration`, the default post-processing step may trim trailing silence, causing the actual output to be slightly shorter than the requested duration. If you need the output duration to **exactly** match the specified value, set `postprocess_output=False` to disable silence removal.
## Pre/Post Processing
| Parameter | Type | Default | Description |
|---|---|---|---|
| `preprocess_prompt` | bool | True | Whether to apply preprocessing to the voice-clone prompt audio (remove long silences in reference audio, add punctuation in the end of reference text). |
| `postprocess_output` | bool | True | Apply post-processing to generated audio (remove long silences). |
| `pad_duration` | float | 0.1 | Silence padding duration per side in seconds. Set to 0 to disable. |
| `fade_duration` | float | 0.1 | Fade-in/out curve duration in seconds. Set to 0 to disable. |
## Long-Form Generation
To support stable long-form speech generation with low VRAM consumption, the text is automatically split into smaller segments when the estimated duration of the generated speech exceeds `audio_chunk_duration`, with each segment producing approximately `audio_chunk_duration` seconds of audio. This approach allows the model to accept arbitrarily long text and generate arbitrarily long speech with near-constant VRAM consumption.
| Parameter | Type | Default | Description |
|---|---|---|---|
| `audio_chunk_duration` | float | 15.0 | Target chunk duration (seconds) when splitting long text. |
| `audio_chunk_threshold` | float | 30.0 | Estimated audio duration (seconds) above which chunking is activated. |
---
### Languages (`docs/languages.md`)
# Supported Languages
OmniVoice supports **646 languages** with a total of **581k hours** of training data.
The table below lists each language with its OmniVoice language ID,
ISO 639-3 code, and training data duration (hours).
| # | Language | OmniVoice ID | ISO 639-3 | Duration (h) |
|--:|----------|:------------:|:---------:|:------------:|
| 1 | Abadi | kbt | kbt | 9.73 |
| 2 | Abkhazian | ab | abk | 57.27 |
| 3 | Abron | abr | abr | 9.22 |
| 4 | Abua | abn | abn | 10.27 |
| 5 | Adamawa Fulfulde | fub | fub | 13.12 |
| 6 | Adyghe | ady | ady | 32.6 |
| 7 | Afade | aal | aal | 10.19 |
| 8 | Afrikaans | af | afr | 4.4 |
| 9 | Agwagwune | yay | yay | 8.26 |
| 10 | Aja (Benin) | ajg | ajg | 5.63 |
| 11 | Akebu | keu | keu | 9.1 |
| 12 | Alago | ala | ala | 11.04 |
| 13 | Albanian | sq | sqi | 8.59 |
| 14 | Algerian Arabic | arq | arq | 9.64 |
| 15 | Algerian Saharan Arabic | aao | aao | 2.02 |
| 16 | Ambo-Pasco Quechua | qva | qva | 9.59 |
| 17 | Ambonese Malay | abs | abs | 10.03 |
| 18 | Amdo Tibetan | adx | adx | 56.94 |
| 19 | Amharic | am | amh | 12.83 |
| 20 | Anaang | anw | anw | 9.65 |
| 21 | Angika | anp | anp | 10.65 |
| 22 | Antankarana Malagasy | xmv | xmv | 17.9 |
| 23 | Aragonese | an | arg | 16.4 |
| 24 | Arbëreshë Albanian | aae | aae | 6.11 |
| 25 | Arequipa-La Unión Quechua | qxu | qxu | 10.12 |
| 26 | Armenian | hy | hye | 42.15 |
| 27 | Ashe | ahs | ahs | 10.62 |
| 28 | Ashéninka Perené | prq | prq | 7.16 |
| 29 | Askopan | eiv | eiv | 10.44 |
| 30 | Assamese | as | asm | 270.85 |
| 31 | Asturian | ast | ast | 8.48 |
| 32 | Atayal | tay | tay | 7.02 |
| 33 | Awak | awo | awo | 10.22 |
| 34 | Ayacucho Quechua | quy | quy | 0.05 |
| 35 | Azerbaijani | az | aze | 9.84 |
| 36 | Baatonum | bba | bba | 10.53 |
| 37 | Bacama | bcy | bcy | 9.94 |
| 38 | Bade | bde | bde | 9.89 |
| 39 | Bafia | ksf | ksf | 16.43 |
| 40 | Bafut | bfd | bfd | 9.03 |
| 41 | Bagirmi Fulfulde | fui | fui | 15.04 |
| 42 | Bago-Kusuntu | bqg | bqg | 8.86 |
| 43 | Baharna Arabic | abv | abv | 10.41 |
| 44 | Bakoko | bkh | bkh | 6.0 |
| 45 | Balanta-Ganja | bjt | bjt | 9.41 |
| 46 | Balti | bft | bft | 16.28 |
| 47 | Bamenyam | bce | bce | 9.9 |
| 48 | Bamun | bax | bax | 10.24 |
| 49 | Bangwinji | bsj | bsj | 10.0 |
| 50 | Banjar | bjn | bjn | 11.68 |
| 51 | Bankon | abb | abb | 11.2 |
| 52 | Baoulé | bci | bci | 10.21 |
| 53 | Bara Malagasy | bhr | bhr | 12.14 |
| 54 | Barok | bjk | bjk | 10.16 |
| 55 | Basa (Cameroon) | bas | bas | 10.66 |
| 56 | Basa (Nigeria) | bzw | bzw | 10.27 |
| 57 | Bashkir | ba | bak | 249.1 |
| 58 | Basque | eu | eus | 479.86 |
| 59 | Batak Mandailing | btm | btm | 11.09 |
| 60 | Batanga | bnm | bnm | 15.01 |
| 61 | Bateri | btv | btv | 9.8 |
| 62 | Bats | bbl | bbl | 11.22 |
| 63 | Bayot | bda | bda | 9.47 |
| 64 | Bebele | beb | beb | 7.52 |
| 65 | Belarusian | be | bel | 1809.43 |
| 66 | Bengali | bn | ben | 271.76 |
| 67 | Betawi | bew | bew | 11.15 |
| 68 | Bhili | bhb | bhb | 9.98 |
| 69 | Bhojpuri | bho | bho | 10.05 |
| 70 | Bilur | bxf | bxf | 10.84 |
| 71 | Bima | bhp | bhp | 10.67 |
| 72 | Bodo | brx | brx | 231.57 |
| 73 | Boghom | bux | bux | 10.48 |
| 74 | Bokyi | bky | bky | 9.85 |
| 75 | Bomu | bmq | bmq | 10.68 |
| 76 | Bondei | bou | bou | 9.98 |
| 77 | Borgu Fulfulde | fue | fue | 20.1 |
| 78 | Bosnian | bs | bos | 690.73 |
| 79 | Brahui | brh | brh | 19.89 |
| 80 | Braj | bra | bra | 10.68 |
| 81 | Breton | br | bre | 25.48 |
| 82 | Buduma | bdm | bdm | 10.17 |
| 83 | Buginese | bug | bug | 11.09 |
| 84 | Bukharic | bhh | bhh | 11.38 |
| 85 | Bulgarian | bg | bul | 2190.76 |
| 86 | Bulu (Cameroon) | bum | bum | 9.06 |
| 87 | Bundeli | bns | bns | 10.88 |
| 88 | Bunun | bnn | bnn | 9.26 |
| 89 | Bura-Pabir | bwr | bwr | 10.4 |
| 90 | Burak | bys | bys | 9.92 |
| 91 | Burmese | my | mya | 12.14 |
| 92 | Burushaski | bsk | bsk | 9.14 |
| 93 | Cacaloxtepec Mixtec | miu | miu | 9.18 |
| 94 | Cajatambo North Lima Quechua | qvl | qvl | 9.95 |
| 95 | Cakfem-Mushere | cky | cky | 8.96 |
| 96 | Cameroon Pidgin | wes | wes | 10.06 |
| 97 | Campidanese Sardinian | sro | sro | 10.16 |
| 98 | Cantonese | yue | yue | 13302.38 |
| 99 | Catalan | ca | cat | 3358.6 |
| 100 | Cebuano | ceb | ceb | 12.17 |
| 101 | Cen | cen | cen | 9.85 |
| 102 | Central Kurdish | ckb | ckb | 137.52 |
| 103 | Central Nahuatl | nhn | nhn | 9.51 |
| 104 | Central Pame | pbs | pbs | 9.69 |
| 105 | Central Pashto | pst | pst | 11.4 |
| 106 | Central Puebla Nahuatl | ncx | ncx | 9.86 |
| 107 | Central Tarahumara | tar | tar | 9.73 |
| 108 | Central Yupik | esu | esu | 2.18 |
| 109 | Central-Eastern Niger Fulfulde | fuq | fuq | 9.28 |
| 110 | Chadian Arabic | shu | shu | 2.29 |
| 111 | Chichewa | ny | nya | 10.8 |
| 112 | Chichicapan Zapotec | zpv | zpv | 9.85 |
| 113 | Chiga | cgg | cgg | 10.84 |
| 114 | Chimalapa Zoque | zoh | zoh | 9.35 |
| 115 | Chimborazo Highland Quichua | qug | qug | 10.12 |
| 116 | Chinese | zh | cmn | 111343.3 |
| 117 | Chiquián Ancash Quechua | qxa | qxa | 9.99 |
| 118 | Chitwania Tharu | the | the | 10.06 |
| 119 | Chokwe | cjk | cjk | 11.01 |
| 120 | Chuvash | cv | chv | 23.96 |
| 121 | Cibak | ckl | ckl | 10.91 |
| 122 | Coastal Konjo | kjc | kjc | 10.18 |
| 123 | Copainalá Zoque | zoc | zoc | 10.07 |
| 124 | Cornish | kw | cor | 12.15 |
| 125 | Corongo Ancash Quechua | qwa | qwa | 9.72 |
| 126 | Croatian | hr | hrv | 2795.31 |
| 127 | Cross River Mbembe | mfn | mfn | 10.03 |
| 128 | Cuyamecalco Mixtec | xtu | xtu | 9.4 |
| 129 | Czech | cs | ces | 148.13 |
| 130 | Dadiya | dbd | dbd | 9.61 |
| 131 | Dagbani | dag | dag | 10.14 |
| 132 | Dameli | dml | dml | 9.18 |
| 133 | Danish | da | dan | 1665.98 |
| 134 | Dargwa | dar | dar | 1.22 |
| 135 | Dazaga | dzg | dzg | 9.96 |
| 136 | Deccan | dcc | dcc | 10.38 |
| 137 | Degema | deg | deg | 11.07 |
| 138 | Dera (Nigeria) | kna | kna | 11.91 |
| 139 | Dghwede | dgh | dgh | 9.95 |
| 140 | Dhatki | mki | mki | 8.83 |
| 141 | Dhivehi | dv | div | 38.61 |
| 142 | Dhofari Arabic | adf | adf | 0.31 |
| 143 | Dijim-Bwilim | cfa | cfa | 10.32 |
| 144 | Dogri | dgo | dgo | 117.04 |
| 145 | Domaaki | dmk | dmk | 6.38 |
| 146 | Dotyali | dty | dty | 10.85 |
| 147 | Duala | dua | dua | 12.13 |
| 148 | Dutch | nl | nld | 2264.13 |
| 149 | Dũya | ldb | ldb | 11.31 |
| 150 | Dyula | dyu | dyu | 0.34 |
| 151 | Eastern Balochi | bgp | bgp | 10.98 |
| 152 | Eastern Bolivian Guaraní | gui | gui | 22.72 |
| 153 | Eastern Egyptian Bedawi Arabic | avl | avl | 1.86 |
| 154 | Eastern Krahn | kqo | kqo | 9.28 |
| 155 | Eastern Mari | mhr | mhr | 272.31 |
| 156 | Eastern Yiddish | ydd | ydd | 18.43 |
| 157 | Ebrié | ebr | ebr | 1.5 |
| 158 | Eggon | ego | ego | 9.95 |
| 159 | Egyptian Arabic | arz | arz | 23.23 |
| 160 | Ejagham | etu | etu | 10.3 |
| 161 | Eleme | elm | elm | 11.27 |
| 162 | Eloyi | afo | afo | 11.21 |
| 163 | Embu | ebu | ebu | 9.81 |
| 164 | English | en | eng | 206061.1 |
| 165 | Erzya | myv | myv | 3.1 |
| 166 | Esan | ish | ish | 10.05 |
| 167 | Esperanto | eo | epo | 1396.64 |
| 168 | Estonian | et | est | 960.37 |
| 169 | Eton (Cameroon) | eto | eto | 7.43 |
| 170 | Ewondo | ewo | ewo | 12.71 |
| 171 | Extremaduran | ext | ext | 13.59 |
| 172 | Fang (Equatorial Guinea) | fan | fan | 3.51 |
| 173 | Fanti | fat | fat | 11.38 |
| 174 | Farefare | gur | gur | 9.24 |
| 175 | Fe'fe' | fmp | fmp | 9.86 |
| 176 | Filipino | fil | fil | 7.71 |
| 177 | Filomena Mata-Coahuitlán Totonac | tlp | tlp | 11.35 |
| 178 | Finnish | fi | fin | 468.62 |
| 179 | Fipa | fip | fip | 10.55 |
| 180 | French | fr | fra | 23675.32 |
| 181 | Fulah | ff | ful | 13.84 |
| 182 | Galician | gl | glg | 208.81 |
| 183 | Gambian Wolof | wof | wof | 9.46 |
| 184 | Ganda | lg | lug | 447.82 |
| 185 | Garhwali | gbm | gbm | 19.14 |
| 186 | Gawar-Bati | gwt | gwt | 12.16 |
| 187 | Gawri | gwc | gwc | 10.83 |
| 188 | Gbagyi | gbr | gbr | 12.12 |
| 189 | Gbari | gby | gby | 12.59 |
| 190 | Geji | gyz | gyz | 10.49 |
| 191 | Gen | gej | gej | 5.39 |
| 192 | Georgian | ka | kat | 156.96 |
| 193 | German | de | deu | 21927.13 |
| 194 | Geser-Gorom | ges | ges | 10.08 |
| 195 | Gheg Albanian | aln | aln | 3.92 |
| 196 | Ghomálá' | bbj | bbj | 7.32 |
| 197 | Gidar | gid | gid | 10.06 |
| 198 | Glavda | glw | glw | 10.51 |
| 199 | Goan Konkani | gom | gom | 9.82 |
| 200 | Goaria | gig | gig | 9.41 |
| 201 | Goemai | ank | ank | 10.0 |
| 202 | Gola | gol | gol | 9.26 |
| 203 | Greek | el | ell | 2412.54 |
| 204 | Guarani | gn | grn | 4.06 |
| 205 | Guduf-Gava | gdf | gdf | 12.21 |
| 206 | Guerrero Amuzgo | amu | amu | 10.1 |
| 207 | Gujarati | gu | guj | 91.18 |
| 208 | Gujari | gju | gju | 8.66 |
| 209 | Gulf Arabic | afb | afb | 98.55 |
| 210 | Gurgula | ggg | ggg | 7.12 |
| 211 | Gusii | guz | guz | 9.5 |
| 212 | Gusilay | gsl | gsl | 10.0 |
| 213 | Gweno | gwe | gwe | 8.87 |
| 214 | Güilá Zapotec | ztu | ztu | 9.17 |
| 215 | Hadothi | hoj | hoj | 10.08 |
| 216 | Hahon | hah | hah | 9.64 |
| 217 | Haitian | ht | hat | 0.04 |
| 218 | Hakha Chin | cnh | cnh | 2.24 |
| 219 | Hakö | hao | hao | 8.56 |
| 220 | Halia | hla | hla | 9.86 |
| 221 | Hausa | ha | hau | 17.75 |
| 222 | Hawaiian | haw | haw | 11.79 |
| 223 | Hazaragi | haz | haz | 9.69 |
| 224 | Hebrew | he | heb | 13.4 |
| 225 | Hemba | hem | hem | 9.53 |
| 226 | Herero | hz | her | 9.59 |
| 227 | Highland Konjo | kjk | kjk | 10.21 |
| 228 | Hijazi Arabic | acw | acw | 22.32 |
| 229 | Hindi | hi | hin | 117.17 |
| 230 | Huarijio | var | var | 9.28 |
| 231 | Huautla Mazatec | mau | mau | 6.39 |
| 232 | Huaxcaleca Nahuatl | nhq | nhq | 5.07 |
| 233 | Huba | hbb | hbb | 10.7 |
| 234 | Huitepec Mixtec | mxs | mxs | 9.64 |
| 235 | Hula | hul | hul | 10.33 |
| 236 | Hungarian | hu | hun | 255.83 |
| 237 | Hunjara-Kaina Ke | hkk | hkk | 8.69 |
| 238 | Hwana | hwo | hwo | 11.23 |
| 239 | Ibibio | ibb | ibb | 7.38 |
| 240 | Icelandic | is | isl | 647.29 |
| 241 | Idakho-Isukha-Tiriki | ida | ida | 9.31 |
| 242 | Idoma | idu | idu | 11.16 |
| 243 | Igbo | ig | ibo | 13.69 |
| 244 | Igo | ahl | ahl | 9.22 |
| 245 | Ikposo | kpo | kpo | 7.83 |
| 246 | Ikwere | ikw | ikw | 10.0 |
| 247 | Imbabura Highland Quichua | qvi | qvi | 11.0 |
| 248 | Indonesian | id | ind | 6327.87 |
| 249 | Indus Kohistani | mvy | mvy | 21.64 |
| 250 | Interlingua (International Auxiliary Language Association) | ia | ina | 13.48 |
| 251 | Inupiaq | ik | ipk | 2.11 |
| 252 | Irish | ga | gle | 21.4 |
| 253 | Iron Ossetic | os | oss | 1.38 |
| 254 | Isekiri | its | its | 11.85 |
| 255 | Isoko | iso | iso | 10.33 |
| 256 | Italian | it | ita | 9402.46 |
| 257 | Ito | itw | itw | 9.19 |
| 258 | Itzá | itz | itz | 7.08 |
| 259 | Ixtayutla Mixtec | vmj | vmj | 10.17 |
| 260 | Izon | ijc | ijc | 9.95 |
| 261 | Jambi Malay | jax | jax | 10.29 |
| 262 | Japanese | ja | jpn | 36914.4 |
| 263 | Jaqaru | jqr | jqr | 9.32 |
| 264 | Jauja Wanca Quechua | qxw | qxw | 11.42 |
| 265 | Jaunsari | jns | jns | 11.25 |
| 266 | Javanese | jv | jav | 11.19 |
| 267 | Jiba | juo | juo | 10.43 |
| 268 | Jju | kaj | kaj | 10.16 |
| 269 | Judeo-Moroccan Arabic | aju | aju | 7.21 |
| 270 | Juxtlahuaca Mixtec | vmc | vmc | 9.43 |
| 271 | Kabardian | kbd | kbd | 108.35 |
| 272 | Kabras | lkb | lkb | 9.99 |
| 273 | Kabuverdianu | kea | kea | 10.51 |
| 274 | Kabyle | kab | kab | 529.52 |
| 275 | Kachi Koli | gjk | gjk | 20.83 |
| 276 | Kairak | ckr | ckr | 10.51 |
| 277 | Kalabari | ijn | ijn | 11.04 |
| 278 | Kalasha | kls | kls | 9.11 |
| 279 | Kalenjin | kln | kln | 40.42 |
| 280 | Kalkoti | xka | xka | 8.0 |
| 281 | Kamba | kam | kam | 14.72 |
| 282 | Kamo | kcq | kcq | 10.49 |
| 283 | Kanauji | bjj | bjj | 11.01 |
| 284 | Kanembu | kbl | kbl | 10.19 |
| 285 | Kannada | kn | kan | 128.06 |
| 286 | Karekare | kai | kai | 10.52 |
| 287 | Kashmiri | ks | kas | 110.42 |
| 288 | Kathoriya Tharu | tkt | tkt | 10.64 |
| 289 | Kati | bsh | bsh | 8.77 |
| 290 | Kazakh | kk | kaz | 1537.29 |
| 291 | Keiyo | eyo | eyo | 9.24 |
| 292 | Khams Tibetan | khg | khg | 6.38 |
| 293 | Khana | ogo | ogo | 10.51 |
| 294 | Khetrani | xhe | xhe | 9.4 |
| 295 | Khmer | km | khm | 7.1 |
| 296 | Khowar | khw | khw | 15.55 |
| 297 | Kinga | zga | zga | 9.5 |
| 298 | Kinnauri | kfk | kfk | 10.32 |
| 299 | Kinyarwanda | rw | kin | 2021.66 |
| 300 | Kirghiz | ky | kir | 46.63 |
| 301 | Kirya-Konzəl | fkk | fkk | 9.98 |
| 302 | Kochila Tharu | thq | thq | 10.28 |
| 303 | Kohistani Shina | plk | plk | 12.75 |
| 304 | Kohumono | bcs | bcs | 10.45 |
| 305 | Kok Borok | trp | trp | 10.74 |
| 306 | Kol (Papua New Guinea) | kol | kol | 9.95 |
| 307 | Kom (Cameroon) | bkm | bkm | 10.76 |
| 308 | Koma | kmy | kmy | 10.28 |
| 309 | Konkani | knn | knn | 112.83 |
| 310 | Konzo | koo | koo | 13.23 |
| 311 | Korean | ko | kor | 8609.28 |
| 312 | Korwa | kfp | kfp | 11.87 |
| 313 | Kota (India) | kfe | kfe | 10.25 |
| 314 | Koti | eko | eko | 8.15 |
| 315 | Kuanua | ksd | ksd | 9.91 |
| 316 | Kuanyama | kj | kua | 9.88 |
| 317 | Kui (India) | uki | uki | 10.77 |
| 318 | Kulung (Nigeria) | bbu | bbu | 10.39 |
| 319 | Kuot | kto | kto | 9.77 |
| 320 | Kushi | kuh | kuh | 10.35 |
| 321 | Kwambi | kwm | kwm | 9.9 |
| 322 | Kwasio | nmg | nmg | 10.39 |
| 323 | Lala-Roba | lla | lla | 10.38 |
| 324 | Lamang | hia | hia | 11.07 |
| 325 | Lao | lo | lao | 7.63 |
| 326 | Larike-Wakasihu | alo | alo | 9.97 |
| 327 | Lasi | lss | lss | 6.53 |
| 328 | Latgalian | ltg | ltg | 27.23 |
| 329 | Latvian | lv | lav | 1441.58 |
| 330 | Levantine Arabic | apc | apc | 15.65 |
| 331 | Liana-Seti | ste | ste | 10.43 |
| 332 | Liberia Kpelle | xpe | xpe | 9.5 |
| 333 | Liberian English | lir | lir | 10.26 |
| 334 | Libyan Arabic | ayl | ayl | 20.13 |
| 335 | Ligurian | lij | lij | 15.97 |
| 336 | Lijili | mgi | mgi | 10.89 |
| 337 | Lingala | ln | lin | 17.99 |
| 338 | Lithuanian | lt | lit | 2629.45 |
| 339 | Loarki | lrk | lrk | 10.5 |
| 340 | Logooli | rag | rag | 9.39 |
| 341 | Logudorese Sardinian | src | src | 10.67 |
| 342 | Loja Highland Quichua | qvj | qvj | 10.59 |
| 343 | Loloda | loa | loa | 9.31 |
| 344 | Longuda | lnu | lnu | 10.46 |
| 345 | Loxicha Zapotec | ztp | ztp | 9.62 |
| 346 | Luba-Lulua | lua | lua | 8.47 |
| 347 | Luo | luo | luo | 36.17 |
| 348 | Lushai | lus | lus | 20.24 |
| 349 | Luxembourgish | lb | ltz | 8.46 |
| 350 | Maasina Fulfulde | ffm | ffm | 10.46 |
| 351 | Maba (Chad) | mde | mde | 9.5 |
| 352 | Macedo-Romanian | rup | rup | 0.02 |
| 353 | Macedonian | mk | mkd | 27.21 |
| 354 | Mada (Cameroon) | mxu | mxu | 12.0 |
| 355 | Mafa | maf | maf | 9.97 |
| 356 | Maithili | mai | mai | 131.37 |
| 357 | Malay | ms | msa | 9.57 |
| 358 | Malayalam | ml | mal | 166.57 |
| 359 | Mali | gcc | gcc | 9.87 |
| 360 | Malinaltepec Me'phaa | tcf | tcf | 9.04 |
| 361 | Maltese | mt | mlt | 630.29 |
| 362 | Mandara | tbf | tbf | 10.01 |
| 363 | Mandjak | mfv | mfv | 9.55 |
| 364 | Manggarai | mqy | mqy | 10.5 |
| 365 | Manipuri | mni | mni | 44.46 |
| 366 | Mansoanka | msw | msw | 9.32 |
| 367 | Manx | gv | glv | 10.07 |
| 368 | Maori | mi | mri | 18.02 |
| 369 | Marathi | mr | mar | 156.71 |
| 370 | Marghi Central | mrt | mrt | 10.36 |
| 371 | Marghi South | mfm | mfm | 10.05 |
| 372 | Maria (India) | mrr | mrr | 11.0 |
| 373 | Marwari (Pakistan) | mve | mve | 9.96 |
| 374 | Masana | mcn | mcn | 10.09 |
| 375 | Masikoro Malagasy | msh | msh | 14.16 |
| 376 | Matsés | mcf | mcf | 9.61 |
| 377 | Mazaltepec Zapotec | zpy | zpy | 9.47 |
| 378 | Mazatlán Mazatec | vmz | vmz | 9.82 |
| 379 | Mazatlán Mixe | mzl | mzl | 10.05 |
| 380 | Mbe | mfo | mfo | 10.24 |
| 381 | Mbo (Cameroon) | mbo | mbo | 9.51 |
| 382 | Mbum | mdd | mdd | 9.82 |
| 383 | Medumba | byv | byv | 10.95 |
| 384 | Mekeo | mek | mek | 9.18 |
| 385 | Meru | mer | mer | 9.89 |
| 386 | Mesopotamian Arabic | acm | acm | 3.78 |
| 387 | Mewari | mtr | mtr | 10.58 |
| 388 | Min Nan Chinese | nan | nan | 17.55 |
| 389 | Mingrelian | xmf | xmf | 11.47 |
| 390 | Mitlatongo Mixtec | vmm | vmm | 9.95 |
| 391 | Miya | mkf | mkf | 10.16 |
| 392 | Mokpwe | bri | bri | 7.53 |
| 393 | Moksha | mdf | mdf | 0.47 |
| 394 | Mom Jango | ver | ver | 10.93 |
| 395 | Mongolian | mn | mon | 269.08 |
| 396 | Moroccan Arabic | ary | ary | 104.67 |
| 397 | Motu | meu | meu | 9.88 |
| 398 | Mpiemo | mcx | mcx | 9.88 |
| 399 | Mpumpong | mgg | mgg | 4.94 |
| 400 | Mundang | mua | mua | 9.2 |
| 401 | Mungaka | mhk | mhk | 7.53 |
| 402 | Musey | mse | mse | 7.21 |
| 403 | Musgu | mug | mug | 4.74 |
| 404 | Musi | mui | mui | 10.52 |
| 405 | Naba | mne | mne | 10.37 |
| 406 | Najdi Arabic | ars | ars | 203.54 |
| 407 | Nalik | nal | nal | 10.33 |
| 408 | Nawdm | nmz | nmz | 6.3 |
| 409 | Ndonga | ng | ndo | 9.08 |
| 410 | Neapolitan | nap | nap | 9.97 |
| 411 | Nepali | npi | npi | 171.5 |
| 412 | Ngamo | nbh | nbh | 10.04 |
| 413 | Ngas | anc | anc | 10.14 |
| 414 | Ngiemboon | nnh | nnh | 16.15 |
| 415 | Ngizim | ngi | ngi | 10.06 |
| 416 | Ngomba | jgo | jgo | 10.15 |
| 417 | Ngombale | nla | nla | 8.79 |
| 418 | Nigerian Fulfulde | fuv | fuv | 9.97 |
| 419 | Nigerian Pidgin | pcm | pcm | 11.04 |
| 420 | Nimadi | noe | noe | 11.12 |
| 421 | Nobiin | fia | fia | 9.96 |
| 422 | North Mesopotamian Arabic | ayp | ayp | 10.92 |
| 423 | North Moluccan Malay | max | max | 9.43 |
| 424 | Northern Betsimisaraka Malagasy | bmm | bmm | 19.12 |
| 425 | Northern Hindko | hno | hno | 20.04 |
| 426 | Northern Kurdish | kmr | kmr | 69.59 |
| 427 | Northern Pame | pmq | pmq | 10.24 |
| 428 | Northern Pashto | pbu | pbu | 11.03 |
| 429 | Northern Uzbek | uzn | uzn | 15.23 |
| 430 | Northwest Gbaya | gya | gya | 8.45 |
| 431 | Norwegian | no | nor | 3849.8 |
| 432 | Norwegian Bokmål | nb | nob | 12.7 |
| 433 | Norwegian Nynorsk | nn | nno | 1.54 |
| 434 | Notsi | ncf | ncf | 9.84 |
| 435 | Nyankpa | yes | yes | 10.26 |
| 436 | Nyungwe | nyu | nyu | 8.98 |
| 437 | Nzanyi | nja | nja | 10.02 |
| 438 | Nüpode Huitoto | hux | hux | 9.04 |
| 439 | Occitan | oc | oci | 16.8 |
| 440 | Od | odk | odk | 20.26 |
| 441 | Odia | ory | ory | 144.81 |
| 442 | Odual | odu | odu | 10.57 |
| 443 | Omani Arabic | acx | acx | 22.03 |
| 444 | Orizaba Nahuatl | nlv | nlv | 11.42 |
| 445 | Orma | orc | orc | 22.01 |
| 446 | Ormuri | oru | oru | 16.74 |
| 447 | Oromo | om | orm | 6.6 |
| 448 | Pahari-Potwari | phr | phr | 24.03 |
| 449 | Paiwan | pwn | pwn | 13.76 |
| 450 | Panjabi | pa | pan | 147.37 |
| 451 | Papuan Malay | pmy | pmy | 10.17 |
| 452 | Parkari Koli | kvx | kvx | 11.04 |
| 453 | Pedi | nso | nso | 12.64 |
| 454 | Pero | pip | pip | 9.85 |
| 455 | Persian | fa | fas | 366.07 |
| 456 | Petats | pex | pex | 10.2 |
| 457 | Phalura | phl | phl | 20.69 |
| 458 | Piemontese | pms | pms | 16.01 |
| 459 | Piya-Kwonci | piy | piy | 10.38 |
| 460 | Plateau Malagasy | plt | plt | 19.39 |
| 461 | Polish | pl | pol | 911.68 |
| 462 | Poqomam | poc | poc | 9.63 |
| 463 | Portuguese | pt | por | 16855.05 |
| 464 | Pulaar | fuc | fuc | 14.77 |
| 465 | Pular | fuf | fuf | 13.77 |
| 466 | Puno Quechua | qxp | qxp | 9.81 |
| 467 | Pushto | ps | pus | 88.62 |
| 468 | Pökoot | pko | pko | 10.4 |
| 469 | Qaqet | byx | byx | 9.79 |
| 470 | Quiotepec Chinantec | chq | chq | 9.76 |
| 471 | Rana Tharu | thr | thr | 9.99 |
| 472 | Rangi | lag | lag | 9.47 |
| 473 | Rapoisi | kyx | kyx | 9.17 |
| 474 | Ratahan | rth | rth | 9.34 |
| 475 | Rayón Zoque | zor | zor | 9.04 |
| 476 | Romanian | ro | ron | 70.23 |
| 477 | Romansh | rm | roh | 9.21 |
| 478 | Rombo | rof | rof | 18.9 |
| 479 | Rotokas | roo | roo | 9.07 |
| 480 | Rukai | dru | dru | 9.26 |
| 481 | Russian | ru | rus | 20338.5 |
| 482 | Sacapulteco | quv | quv | 8.9 |
| 483 | Saidi Arabic | aec | aec | 9.28 |
| 484 | Sakalava Malagasy | skg | skg | 9.02 |
| 485 | Sakizaya | szy | szy | 11.47 |
| 486 | Saleman | sau | sau | 10.53 |
| 487 | Samba Daka | ccg | ccg | 10.11 |
| 488 | Samba Leko | ndi | ndi | 11.27 |
| 489 | San Felipe Otlaltepec Popoloca | pow | pow | 8.84 |
| 490 | San Francisco Del Mar Huave | hue | hue | 9.45 |
| 491 | San Juan Atzingo Popoloca | poe | poe | 10.01 |
| 492 | San Martín Itunyoso Triqui | trq | trq | 8.29 |
| 493 | San Miguel El Grande Mixtec | mig | mig | 9.66 |
| 494 | Sansi | ssi | ssi | 10.47 |
| 495 | Sanskrit | sa | san | 84.44 |
| 496 | Santa Ana de Tusi Pasco Quechua | qxt | qxt | 10.05 |
| 497 | Santa Catarina Albarradas Zapotec | ztn | ztn | 10.02 |
| 498 | Santali | sat | sat | 98.37 |
| 499 | Santiago del Estero Quichua | qus | qus | 9.55 |
| 500 | Saposa | sps | sps | 9.81 |
| 501 | Saraiki | skr | skr | 4.13 |
| 502 | Sardinian | sc | srd | 2.77 |
| 503 | Saya | say | say | 10.02 |
| 504 | Sediq | trv | trv | 7.77 |
| 505 | Serbian | sr | srp | 1855.33 |
| 506 | Seri | sei | sei | 9.81 |
| 507 | Shina | scl | scl | 9.84 |
| 508 | Shona | sn | sna | 9.96 |
| 509 | Siar-Lak | sjr | sjr | 9.87 |
| 510 | Sibe | nco | nco | 9.96 |
| 511 | Sicilian | scn | scn | 13.35 |
| 512 | Sihuas Ancash Quechua | qws | qws | 10.18 |
| 513 | Sikkimese | sip | sip | 10.07 |
| 514 | Sinaugoro | snc | snc | 10.38 |
| 515 | Sindhi | sd | snd | 46.27 |
| 516 | Sindhi Bhil | sbn | sbn | 10.53 |
| 517 | Sinhala | si | sin | 11.98 |
| 518 | Sinicahua Mixtec | xti | xti | 9.5 |
| 519 | Sipacapense | qum | qum | 9.37 |
| 520 | Siwai | siw | siw | 10.47 |
| 521 | Slovak | sk | slk | 2478.46 |
| 522 | Slovenian | sl | slv | 1172.61 |
| 523 | Solos | sol | sol | 9.95 |
| 524 | Somali | so | som | 13.22 |
| 525 | Soninke | snk | snk | 10.04 |
| 526 | South Giziga | giz | giz | 10.03 |
| 527 | South Ucayali Ashéninka | cpy | cpy | 9.15 |
| 528 | Southeastern Nochixtlán Mixtec | mxy | mxy | 9.48 |
| 529 | Southern Betsimisaraka Malagasy | bzc | bzc | 17.45 |
| 530 | Southern Pashto | pbt | pbt | 11.6 |
| 531 | Southern Pastaza Quechua | qup | qup | 11.13 |
| 532 | Soyaltepec Mazatec | vmp | vmp | 10.17 |
| 533 | Spanish | es | spa | 27559.74 |
| 534 | Standard Arabic | arb | arb | 1483.53 |
| 535 | Standard Moroccan Tamazight | zgh | zgh | 1.19 |
| 536 | Sudanese Arabic | apd | apd | 9.93 |
| 537 | Sulka | sua | sua | 10.12 |
| 538 | Svan | sva | sva | 15.11 |
| 539 | Swahili | sw | swa | 418.41 |
| 540 | Swedish | sv | swe | 2453.14 |
| 541 | Tae' | rob | rob | 9.02 |
| 542 | Tahaggart Tamahaq | thv | thv | 4.25 |
| 543 | Taita | dav | dav | 9.12 |
| 544 | Tajik | tg | tgk | 9.23 |
| 545 | Tamil | ta | tam | 423.09 |
| 546 | Tandroy-Mahafaly Malagasy | tdx | tdx | 3.81 |
| 547 | Tangale | tan | tan | 10.14 |
| 548 | Tanosy Malagasy | txy | txy | 12.07 |
| 549 | Tarok | yer | yer | 10.08 |
| 550 | Tatar | tt | tat | 30.03 |
| 551 | Tedaga | tuq | tuq | 10.0 |
| 552 | Telugu | te | tel | 230.21 |
| 553 | Tem | kdh | kdh | 4.07 |
| 554 | Teop | tio | tio | 9.85 |
| 555 | Tepeuxila Cuicatec | cux | cux | 7.83 |
| 556 | Tepinapa Chinantec | cte | cte | 9.54 |
| 557 | Tera | ttr | ttr | 9.89 |
| 558 | Terei | buo | buo | 9.48 |
| 559 | Termanu | twu | twu | 11.45 |
| 560 | Tesaka Malagasy | tkg | tkg | 17.86 |
| 561 | Tetelcingo Nahuatl | nhg | nhg | 8.92 |
| 562 | Teutila Cuicatec | cut | cut | 8.04 |
| 563 | Thai | th | tha | 10499.77 |
| 564 | Tibetan | bo | bod | 82.27 |
| 565 | Tidaá Mixtec | mtx | mtx | 9.09 |
| 566 | Tidore | tvo | tvo | 10.31 |
| 567 | Tigak | tgc | tgc | 9.71 |
| 568 | Tigre | tig | tig | 7.49 |
| 569 | Tigrinya | ti | tir | 0.08 |
| 570 | Tilquiapan Zapotec | zts | zts | 9.33 |
| 571 | Tinputz | tpz | tpz | 9.33 |
| 572 | Tlacoapa Me'phaa | tpl | tpl | 9.28 |
| 573 | Tlacoatzintepec Chinantec | ctl | ctl | 10.04 |
| 574 | Tlingit | tli | tli | 0.41 |
| 575 | Toki Pona | tok | tok | 13.51 |
| 576 | Tomoip | tqp | tqp | 10.1 |
| 577 | Tondano | tdn | tdn | 9.14 |
| 578 | Tonsea | txs | txs | 9.32 |
| 579 | Tooro | ttj | ttj | 10.31 |
| 580 | Torau | ttu | ttu | 9.87 |
| 581 | Torwali | trw | trw | 14.98 |
| 582 | Tsimihety Malagasy | xmw | xmw | 11.53 |
| 583 | Tsotso | lto | lto | 9.77 |
| 584 | Tswana | tn | tsn | 4.24 |
| 585 | Tugen | tuy | tuy | 8.79 |
| 586 | Tuki | bag | bag | 10.97 |
| 587 | Tula | tul | tul | 9.79 |
| 588 | Tulu | tcy | tcy | 11.72 |
| 589 | Tunen | tvu | tvu | 9.85 |
| 590 | Tungag | lcm | lcm | 9.77 |
| 591 | Tunisian Arabic | aeb | aeb | 21.63 |
| 592 | Tupuri | tui | tui | 9.26 |
| 593 | Turkana | tuv | tuv | 10.17 |
| 594 | Turkish | tr | tur | 125.36 |
| 595 | Turkmen | tk | tuk | 2.86 |
| 596 | Tututepec Mixtec | mtu | mtu | 10.13 |
| 597 | Twi | tw | twi | 0.25 |
| 598 | Ubaghara | byc | byc | 11.11 |
| 599 | Uighur | ug | uig | 428.77 |
| 600 | Ukrainian | uk | ukr | 1851.97 |
| 601 | Umbundu | umb | umb | 10.59 |
| 602 | Upper Sorbian | hsb | hsb | 2.71 |
| 603 | Urdu | ur | urd | 211.27 |
| 604 | Ushojo | ush | ush | 6.36 |
| 605 | Uzbek | uz | uzb | 115.28 |
| 606 | Vai | vai | vai | 8.76 |
| 607 | Vietnamese | vi | vie | 8481.98 |
| 608 | Votic | vot | vot | 0.1 |
| 609 | Võro | vro | vro | 15.66 |
| 610 | Waci Gbe | wci | wci | 8.02 |
| 611 | Wadiyara Koli | kxp | kxp | 20.0 |
| 612 | Waja | wja | wja | 10.22 |
| 613 | Wakhi | wbl | wbl | 11.67 |
| 614 | Wanga | lwg | lwg | 9.36 |
| 615 | Wapan | juk | juk | 10.22 |
| 616 | Warji | wji | wji | 11.39 |
| 617 | Welsh | cy | cym | 131.21 |
| 618 | Wemale | weo | weo | 9.09 |
| 619 | Western Frisian | fy | fry | 70.41 |
| 620 | Western Highland Purepecha | pua | pua | 10.17 |
| 621 | Western Juxtlahuaca Mixtec | jmx | jmx | 10.01 |
| 622 | Western Maninkakan | mlq | mlq | 9.83 |
| 623 | Western Mari | mrj | mrj | 32.26 |
| 624 | Western Niger Fulfulde | fuh | fuh | 9.69 |
| 625 | Western Panjabi | pnb | pnb | 10.0 |
| 626 | Wolof | wo | wol | 8.71 |
| 627 | Wuzlam | udl | udl | 9.23 |
| 628 | Xanaguía Zapotec | ztg | ztg | 9.86 |
| 629 | Xhosa | xh | xho | 13.35 |
| 630 | Yace | ekr | ekr | 10.76 |
| 631 | Yakut | sah | sah | 16.08 |
| 632 | Yalahatan | jal | jal | 11.18 |
| 633 | Yanahuanca Pasco Quechua | qur | qur | 9.95 |
| 634 | Yangben | yav | yav | 8.7 |
| 635 | Yaqui | yaq | yaq | 9.93 |
| 636 | Yauyos Quechua | qux | qux | 9.35 |
| 637 | Yekhee | ets | ets | 10.11 |
| 638 | Yiddish | yi | yid | 1.81 |
| 639 | Yidgha | ydg | ydg | 9.89 |
| 640 | Yoruba | yo | yor | 15.66 |
| 641 | Yutanduchi Mixtec | mab | mab | 9.26 |
| 642 | Zacatlán-Ahuacatlán-Tepetzintla Nahuatl | nhi | nhi | 0.05 |
| 643 | Zarma | dje | dje | 10.72 |
| 644 | Zaza | zza | zza | 1.52 |
| 645 | Zulu | zu | zul | 14.83 |
| 646 | Ömie | aom | aom | 8.19 |
*646 languages, 581k hours total.*
Data source: [docs/lang_id_name_map.tsv](lang_id_name_map.tsv)
---
### Lora Finetuning (`docs/lora_finetuning.md`)
# LoRA Finetuning
In addition to full finetuning (see [training.md](training.md)), OmniVoice supports parameter-efficient finetuning via [LoRA](https://arxiv.org/abs/2106.09685) (Low-Rank Adaptation), using [PEFT](https://github.com/huggingface/peft). LoRA freezes the pretrained LLM backbone and trains small low-rank adapter matrices instead — typically **3-5% of parameters trainable** — which means:
- Much lower optimizer memory (Adam keeps 2 extra tensors per trainable parameter).
- Much smaller adapter checkpoints to ship (tens of MB vs. the full model size).
- The original pretrained weights stay untouched on disk; the adapter can be dropped to recover the original model exactly.
LoRA is **opt-in** via config — with `use_lora: false` (the default), training behaves exactly as documented in [training.md](training.md).
## Installation
```bash
pip install "omnivoice[lora]"
# or, from a source checkout:
pip install peft
```
## What Gets Adapted
| Config field | Applies to | Why |
|---|---|---|
| `lora_target_modules` | LLM attention/MLP projections (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` by default) | This is where the pretrained language/text-conditioning knowledge lives — the standard set of layers LoRA adapts for transformer LLMs. |
| `lora_modules_to_save` | `audio_embeddings`, `audio_heads` by default | These are OmniVoice-specific audio token I/O layers, not part of the pretrained LLM. There's no pretrained knowledge in them to preserve via low-rank adapters, so they're kept **fully trainable** instead and saved alongside the adapter. |
Everything else (the LLM's other weights, layernorms, etc.) stays frozen, exactly as loaded from the base checkpoint.
## Configuration
Start from [examples/config/train_config_finetune_lora.json](../examples/config/train_config_finetune_lora.json) — identical to [train_config_finetune_sdpa.json](../examples/config/train_config_finetune_sdpa.json) plus the LoRA fields:
```json
{
"init_from_checkpoint": "k2-fsa/OmniVoice",
"use_lora": true,
"lora_r": 16,
"lora_alpha": 32,
"lora_dropout": 0.05,
"lora_bias": "none",
"lora_target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
"lora_modules_to_save": ["audio_embeddings", "audio_heads"],
"learning_rate": 1e-4,
"attn_implementation": "sdpa"
}
```
| Field | Description | Default |
|---|---|---|
| `use_lora` | Enable LoRA finetuning instead of full finetuning | `false` |
| `lora_r` | LoRA rank — higher = more capacity, more trainable params | `16` |
| `lora_alpha` | LoRA scaling factor (effective scale is `lora_alpha / lora_r`) | `32` |
| `lora_dropout` | Dropout applied inside the LoRA adapter path | `0.05` |
| `lora_bias` | Which biases to train: `"none"`, `"all"`, or `"lora_only"` | `"none"` |
| `lora_target_modules` | LLM submodule names to attach LoRA adapters to | attention + MLP projections |
| `lora_modules_to_save` | Non-LLM modules trained in full instead of via LoRA | `audio_embeddings`, `audio_heads` |
A higher `learning_rate` than full finetuning (`1e-4` vs. `5e-5`) is typically used since the adapter matrices are randomly initialized and trained from scratch, rather than finetuning already-converged pretrained weights — a higher LR converges faster without the instability risk full finetuning has at that LR.
## Launching Training
```bash
accelerate launch \
--gpu_ids "0,1" \
--num_processes 2 \
-m omnivoice.cli.train \
--train_config examples/config/train_config_finetune_lora.json \
--data_config examples/config/data_config_finetune.json \
--output_dir exp/omnivoice_finetune_lora
```
Or run the full example pipeline (tokenization + training):
```bash
bash examples/run_finetune_lora.sh
```
On startup you'll see a line like:
```text
trainable params: 19,316,736 || all params: 631,894,016 || trainable%: 3.0570
```
confirming only the adapters plus `audio_embeddings`/`audio_heads` are being trained.
## How Many Checkpoints Get Saved
Same three fields as full finetuning govern this:
| Field | Effect |
|---|---|
| `steps` | Total training steps |
| `save_steps` | A checkpoint is saved every `save_steps` steps |
| `keep_last_n_checkpoints` | `-1` keeps every checkpoint; a positive `N` rotates old ones out, keeping only the most recent `N` |
Number of checkpoints saved = `steps / save_steps` (when `steps` is a multiple of `save_steps`), capped at `keep_last_n_checkpoints` if positive.
## Resuming Training
Identical mechanism to full finetuning — set `resume_from_checkpoint`:
```json
{
"resume_from_checkpoint": "exp/omnivoice_finetune_lora/checkpoint-500"
}
```
Each run rebuilds the same LoRA-wrapped model architecture from `init_from_checkpoint` plus the `lora_*` config fields, then restores optimizer/scheduler/RNG state and adapter weights from the checkpoint on top of it. Keep `init_from_checkpoint` and the `lora_*` fields unchanged across a resume — a mismatch will surface as a shape/key error when `accelerate` restores state.
## Adapter Inference
Apply a trained adapter on top of the base model at inference time — merged into the model in-memory (not written to disk), so generation uses the regular `OmniVoice.generate()` path unmodified:
```bash
omnivoice-infer \
--model k2-fsa/OmniVoice \
--lora_adapter exp/omnivoice_finetune_lora/checkpoint-500 \
--text "Hello, this is a text for text-to-speech." \
--ref_audio ref.wav --ref_text "Reference transcript." \
--output out.wav
```
`--lora_adapter` accepts any directory containing `adapter_config.json` + `adapter_model.safetensors` — i.e. any LoRA training checkpoint directory.
## Adapter Merging
To deploy with **no PEFT dependency at inference time**, merge the adapter into the base model once and save the result as a normal, standalone OmniVoice checkpoint:
```bash
omnivoice-merge-lora \
--base_model k2-fsa/OmniVoice \
--lora_adapter exp/omnivoice_finetune_lora/checkpoint-500 \
--output_dir exp/omnivoice_finetune_lora/merged
```
This loads the base model, merges the LoRA deltas into its weights, and writes a complete, self-contained OmniVoice directory (model weights, tokenizer, and audio tokenizer).
## Using the Merged Model
The merged directory is a normal OmniVoice checkpoint — use it exactly like any other:
```bash
omnivoice-infer \
--model exp/omnivoice_finetune_lora/merged \
--text "Hello, this is a text for text-to-speech." \
--instruct "male, British accent" \
--output out.wav
```
```python
from omnivoice.models.omnivoice import OmniVoice
model = OmniVoice.from_pretrained("exp/omnivoice_finetune_lora/merged", device_map="cuda:0")
audios = model.generate(text="Hello world", language="en")
```
The merged model is saved in whatever dtype the base model was loaded in when merging (`float32` by default, since `omnivoice-merge-lora` loads the base model with `dtype=torch.float32`). Pass `--dtype fp32` / `dtype=torch.float32` at inference to match exactly, or `fp16`/`bf16` for less VRAM and faster inference — `from_pretrained` casts down automatically, and quality differences are usually negligible.
## Checkpoint Structure
A LoRA training checkpoint (`exp/.../checkpoint-/`) contains two independent sets of files:
```text
checkpoint-500/
├── adapter_config.json # PEFT LoRA config (small)
├── adapter_model.safetensors # LoRA weights + audio_embeddings/audio_heads (small)
├── tokenizer.json # text tokenizer, saved for adapter/merge use
├── tokenizer_config.json
├── model.safetensors # accelerate's full training state (base + adapter weights)
├── optimizer.bin # optimizer state, for resuming
├── scheduler.bin # LR scheduler state, for resuming
└── random_states_0.pkl # RNG state, for resuming
```
- **For adapter inference / merging**, only `adapter_config.json` + `adapter_model.safetensors` (+ tokenizer) are used.
- **For resuming training**, `accelerate`'s own state files (`model.safetensors`, `optimizer.bin`, `scheduler.bin`, `random_states_*.pkl`) are used — these include the full (frozen + trainable) model state, since `accelerate`'s checkpointing does not distinguish frozen from trainable parameters. This makes `checkpoint-*/` directories larger than the adapter alone, but keeps resuming exactly as reliable as full finetuning's. If you only need the adapter for deployment, you can delete everything except `adapter_config.json`/`adapter_model.safetensors`/the tokenizer files once training is done.
## Troubleshooting
**`ModuleNotFoundError: No module named 'peft'`**
Run `pip install peft` (or `pip install "omnivoice[lora]"`).
**Resume fails with a shape/key mismatch error**
`resume_from_checkpoint` must be resumed with the *same* `init_from_checkpoint`, `lora_r`, `lora_alpha`, `lora_target_modules`, and `lora_modules_to_save` as the run that produced the checkpoint — these determine the model architecture `accelerate`'s state restore expects to find.
**`omnivoice-merge-lora` / `--lora_adapter` can't find the adapter**
Point it at a checkpoint *directory*, not a specific file — it must contain `adapter_config.json`.
**Training loss looks the same as a frozen model / doesn't move**
Check the `trainable params: ...` line printed at startup. If it's `0` or doesn't include LoRA layers, verify `use_lora: true` is set in the config passed to `--train_config`, and that `lora_target_modules` names match submodules in your `llm_name_or_path`'s architecture.
**Checkpoint saves or merging are unexpectedly slow**
Each checkpoint writes a full base-model-sized file (accelerate's training state — see [Checkpoint Structure](#checkpoint-structure)), and merging both reads and writes one. If disk space is tight or I/O is contended, these can take much longer than expected; check available disk space first.
---
### Tips (`docs/tips.md`)
# Tips & Notes
- **Combination of `ref_audio` and `instruct`**:
When both `ref_audio` and `instruct` are provided and they **conflict**, the model will most likely follow the style of the reference audio. When the two are **consistent**, `instruct` can improve cloning stability for the attributes it describes. A typical example is **Chinese dialect cloning**: provide both dialect reference audio and a matching dialect instruct (e.g., `ref_audio="sichuan.wav", instruct="四川话"`) for more stable dialect output.
- **Short Audio Generation**:
The model may not reliably generate short audio clips (e.g., 1–2 seconds) without reference audio. If you need to generate short clips, provide reference audio to the model.
- **Min Nan Chinese (Hokkien) Input Format**:
Min Nan Chinese (闽南语, also known as Hokkien) can only be synthesized using [Tai-lo romanization](https://en.wikipedia.org/wiki/T%C3%A2i-l%C3%B4) as input; Chinese characters are not supported for Min Nan Chinese in the current model version.
---
### Training (`docs/training.md`)
# Training
## Training Config
All training is controlled by a JSON training config file and a JSON data config file.
See [examples/config/](../examples/config/) for ready-to-use configs.
Training config file on Emilia is: [examples/config/train_config_emilia.json](../examples/config/train_config_emilia.json)
Data config file for Emilia is: [examples/config/data_config_emilia.json](../examples/config/data_config_emilia.json)
Key fields in training config file:
| Field | Description | Default |
|---|---|---|
| `llm_name_or_path` | local LLM path or huggingface id | Qwen/Qwen3-0.6B |
| `steps` | Total training steps | 300,000 |
| `learning_rate` | Peak learning rate | 1e-4 |
| `batch_tokens` | Tokens per batch on each GPU | 8192 |
| `attn_implementation` | Attention backend: `"flex_attention"` or `"sdpa"` | `"flex_attention"` |
`output_dir` and `data_config` are passed via command line (see below).
## Attention Implementation
By default, training uses `flex_attention`, which requires PyTorch ≥ 2.5 and a compatible GPU (e.g. NVIDIA Ampere or newer). If your environment does not support `flex_attention`, set `attn_implementation` to `"sdpa"` in your training config. See [examples/config/train_config_finetune_sdpa.json](../examples/config/train_config_finetune_sdpa.json) for a ready-to-use SDPA config:
```json
{
"attn_implementation": "sdpa",
"max_sample_tokens": 2000,
"min_sample_tokens": 50,
"max_batch_size": 64
}
```
`"sdpa"` uses PyTorch's built-in scaled dot-product attention and works on a wider range of hardware.
The following fields only apply when `attn_implementation != "flex_attention"`:
| Field | Description | Default |
|---|---|---|
| `max_sample_tokens` | Maximum token length per sample; longer samples are dropped | 2000 |
| `min_sample_tokens` | Minimum token length per sample; shorter samples are dropped | 50 |
| `max_batch_size` | Cap on the number of samples per batch | 64 |
`batch_tokens` remains the primary control for memory usage — it sets the total token budget per batch. `max_batch_size` is a safety guard to prevent a batch of many short samples from creating an unusually large batch dimension.
### Batching strategy
The two backends use **different batching strategies**, which are selected automatically:
| Backend | Batching strategy | Batch shape | Notes |
|---|---|---|---|
| `flex_attention` | Sequence packing | `[1, C, batch_tokens]` | Multiple samples concatenated into one long sequence; document boundaries tracked via `document_ids` |
| `sdpa` | Length-grouped padding | `[B, C, max_len]` | Samples with similar token lengths are grouped into the same batch and padded to the local maximum length |
**Why different strategies?**
- With `flex_attention`, sequence packing is memory-efficient because a compact `BlockMask` (not a dense matrix) describes which tokens can attend to each other across document boundaries.
- With `sdpa`, length-grouped padding is used instead: samples of similar token lengths are batched together and padded to the local maximum, so a lightweight `[B, 1, max_len, max_len]` boolean attention mask suffices with low overhead and minimal wasted padding.
## Launching Training
```bash
accelerate launch \
--gpu_ids "0,1,2,3,4,5,6,7" \
--num_processes 8 \
-m omnivoice.cli.train \
--train_config config/train_config_emilia.json \
--data_config config/data_config_emilia.json \
--output_dir exp/omnivoice_emilia
```
## Resuming Training
Set `resume_from_checkpoint` in your training config to resume from an existing checkpoint:
```json
{
"resume_from_checkpoint": "exp/omnivoice/checkpoint-100000"
}
```
## Initializing from a Pretrained Model
To start training from a pretrained OmniVoice checkpoint (for fine-tuning):
```json
{
"init_from_checkpoint": "exp/omnivoice/checkpoint-100000"
}
```
## LoRA Finetuning
For parameter-efficient finetuning, set `"use_lora": true` in your training config to train low-rank adapters instead of the full model. See [lora_finetuning.md](lora_finetuning.md) for configuration, adapter inference, and merging back into a standalone checkpoint.
## Monitoring
Training logs to TensorBoard:
```bash
tensorboard --logdir exp/omnivoice_emilia/tensorboard
```
---
### Voice Design (`docs/voice-design.md`)
# Voice Design
Voice Design mode lets you describe the desired speaker through speaker attributes (`instruct` parameter) — no reference audio needed. The model
generates a matching voice on the fly.
## Quick Example
```python
import torch
from omnivoice import OmniVoice
model = OmniVoice.from_pretrained(
"k2-fsa/OmniVoice",
device_map="cuda:0",
dtype=torch.float16
)
audio = model.generate(
text="This is a test for voice design.",
instruct="female, young adult, high pitch, british accent",
)
```
## How It Works
The `instruct` parameter accepts a comma-separated string of speaker attributes.
Each attribute belongs to a **category** (gender, age, pitch, style, accent,
or dialect). Within a category, only one attribute may be selected at a time.
Attributes from different categories can be freely combined.
The model auto-detects the language of the instruct text and normalises it
internally — you can write in English, Chinese, or a mix of both.
## Supported Attributes
### Gender
| English | Chinese |
|---------|---------|
| male | 男 |
| female | 女 |
### Age
| English | Chinese |
|---------|---------|
| child | 儿童 |
| teenager | 少年 |
| young adult | 青年 |
| middle-aged | 中年 |
| elderly | 老年 |
### Pitch
| English | Chinese |
|---------|---------|
| very low pitch | 极低音调 |
| low pitch | 低音调 |
| moderate pitch | 中音调 |
| high pitch | 高音调 |
| very high pitch | 极高音调 |
### Style
| English | Chinese |
|---------|---------|
| whisper | 耳语 |
### English Accent
Only effective when the synthesis text is in English.
| Accent |
|--------|
| american accent |
| british accent |
| australian accent |
| canadian accent |
| indian accent |
| chinese accent |
| korean accent |
| japanese accent |
| portuguese accent |
| russian accent |
### Chinese Dialect
Only effective when the synthesis text is in Chinese.
| Dialect |
|---------|
| 河南话 |
| 陕西话 |
| 四川话 |
| 贵州话 |
| 云南话 |
| 桂林话 |
| 济南话 |
| 石家庄话 |
| 甘肃话 |
| 宁夏话 |
| 青岛话 |
| 东北话 |
## Writing Instruct Strings
Separate attributes with commas (half-width `,` for English, full-width `，`
for Chinese — the model auto-fixes mismatches).
```
# English
"female, young adult, high pitch, british accent"
# Chinese
"女，青年，高音调，四川话"
# Mixed (auto-normalised)
"female, young adult, 四川话"
```
### Tips
- **Combine freely** across categories: `"male, elderly, low pitch, whisper"`.
- **Leave it to the model**: omit attributes you don't care about — the model
fills in the rest. For example `"female"` alone is valid.
- **Case-insensitive**: `"Male"`, `"MALE"`, and `"male"` are all accepted, the code will normalize them to lower case.
- **Accent vs Dialect**: English accents are only applied to English speech, Chinese dialects are only applied to Chinese speech.
- **Attribute combinations**: Due to training data limitations, some attribute combinations may not work well — the model may ignore certain attributes in a combination. If the output doesn't match your expectation, try simplifying the instruct string.
---
--- METRICS ---
- Files Extracted: 12
- Estimated Token Budget: ~21189 tokens
- Canonical Reference: https://codewiki.google/github.com/k2-fsa/OmniVoice