OmniVoice Documentation

File: README.md Copy Chapter

OmniVoice 🌍

OmniVoice is a state-of-the-art massively multilingual zero-shot text-to-speech (TTS) model supporting over 600 languages. Built on a novel diffusion language model-style architecture, it generates high-quality speech with superior inference speed, supporting voice cloning and voice design.

Contents: Key Features | Installation | Quick Start | Python API | Command-Line Tools | Training & Evaluation | Discussion | Citation

Key Features Copy Chapter

• 600+ Languages Supported: The broadest language coverage among zero-shot TTS models (full list).
• Voice Cloning: State-of-the-art voice cloning quality.
• Voice Design: Control voices via assigned speaker attributes (gender, age, pitch, dialect/accent, whisper, etc.).
• Fine-grained Control: Non-verbal symbols (e.g., [laughter]) and pronunciation correction via pinyin or phonemes.
• Fast Inference: RTF as low as 0.025 (40x faster than real-time).
• Diffusion Language Model-style Architecture: A clean, streamlined, and scalable design that delivers both quality and speed.

Installation Copy Chapter

Choose one of the following methods: pip or uv.

pip Copy Section

We recommend using a fresh virtual environment (e.g., conda, venv, etc.) to avoid conflicts.

Step 1: Install PyTorch

NVIDIA GPU

# Install pytorch with your CUDA version, e.g.
pip install torch==2.8.0+cu128 torchaudio==2.8.0+cu128 --extra-index-url https://download.pytorch.org/whl/cu128

See PyTorch official site for other versions installation.

Apple Silicon

pip install torch==2.8.0 torchaudio==2.8.0

Intel Arc GPU (XPU)

Intel Arc GPUs (Alchemist and Battlemage architectures) are supported via PyTorch's XPU backend.

1. Install the Intel GPU drivers for your OS.

2. Install PyTorch with XPU support from Intel's wheel index:

pip install torch torchaudio --index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

See Intel's PyTorch XPU guide for version-specific instructions.

3. Verify the backend is working:

python -c "import torch; print(torch.xpu.is_available(), torch.xpu.device_count())"

Notes:

• flash_attn is not available on XPU; the model automatically falls back to SDPA.
• Training with packed sequences (flex_attention) has partial XPU support; single-GPU SDPA training should work.
• Tested on Arc A310 (Alchemist, 4 GB) and Arc Pro B50 (Battlemage, 16 GB).

Step 2: Install OmniVoice (choose one)

# From PyPI (stable release)
pip install omnivoice

# From the latest source on GitHub (no need to clone)
pip install git+https://github.com/k2-fsa/OmniVoice.git

# For development (clone first, editable install)
git clone https://github.com/k2-fsa/OmniVoice.git
cd OmniVoice
pip install -e .

uv Copy Section

Clone the repository and sync dependencies:

git clone https://github.com/k2-fsa/OmniVoice.git
cd OmniVoice
uv sync

Tip: Can use mirror with uv sync --default-index "https://mirrors.aliyun.com/pypi/simple"

Quick Start Copy Chapter

Try OmniVoice without coding:

• Launch the local web UI: omnivoice-demo --ip 0.0.0.0 --port 8001

• Or try it directly on HuggingFace Space

• Or run it in Google Colab:

If you have trouble connecting to HuggingFace when downloading the pre-trained models, set export HF_ENDPOINT="https://hf-mirror.com" before running.

For full usage, see the Python API and Command-Line Tools sections below.

Python API Copy Chapter

OmniVoice supports three generation modes. All features in this section are also available via command-line tools.

Voice Cloning Copy Section

Clone a voice from a short reference audio. Provide ref_audio and ref_text:

from omnivoice import OmniVoice
import soundfile as sf
import torch

model = OmniVoice.from_pretrained(
    "k2-fsa/OmniVoice",
    device_map="cuda:0",
    dtype=torch.float16
)
# Apple Silicon users: use device_map="mps" instead
# Intel Arc GPU users: use device_map="xpu" instead

audio = model.generate(
    text="Hello, this is a test of zero-shot voice cloning.",
    ref_audio="ref.wav",
    ref_text="Transcription of the reference audio.",
) # audio is a list of `np.ndarray` with shape (T,) at 24 kHz.

# If you don't want to input `ref_text` manually, you can directly omit the `ref_text`.
# The model will use Whisper ASR to auto-transcribe it. To use a local copy (or
# a different Whisper model), pass `asr_model_name="..."` to `from_pretrained`.
# To control which device Whisper is loaded on (e.g. another GPU in multi-GPU
# setups, or the CPU), pass `asr_device="cuda:1"` (or `"cpu"`).

sf.write("out.wav", audio[0], 24000)

Reusing a cloned voice across sessions

Encode the reference audio once, save the resulting prompt, and skip the
audio loading / auto-transcription steps in later sessions:

prompt = model.create_voice_clone_prompt(
    ref_audio="ref.wav", ref_text="Transcription of the reference audio."
)
prompt.save("my_voice.pt")

# Later, in a new session:
from omnivoice import VoiceClonePrompt

prompt = VoiceClonePrompt.load("my_voice.pt")
audio = model.generate(text="Hello again!", voice_clone_prompt=prompt)

Tips

• Use a 3–10 seconds reference audio clip. Longer audio slows down inference and may degrade cloning quality.

• For standard pronunciation, use a reference audio in the same language as the target speech. In cross-lingual voice cloning (i.e., the reference audio and target speech are in different languages), the generated speech will carry an accent from the reference audio's language.

• For better results with Arabic numerals, normalize them to words first (e.g., "123" → "one hundred twenty-three"). You can pass normalize_text=True to generate() to do this automatically (opt-in; install the extra with pip install "omnivoice[tn]", which pulls in WeTextProcessing):

  python Copy

  # "I have 2345 apples." is read correctly instead of digit-by-digit.
  audio = model.generate(text="I have 2345 apples.", normalize_text=True)

  Chinese and English use WeTextProcessing; other languages fall back to num2words for integers. Inline control syntax ([laughter], [B EY1 S], pinyin tone markers) is preserved. On macOS (Apple Silicon), pynini has no wheel — install it via conda install -c conda-forge pynini first.

For more tips, see docs/tips.md.

Voice Design Copy Section

Describe the desired voice with speaker attributes — no reference audio needed.
Supported attributes: gender (male/female), age (child to elderly),
pitch (very low to very high), style (whisper), English accent
(American, British, etc.), and Chinese dialect (四川话, 陕西话, etc.).
Attributes are comma-separated and freely combinable across categories.

audio = model.generate(
    text="Hello, this is a test of zero-shot voice design.",
    instruct="female, low pitch, british accent",
)

Note: The model is primarily trained on the voice cloning task, so voice cloning is the most stable mode. Voice design is trained on Chinese and English data only. It can generalize to other languages, but may produce unstable results for some low-resource languages or edge cases.

See docs/voice-design.md for the full attribute
reference, Chinese equivalents, and usage tips.

Auto Voice Copy Section

Let the model choose a voice automatically:

audio = model.generate(text="This is a sentence without any voice prompt.")

Generation Parameters Copy Section

All above three modes share the same model.generate() API. You can further control the generation behavior via keyword arguments:

audio = model.generate(
    text="...",
    num_step=32,  # diffusion steps (or 16 for faster inference)
    speed=1.0,     # speed factor (>1.0 faster, <1.0 slower)
    duration=10.0, # fixed output duration in seconds (overrides speed)
    # ... more options
)

See more detailed control in docs/generation-parameters.md.

Non-Verbal & Pronunciation Control Copy Section

OmniVoice supports inline non-verbal symbols and pronunciation correction within the input text.

Non-verbal symbols: Insert tags like [laughter] directly in the text to add expressive non-verbal sounds.

audio = model.generate(text="[laughter] You really got me. I didn't see that coming at all.")

Supported tags: [laughter], [sigh], [confirmation-en], [question-en], [question-ah], [question-oh], [question-ei], [question-yi], [surprise-ah], [surprise-oh], [surprise-wa], [surprise-yo], [dissatisfaction-hnn].

Pronunciation control (Chinese): Use pinyin with tone numbers to correct specific character pronunciations.

audio = model.generate(text="这批货物打ZHE2出售后他严重SHE2本了，再也经不起ZHE1腾了。")

Pronunciation control (English): Use CMU pronunciation dictionary (uppercase, in brackets) to override default English pronunciations.

audio = model.generate(text="He plays the [B EY1 S] guitar while catching a [B AE1 S] fish.")

Command-Line Tools Copy Chapter

Three CLI entry points are provided. The CLI tools support all features available in the Python API (voice cloning, voice design, auto voice, generation parameters, etc.) — all controlled via command-line arguments.

Demo Copy Section

omnivoice-demo --ip 0.0.0.0 --port 8001

Provides a web UI for voice cloning and voice design. See omnivoice-demo --help for all options.

Single Inference Copy Section

# Voice Cloning
# ref_text can be omitted (Whisper will auto-transcribe ref_audio to get it).
omnivoice-infer \
    --model k2-fsa/OmniVoice \
    --text "This is a test for text to speech." \
    --ref_audio ref.wav \
    --ref_text "Transcription of the reference audio." \
    --output hello.wav

# Voice Design
omnivoice-infer --model k2-fsa/OmniVoice \
    --text "This is a test for text to speech." \
    --instruct "male, British accent" \
    --output hello.wav

# Auto Voice
omnivoice-infer \
    --model k2-fsa/OmniVoice \
    --text "This is a test for text to speech."\
    --output hello.wav

Batch Inference Copy Section

omnivoice-infer-batch can distribute batch inference across multiple GPUs, designed for large-scale TTS tasks.

omnivoice-infer-batch \
    --model k2-fsa/OmniVoice \
    --test_list test.jsonl \
    --res_dir results/

The test list is a JSONL file where each line is a JSON object:

{"id": "sample_001", "text": "Hello world", "ref_audio": "/path/to/ref.wav", "ref_text": "Reference transcript", "instruct": "female, british accent", "language_id": "en", "duration": 10.0, "speed": 1.0}

Only id and text are mandatory fields. ref_audio and ref_text are used in voice cloning mode. instruct is used in voice design mode. If no reference audio or instruct are provided, the model will generate text in a random voice.

language_id, duration, and speed are optional. duration (in seconds) fixes the output length; speed controls the speaking rate. If duration and speed are both provided, speed will be ignored.

FlashInfer Acceleration Copy Section

Inference can be accelerated ~2-2.9x losslessly with FlashInfer kernels (sequence packing for the CFG cond/uncond pair, fused RMSNorm/RoPE/GEMM kernels, and optional CUDA graphs).

Installation (NVIDIA GPUs; pick the index matching your CUDA version, e.g. cu128 for PyTorch built with CUDA 12.8):

pip install flashinfer-python==0.6.15.post1 "flashinfer-jit-cache==0.6.15.post1+cu128" \
    --extra-index-url https://flashinfer.ai/whl/cu128/

Usage with the batch inference CLI:

omnivoice-infer-batch \
    --model k2-fsa/OmniVoice \
    --test_list test.jsonl \
    --res_dir results/ \
    --batch_size 8 \
    --enable_flashinfer true

or with the Python API:

from omnivoice.models.omnivoice_flashinfer import apply_flashinfer

model = OmniVoice.from_pretrained("k2-fsa/OmniVoice", device_map="cuda", dtype=torch.float16)
apply_flashinfer(model)                          # throughput / batched inference
apply_flashinfer(model, enable_cuda_graph=True)  # recommended for batch=1 (low latency)

CUDA graphs are recommended for single-stream (batch=1) usage, where kernel-launch overhead dominates; at batch >= 4 the plain FlashInfer path is already the fastest configuration.

Benchmark (seed-tts zh testset, 2020 samples / 3.3h audio, voice cloning, single H100, fp16, num_step=32; Average RTF as reported by omnivoice-infer-batch, outputs ASR-verified lossless):

Training & Evaluation Copy Chapter

See examples/ for the complete pipeline — from data preparation to training, evaluation, and finetuning.

Discussion & Communication Copy Chapter

You can directly discuss on GitHub Issues.

You can also scan the QR code to join our wechat group or follow our wechat official account.

Community Projects Copy Chapter

OmniVoice is supported by a growing ecosystem of community projects.
Explore them in Community Projects.

Citation Copy Chapter

@article{zhu2026omnivoice,
      title={OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models},
      author={Zhu, Han and Ye, Lingxuan and Kang, Wei and Yao, Zengwei and Guo, Liyong and Kuang, Fangjun and Han, Zhifeng and Zhuang, Weiji and Lin, Long and Povey, Daniel},
      journal={arXiv preprint arXiv:2604.00688},
      year={2026}
}

Disclaimer Copy Chapter

Users are strictly prohibited from using this model for unauthorized voice cloning, voice impersonation, fraud, scams, or any other illegal or unethical activities. All users shall ensure full compliance with applicable local laws, regulations, and ethical standards. The developers assume no liability for any misuse of this model and advocate for responsible AI development and use, encouraging the community to uphold safety and ethical principles in AI research and applications.

File: CONTRIBUTING.md Copy Chapter

Contributing to OmniVoice

Thank you for your interest in contributing to OmniVoice! We use pre-commit hooks to ensure code quality and consistency. Before contributing, please follow these guidelines to enable and use the pre-commit hooks.

Setup Copy Chapter

pip install pre-commit
pre-commit install

This enables automatic code style checks (linting, formatting, trailing whitespace, etc.) on every git commit.

Before committing, run:

pre-commit run --all-files

This will auto-fix any style issues. Then stage and commit as usual:

git add .
git commit -m "your message"

File: .pre-commit-config.yaml Copy Chapter

repos:

• repo: https://github.com/astral-sh/ruff-pre-commit
  rev: v0.11.13
  hooks:
  • id: ruff
    args: ["--fix"]
    exclude: ^(docs|examples)/
  • id: ruff-format
    exclude: ^(docs|examples)/

File: docs/community-projects.md Copy Chapter

Community Projects

The following projects are built and maintained by the community. We appreciate all contributions! Note that these projects are not officially supported by the OmniVoice team.

If you have a project you'd like to add, please open a PR.

• ComfyUI-OmniVoice-TTS —
  ComfyUI custom node for OmniVoice text-to-speech generation.

• vLLM-Omni —
  A framework for efficient model inference with omni-modality model. Supports OmniVoice serving.

• audio.cpp —
  An all-in-one, pure C++ inference engine for audio models (TTS, ASR, music gen,etc), powered by GGML.

• pyVideoTrans —
  Video translation tool with dubbing & subtitles. Supports OmniVoice as a TTS engine.

• MLX-Audio —
  TTS, STT, and STS library built on Apple's MLX framework. Supports
  OmniVoice among other models for efficient speech processing on Apple Silicon.

• OmniVoice-MLX —
  MLX inference backend and conversion/staging tools for running OmniVoice on
  Apple Silicon, with community model weights hosted under
  mlx-community.

• RealtimeTTS —
  Converts text to speech in realtime. Supports OmniVoice as a TTS engine.

• TTS-WebUI —
  Gradio web UI for multiple TTS models. Supports OmniVoice as one of its backends.

• OmniVoice-Studio —
  Desktop application for OmniVoice voice generation.

• omnivoice-server —
  OpenAI-compatible HTTP server for serving OmniVoice via /v1/audio/speech.
  Supports voice profiles for persistent cloning, sentence-level streaming,
  and optional Bearer auth.

• omnivoice-rs —
  GPU-first Rust workspace for OmniVoice inference, parity validation, CLI
  execution, and an OpenAI-compatible HTTP server built with Candle.

• omnivoice-trtllm —
  Deploy OmniVoice TTS model using TensorRT-LLM and Triton Inference Server
  on Modal, faster than PyTorch.

• Auris —
  Offline audiobook reader for EPUB, PDF, and TXT with local OmniVoice TTS, character-aware voices, and per-book narrator control.

• LA Studio —
  Offline desktop AI audio workstation for local TTS, voice cloning, and voice design workflows with OmniVoice support.

• tts-audiobook-tool —
  Audiobook creation tool supporting a dozen different TTS models including OmniVoice, Qwen3-TTS, VibeVoice, etc.,
  focused on high-quality output

• LocalText2Voice —
  User-friendly Windows desktop pipeline for long-form audiobook and podcast production with OmniVoice support, voice-library management, chapter-aware generation, Whisper-based segment review and retries, multi-voice markup, music/SFX mixing, and MCP automation.

File: docs/data_preparation.md Copy Chapter

Data Preparation

OmniVoice trains on a custom WebDataset format where audio data is packed into tar shards with paired JSONL metadata files. Each tar shard contains hundreds to thousands of samples (as .npy audio token arrays), drastically reducing disk I/O during training. The separated jsonl file allows for easier modification of metadata. This document explains the data format in detail and walks through the preparation pipeline.

1. Input Format Copy Chapter

Prepare a JSONL file where each line is a JSON object:

{"id": "sample_001", "audio_path": "/data/audio/001.wav", "text": "Hello world", "language_id": "en"}
{"id": "sample_002", "audio_path": "/data/audio/002.wav", "text": "你好世界", "language_id": "zh"}

Fields:

• id — unique sample identifier (used to match samples across shards and label files)
• audio_path — absolute path to the audio file (wav/flac/mp3, will be resampled to 24 kHz)
• text — transcript text
• language_id — (optional) language code, used for multilingual training, can be omitted

2. Processing Copy Chapter

The tokenization script extract_audio_tokens.py converts audio into 8-layer discrete tokens and packs them into WebDataset shards.

export CUDA_VISIBLE_DEVICES="0,1,2,4"  # GPUs used for token extraction
python -m omnivoice.scripts.extract_audio_tokens \
    --input_jsonl data.jsonl \
    --tar_output_pattern output/audios/shard-%06d.tar \
    --jsonl_output_pattern output/txts/shard-%06d.jsonl \
    --tokenizer_path eustlb/higgs-audio-v2-tokenizer \
    --nj_per_gpu 3 \
    --shuffle True

What it does:

1. Reads your JSONL manifest
2. Encodes each audio file into discrete tokens using audio tokenizer
3. Packs tokens into WebDataset tar shards with paired jsonl metadata files
4. Generates a data.lst manifest file

Alternative: WebDataset Input (if you already have raw-audio tar shards)

Pass the data.lst manifest instead of --input_jsonl:

export CUDA_VISIBLE_DEVICES="0,1,2,4"  # GPUs used for token extraction
python -m omnivoice.scripts.extract_audio_tokens \
    --input_manifest existing_data/data.lst \
    --tar_output_pattern output/audios/shard-%06d.tar \
    --jsonl_output_pattern output/txts/shard-%06d.jsonl \
    --tokenizer_path eustlb/higgs-audio-v2-tokenizer \
    --nj_per_gpu 3 \
    --shuffle True

The existing_data/data.lst is generated with:

python -m omnivoice.scripts.jsonl_to_webdataset \
    --input data.jsonl \
    --output data/shards \
    --sr 24000 \
    --shard-size 1000

This resamples audio to the target sample rate and packs FLAC files into tar shards with paired jsonl metadata files.

Explanation of the script's options: Copy Section

Output Structure Copy Section

Output structure with the following output patterns

--tar_output_pattern output/audios/shard-%06d.tar \
--jsonl_output_pattern output/txts/shard-%06d.jsonl

will be:

output/
├── audios/                    # WebDataset tar shards (audio tokens)
│   ├── shard-000000.tar       # Each tar packs ~1000 samples
│   ├── shard-000001.tar
│   └── ...
├── txts/                      # Per-shard companion JSONL labels
│   ├── shard-000000.jsonl     # One JSON line per sample in the corresponding tar
│   ├── shard-000001.jsonl
│   └── ...
├── data.lst                   # Manifest linking tar ↔ jsonl shards
└── errors.jsonl               # Samples that failed processing (if any)

data.lst and errors.jsonl are written to the parent directory of audios/ and txts/.

The `data.lst` manifest Copy Section

Each line in data.lst describes one shard:

/path/to/shard-000000.tar /path/to/shard-000000.jsonl 1000 3600.500
/path/to/shard-000001.tar /path/to/shard-000001.jsonl 800 2880.200

Format: <tar_path> <jsonl_path> <num_samples> <total_duration_seconds>

• Paths are absolute
• .tar file contains the audio tokens.
• .jsonl file contains the metadata in the original provided JSONL file, allows easier access and modification of metadata without decompressing the tar file.
• This manifest is what the training data config references.

Inside a tar shard Copy Section

Each .tar file packs many samples (default 1000 per shard) into a single archive. This is the key advantage of WebDataset: instead of reading thousands of tiny files, the dataloader reads sequentially from a few large tars, drastically reducing disk I/O pressure.

Each sample in the tar is a pair of files with matching keys:

shard-000000.tar:
  sample_001.npy    # Audio tokens: numpy array, shape [8, T], dtype int16
  sample_002.npy
  ...
  sample_1000.npy

3. Data Config for Training Copy Chapter

After creating WebDataset shards, write a data config JSON that references them:

{
    "train": [
        {
            "language_id": "en",
            "manifest_path": ["data/custom/tokens/train/data.lst"],
            "repeat": 1
        }
    ],
    "dev": [
        {
            "language_id": "en",
            "manifest_path": ["data/custom/tokens/dev/data.lst"],
            "repeat": 1
        }
    ]
}

• manifest_path — list of data.lst files (one per shard directory)
• repeat — how many times to repeat this dataset per epoch (useful for balancing languages)
• language_id is not used, just for a better data organization.

See examples/config/ for ready-to-use data config files.

See docs/data_preparation_advanced.md for denoising and noise augmentation.

File: docs/data_preparation_advanced.md Copy Chapter

Advanced Data Preparation

The advanced pipeline adds denoising and prompt noise augmentation on top of the basic tokenization workflow. Each stage is optional.

Prerequisites Copy Chapter

• Denoising: Sidon model checkpoints (feature_extractor_cuda.pt, decoder_cuda.pt) from https://huggingface.co/sarulab-speech/sidon-v0.1/tree/main.
• Noise augmentation: noise + RIR tar shards with data.lst manifests

Pipeline Overview Copy Chapter

Step 1 (optional): Denoise
  Raw audio → Sidon denoiser → clean audio

Step 2: Tokenize (with optional noise augmentation)
  Clean audio + noise augment on prefix → audio tokenizer → tokens

Denoise Copy Chapter

Use the Sidon speech enhancement model to remove background noise from raw audio.

export CUDA_VISIBLE_DEVICES="0,1,2,3"
python -m omnivoice.scripts.denoise_audio \
    --input_jsonl data.jsonl \
    --tar_output_pattern data/denoised/audios/shard-%06d.tar \
    --jsonl_output_pattern data/denoised/txts/shard-%06d.jsonl \
    --feature_extractor_path /path/to/sidon_feature_extractor_cuda.pt \
    --decoder_path /path/to/sidon_decoder_cuda.pt \
    --target_sample_rate 24000 \
    --batch_duration 200.0

What it does:

1. Reads your JSONL manifest
2. Runs Sidon denoiser on each audio file
3. Outputs denoised audio as custom WebDataset tar/jsonl shards
4. Generates a data.lst manifest in data/denoised/

You can also pass --input_manifest /path/to/data.lst if you already have a custom webdataset format dataset.
The next step would be passing the generated data.lst file with --input_manifest to omnivoice.scripts.extract_audio_tokens for tokens extraction.

Tokenize with noise augmentation Copy Section

Adds environmental noise and room reverb to prompt audio during tokenization, making the model robust to noisy reference audio at inference time. Note that in our model, we only add noise augmentation for a small proportion of data, making sure the model can also generate good audio with clean reference audio.

You need two additional datasets in WebDataset format:

• Noise recordings: environmental noise tar shards with a data.lst manifest
• Room impulse responses (RIR): RIR tar shards with a data.lst manifest

export CUDA_VISIBLE_DEVICES="0,1,2,4"
python -m omnivoice.scripts.extract_audio_tokens_add_noise \
    --input_jsonl data.jsonl \
    --tar_output_pattern data/tokens/shard-%06d.tar \
    --jsonl_output_pattern data/txts/shard-%06d.jsonl \
    --tokenizer_path eustlb/higgs-audio-v2-tokenizer \
    --noise_manifest data/noise_shards/data.lst \
    --rir_manifest data/rir_shards/data.lst \
    --nj_per_gpu 3

You can also pass --input_manifest /path/to/data.lst if you already have a custom webdataset format dataset.

File: docs/evaluation.md Copy Chapter

Evaluation

Evaluate OmniVoice models with standard TTS metrics: WER (intelligibility), SIM-o (speaker similarity), and UTMOS (naturalness).

Supported Test Sets Copy Chapter

Prerequisites Copy Chapter

pip install omnivoice[eval]
# or
uv sync --extra eval

Quick Start Copy Chapter

cd examples
bash run_eval.sh
# run_eval.sh will
# (1) download all required test sets and test models;
# (2) inference and evaluation for each test set.

Metrics Explained Copy Chapter

WER (Word Error Rate) Copy Section

Measures how intelligible the generated speech is by transcribing it with an ASR model and comparing to the reference text. Lower is better. Note that some languages actually use CER (Character Error Rate).

• LibriSpeech-PC: HuBERT-based ASR
• Seed-TTS: Whisper (en) or Paraformer (zh)
• MiniMax: Whisper for non-Chinese, Paraformer for Chinese
• FLEURS: Omnilingual-ASR multilingual model

Speaker Similarity Copy Section

Cosine similarity between speaker embeddings (ECAPA-TDNN + WavLM) of the reference and generated audio. Higher is better.

UTMOS (Predicted MOS) Copy Section

Neural network that predicts Mean Opinion Score from audio. Higher is better.

File: docs/generation-parameters.md Copy Chapter

Generation Parameters

Parameters can be passed as keyword arguments to model.generate(...) or via the OmniVoiceGenerationConfig dataclass. See below for the full list and which category each belongs to.

# 1) Direct keyword arguments
audio = model.generate(text="Hello world", num_step=32, guidance_scale=2.0)

# 2) Via OmniVoiceGenerationConfig dataclass
from omnivoice import OmniVoiceGenerationConfig

config = OmniVoiceGenerationConfig(num_step=32, guidance_scale=2.0)
audio = model.generate(text="Hello world", generation_config=config)

Decoding Copy Chapter

Sampling Copy Chapter

Duration & Speed Copy Chapter

These accept a single value applied to all items, or a per-item list (useful in batch mode):

# Fixed 10-second output
audio = model.generate(text="Hello, this is a test of duration control", duration=10.0)

# Faster speech (1.2x faster than estimated)
audio = model.generate(text="Hello, this is a test of duration control", speed=1.2)

Priority: duration > speed.

Note: When using duration, the default post-processing step may trim trailing silence, causing the actual output to be slightly shorter than the requested duration. If you need the output duration to exactly match the specified value, set postprocess_output=False to disable silence removal.

Pre/Post Processing Copy Chapter

Long-Form Generation Copy Chapter

To support stable long-form speech generation with low VRAM consumption, the text is automatically split into smaller segments when the estimated duration of the generated speech exceeds audio_chunk_duration, with each segment producing approximately audio_chunk_duration seconds of audio. This approach allows the model to accept arbitrarily long text and generate arbitrarily long speech with near-constant VRAM consumption.

File: docs/languages.md Copy Chapter

Supported Languages

OmniVoice supports 646 languages with a total of 581k hours of training data.

The table below lists each language with its OmniVoice language ID,
ISO 639-3 code, and training data duration (hours).

646 languages, 581k hours total.

Data source: docs/lang_id_name_map.tsv

File: docs/lora_finetuning.md Copy Chapter

LoRA Finetuning

In addition to full finetuning (see training.md), OmniVoice supports parameter-efficient finetuning via LoRA (Low-Rank Adaptation), using PEFT. LoRA freezes the pretrained LLM backbone and trains small low-rank adapter matrices instead — typically 3-5% of parameters trainable — which means:

• Much lower optimizer memory (Adam keeps 2 extra tensors per trainable parameter).
• Much smaller adapter checkpoints to ship (tens of MB vs. the full model size).
• The original pretrained weights stay untouched on disk; the adapter can be dropped to recover the original model exactly.

LoRA is opt-in via config — with use_lora: false (the default), training behaves exactly as documented in training.md.

Installation Copy Chapter

pip install "omnivoice[lora]"
# or, from a source checkout:
pip install peft

What Gets Adapted Copy Chapter

Everything else (the LLM's other weights, layernorms, etc.) stays frozen, exactly as loaded from the base checkpoint.

Configuration Copy Chapter

Start from examples/config/train_config_finetune_lora.json — identical to train_config_finetune_sdpa.json plus the LoRA fields:

{
    "init_from_checkpoint": "k2-fsa/OmniVoice",

    "use_lora": true,
    "lora_r": 16,
    "lora_alpha": 32,
    "lora_dropout": 0.05,
    "lora_bias": "none",
    "lora_target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    "lora_modules_to_save": ["audio_embeddings", "audio_heads"],

    "learning_rate": 1e-4,
    "attn_implementation": "sdpa"
}

A higher learning_rate than full finetuning (1e-4 vs. 5e-5) is typically used since the adapter matrices are randomly initialized and trained from scratch, rather than finetuning already-converged pretrained weights — a higher LR converges faster without the instability risk full finetuning has at that LR.

Launching Training Copy Chapter

accelerate launch \
    --gpu_ids "0,1" \
    --num_processes 2 \
    -m omnivoice.cli.train \
    --train_config examples/config/train_config_finetune_lora.json \
    --data_config examples/config/data_config_finetune.json \
    --output_dir exp/omnivoice_finetune_lora

Or run the full example pipeline (tokenization + training):

bash examples/run_finetune_lora.sh

On startup you'll see a line like:

trainable params: 19,316,736 || all params: 631,894,016 || trainable%: 3.0570

confirming only the adapters plus audio_embeddings/audio_heads are being trained.

How Many Checkpoints Get Saved Copy Chapter

Same three fields as full finetuning govern this:

Number of checkpoints saved = steps / save_steps (when steps is a multiple of save_steps), capped at keep_last_n_checkpoints if positive.

Resuming Training Copy Chapter

Identical mechanism to full finetuning — set resume_from_checkpoint:

{
    "resume_from_checkpoint": "exp/omnivoice_finetune_lora/checkpoint-500"
}

Each run rebuilds the same LoRA-wrapped model architecture from init_from_checkpoint plus the lora_* config fields, then restores optimizer/scheduler/RNG state and adapter weights from the checkpoint on top of it. Keep init_from_checkpoint and the lora_* fields unchanged across a resume — a mismatch will surface as a shape/key error when accelerate restores state.

Adapter Inference Copy Chapter

Apply a trained adapter on top of the base model at inference time — merged into the model in-memory (not written to disk), so generation uses the regular OmniVoice.generate() path unmodified:

omnivoice-infer \
    --model k2-fsa/OmniVoice \
    --lora_adapter exp/omnivoice_finetune_lora/checkpoint-500 \
    --text "Hello, this is a text for text-to-speech." \
    --ref_audio ref.wav --ref_text "Reference transcript." \
    --output out.wav

--lora_adapter accepts any directory containing adapter_config.json + adapter_model.safetensors — i.e. any LoRA training checkpoint directory.

Adapter Merging Copy Chapter

To deploy with no PEFT dependency at inference time, merge the adapter into the base model once and save the result as a normal, standalone OmniVoice checkpoint:

omnivoice-merge-lora \
    --base_model k2-fsa/OmniVoice \
    --lora_adapter exp/omnivoice_finetune_lora/checkpoint-500 \
    --output_dir exp/omnivoice_finetune_lora/merged

This loads the base model, merges the LoRA deltas into its weights, and writes a complete, self-contained OmniVoice directory (model weights, tokenizer, and audio tokenizer).

Using the Merged Model Copy Chapter

The merged directory is a normal OmniVoice checkpoint — use it exactly like any other:

omnivoice-infer \
    --model exp/omnivoice_finetune_lora/merged \
    --text "Hello, this is a text for text-to-speech." \
    --instruct "male, British accent" \
    --output out.wav

from omnivoice.models.omnivoice import OmniVoice

model = OmniVoice.from_pretrained("exp/omnivoice_finetune_lora/merged", device_map="cuda:0")
audios = model.generate(text="Hello world", language="en")

The merged model is saved in whatever dtype the base model was loaded in when merging (float32 by default, since omnivoice-merge-lora loads the base model with dtype=torch.float32). Pass --dtype fp32 / dtype=torch.float32 at inference to match exactly, or fp16/bf16 for less VRAM and faster inference — from_pretrained casts down automatically, and quality differences are usually negligible.

Checkpoint Structure Copy Chapter

A LoRA training checkpoint (exp/.../checkpoint-<step>/) contains two independent sets of files:

checkpoint-500/
├── adapter_config.json        # PEFT LoRA config (small)
├── adapter_model.safetensors  # LoRA weights + audio_embeddings/audio_heads (small)
├── tokenizer.json             # text tokenizer, saved for adapter/merge use
├── tokenizer_config.json
├── model.safetensors          # accelerate's full training state (base + adapter weights)
├── optimizer.bin              # optimizer state, for resuming
├── scheduler.bin              # LR scheduler state, for resuming
└── random_states_0.pkl        # RNG state, for resuming

• For adapter inference / merging, only adapter_config.json + adapter_model.safetensors (+ tokenizer) are used.
• For resuming training, accelerate's own state files (model.safetensors, optimizer.bin, scheduler.bin, random_states_*.pkl) are used — these include the full (frozen + trainable) model state, since accelerate's checkpointing does not distinguish frozen from trainable parameters. This makes checkpoint-*/ directories larger than the adapter alone, but keeps resuming exactly as reliable as full finetuning's. If you only need the adapter for deployment, you can delete everything except adapter_config.json/adapter_model.safetensors/the tokenizer files once training is done.

Troubleshooting Copy Chapter

ModuleNotFoundError: No module named 'peft'
Run pip install peft (or pip install "omnivoice[lora]").

Resume fails with a shape/key mismatch error
resume_from_checkpoint must be resumed with the same init_from_checkpoint, lora_r, lora_alpha, lora_target_modules, and lora_modules_to_save as the run that produced the checkpoint — these determine the model architecture accelerate's state restore expects to find.

omnivoice-merge-lora / --lora_adapter can't find the adapter
Point it at a checkpoint directory, not a specific file — it must contain adapter_config.json.

Training loss looks the same as a frozen model / doesn't move
Check the trainable params: ... line printed at startup. If it's 0 or doesn't include LoRA layers, verify use_lora: true is set in the config passed to --train_config, and that lora_target_modules names match submodules in your llm_name_or_path's architecture.

Checkpoint saves or merging are unexpectedly slow
Each checkpoint writes a full base-model-sized file (accelerate's training state — see Checkpoint Structure), and merging both reads and writes one. If disk space is tight or I/O is contended, these can take much longer than expected; check available disk space first.

File: docs/tips.md Copy Chapter

Tips & Notes

• Combination of ref_audio and instruct:
  When both ref_audio and instruct are provided and they conflict, the model will most likely follow the style of the reference audio. When the two are consistent, instruct can improve cloning stability for the attributes it describes. A typical example is Chinese dialect cloning: provide both dialect reference audio and a matching dialect instruct (e.g., ref_audio="sichuan.wav", instruct="四川话") for more stable dialect output.

• Short Audio Generation:
  The model may not reliably generate short audio clips (e.g., 1–2 seconds) without reference audio. If you need to generate short clips, provide reference audio to the model.

• Min Nan Chinese (Hokkien) Input Format:
  Min Nan Chinese (闽南语, also known as Hokkien) can only be synthesized using Tai-lo romanization as input; Chinese characters are not supported for Min Nan Chinese in the current model version.

File: docs/training.md Copy Chapter

Training

Training Config Copy Chapter

All training is controlled by a JSON training config file and a JSON data config file.

See examples/config/ for ready-to-use configs.

Training config file on Emilia is: examples/config/train_config_emilia.json

Data config file for Emilia is: examples/config/data_config_emilia.json

Key fields in training config file:

output_dir and data_config are passed via command line (see below).

Attention Implementation Copy Chapter

By default, training uses flex_attention, which requires PyTorch ≥ 2.5 and a compatible GPU (e.g. NVIDIA Ampere or newer). If your environment does not support flex_attention, set attn_implementation to "sdpa" in your training config. See examples/config/train_config_finetune_sdpa.json for a ready-to-use SDPA config:

{
    "attn_implementation": "sdpa",
    "max_sample_tokens": 2000,
    "min_sample_tokens": 50,
    "max_batch_size": 64
}

"sdpa" uses PyTorch's built-in scaled dot-product attention and works on a wider range of hardware.

The following fields only apply when attn_implementation != "flex_attention":

batch_tokens remains the primary control for memory usage — it sets the total token budget per batch. max_batch_size is a safety guard to prevent a batch of many short samples from creating an unusually large batch dimension.

Batching strategy Copy Section

The two backends use different batching strategies, which are selected automatically:

Why different strategies?

• With flex_attention, sequence packing is memory-efficient because a compact BlockMask (not a dense matrix) describes which tokens can attend to each other across document boundaries.
• With sdpa, length-grouped padding is used instead: samples of similar token lengths are batched together and padded to the local maximum, so a lightweight [B, 1, max_len, max_len] boolean attention mask suffices with low overhead and minimal wasted padding.

Launching Training Copy Chapter

accelerate launch \
    --gpu_ids "0,1,2,3,4,5,6,7" \
    --num_processes 8 \
    -m omnivoice.cli.train \
    --train_config config/train_config_emilia.json \
    --data_config config/data_config_emilia.json \
    --output_dir exp/omnivoice_emilia

Resuming Training Copy Chapter

Set resume_from_checkpoint in your training config to resume from an existing checkpoint:

{
    "resume_from_checkpoint": "exp/omnivoice/checkpoint-100000"
}

Initializing from a Pretrained Model Copy Chapter

To start training from a pretrained OmniVoice checkpoint (for fine-tuning):

{
    "init_from_checkpoint": "exp/omnivoice/checkpoint-100000"
}

LoRA Finetuning Copy Chapter

For parameter-efficient finetuning, set "use_lora": true in your training config to train low-rank adapters instead of the full model. See lora_finetuning.md for configuration, adapter inference, and merging back into a standalone checkpoint.

Monitoring Copy Chapter

Training logs to TensorBoard:

tensorboard --logdir exp/omnivoice_emilia/tensorboard

File: docs/voice-design.md Copy Chapter

Voice Design

Voice Design mode lets you describe the desired speaker through speaker attributes (instruct parameter) — no reference audio needed. The model
generates a matching voice on the fly.

Quick Example Copy Chapter

import torch
from omnivoice import OmniVoice

model = OmniVoice.from_pretrained(
    "k2-fsa/OmniVoice",
    device_map="cuda:0",
    dtype=torch.float16
)

audio = model.generate(
    text="This is a test for voice design.",
    instruct="female, young adult, high pitch, british accent",
)

How It Works Copy Chapter

The instruct parameter accepts a comma-separated string of speaker attributes.
Each attribute belongs to a category (gender, age, pitch, style, accent,
or dialect). Within a category, only one attribute may be selected at a time.
Attributes from different categories can be freely combined.

The model auto-detects the language of the instruct text and normalises it
internally — you can write in English, Chinese, or a mix of both.

Supported Attributes Copy Chapter

Gender Copy Section

Age Copy Section

Pitch Copy Section

Style Copy Section

English Accent Copy Section

Only effective when the synthesis text is in English.

Chinese Dialect Copy Section

Only effective when the synthesis text is in Chinese.

Writing Instruct Strings Copy Chapter

Separate attributes with commas (half-width , for English, full-width ，
for Chinese — the model auto-fixes mismatches).

# English
"female, young adult, high pitch, british accent"

# Chinese
"女，青年，高音调，四川话"

# Mixed (auto-normalised)
"female, young adult, 四川话"

Tips Copy Section

• Combine freely across categories: "male, elderly, low pitch, whisper".

• Leave it to the model: omit attributes you don't care about — the model
  fills in the rest. For example "female" alone is valid.

• Case-insensitive: "Male", "MALE", and "male" are all accepted, the code will normalize them to lower case.

• Accent vs Dialect: English accents are only applied to English speech, Chinese dialects are only applied to Chinese speech.

• Attribute combinations: Due to training data limitations, some attribute combinations may not work well — the model may ignore certain attributes in a combination. If the output doesn't match your expectation, try simplifying the instruct string.

File: examples/README.md Copy Chapter

OmniVoice Examples

This directory contains scripts and configs for training, fine-tuning, and evaluating OmniVoice.

Training from Scratch (Emilia) Copy Chapter

run_emilia.sh runs the full pipeline in 3 stages:

Prerequisites:

1. Download the Emilia dataset from OpenXLab and place it under download/:

   text Copy

   download/Amphion___Emilia
   └── raw
       ├── EN
       └── ZH

2. Obtain JSONL manifests and place them in data/emilia/manifests/:

   • emilia_en_train.jsonl, emilia_en_dev.jsonl
   • emilia_zh_train.jsonl, emilia_zh_dev.jsonl

   You can generate them from the raw data, or download pre-processed manifests from HuggingFace.

Run the full pipeline:

bash examples/run_emilia.sh

Or run individual stages by setting stage and stop_stage at the top of the script (e.g. stage=1, stop_stage=1 to only tokenize).

See docs/training.md for config details, checkpoint resuming, and TensorBoard monitoring.

Fine-tuning Copy Chapter

run_finetune.sh fine-tunes from a pretrained checkpoint on your own data.

Step 1: Prepare Your Data Copy Section

Create a JSONL manifest where each line describes one audio sample:

{"id": "sample_001", "audio_path": "/data/audio/001.wav", "text": "Hello world", "language_id": "en"}
{"id": "sample_002", "audio_path": "/data/audio/002.wav", "text": "你好世界", "language_id": "zh"}

id, audio_path, and text are mandatory. language_id is optional.

See docs/data_preparation.md for the full data format specification.

Step 2: Configure the Script Copy Section

Edit the variables at the top of run_finetune.sh:

TRAIN_JSONL="data/my_data_train.jsonl"   # path to training JSONL
DEV_JSONL="data/my_data_dev.jsonl"       # path to dev JSONL
GPU_IDS="0,1"                            # GPUs to use
NUM_GPUS=2
OUTPUT_DIR="exp/omnivoice_finetune"      # output directory

Step 3: Run Copy Section

bash examples/run_finetune.sh

The script will:

1. Tokenize your audio into WebDataset shards
2. Launch fine-tuning with accelerate

Main difference between fine-tuning config (config/train_config_finetune.json) and the Emilia training config (config/train_config_emilia.json) are:

To use a different pretrained checkpoint, modify init_from_checkpoint in the config file.

If you encounter issues with flex_attention on your GPU, use config/train_config_finetune_sdpa.json instead, which uses SDPA attention for broader compatibility. See docs/training.md for details.

Evaluation Copy Chapter

Install evaluation dependencies first:

pip install omnivoice[eval]
# or
uv sync --extra eval

Supported test sets: librispeech_pc, seedtts_en, seedtts_zh, fleurs, minimax.

bash examples/run_eval.sh

See docs/evaluation.md for metrics details, test set preparation, and running individual metrics.

File: examples/config/data_config_emilia.json Copy Chapter

{
"train":
[
{
"language_id": "en",
"manifest_path": [
"data/emilia/tokens/emilia_en_train/data.lst"
],
"repeat": 1
},
{
"language_id": "zh",
"manifest_path": [
"data/emilia/tokens/emilia_zh_train/data.lst"
],
"repeat": 1
}
],
"dev":
[
{
"language_id": "en",
"manifest_path": [
"data/emilia/tokens/emilia_en_dev/data.lst"
],
"repeat": 1
},
{
"language_id": "zh",
"manifest_path": [
"data/emilia/tokens/emilia_zh_dev/data.lst"
],
"repeat": 1
}
]
}

File: examples/config/data_config_finetune.json Copy Chapter

{
"train": [
{
"manifest_path": ["data/finetune/tokens/train/data.lst"]
}
],
"dev": [
{
"manifest_path": ["data/finetune/tokens/dev/data.lst"]
}
]
}

File: examples/config/ds_config_zero2.json Copy Chapter

{
"steps_per_print": 100,
"zero_optimization": {
"stage": 2,
"allgather_partitions": true,
"allgather_bucket_size": 2e8,
"overlap_comm": true,
"reduce_scatter": true,
"reduce_bucket_size": 2e8,
"contiguous_gradients": true
},
"gradient_accumulation_steps": "auto",
"gradient_clipping": "auto",
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto",
"bf16": {
"enabled": "auto"
}
}

File: examples/config/train_config_emilia.json Copy Chapter

{
"llm_name_or_path": "Qwen/Qwen3-0.6B",
"audio_vocab_size": 1025,
"audio_mask_id": 1024,
"num_audio_codebook": 8,

"audio_codebook_weights": [8, 8, 6, 6, 4, 4, 2, 2],
"drop_cond_ratio": 0.1,
"prompt_ratio_range": [0.0, 0.3],
"mask_ratio_range": [0.0, 1.0],
"language_ratio": 0.0,
"use_pinyin_ratio": 0.0,
"instruct_ratio": 0.0,
"only_instruct_ratio": 0.0,

"resume_from_checkpoint": null,
"init_from_checkpoint": null,

"learning_rate": 1e-4,
"weight_decay": 0.01,
"max_grad_norm": 1.0,
"steps": 300000,
"seed": 42,
"warmup_type": "ratio",
"warmup_ratio": 0.03,
"warmup_steps": 0,

"batch_tokens": 8192,
"gradient_accumulation_steps": 1,
"num_workers": 4,

"mixed_precision": "bf16",
"allow_tf32": true,

"logging_steps": 100,
"eval_steps": 1000,
"save_steps": 10000,
"keep_last_n_checkpoints": -1

}

File: examples/config/train_config_finetune.json Copy Chapter

{
"llm_name_or_path": "Qwen/Qwen3-0.6B",
"audio_vocab_size": 1025,
"audio_mask_id": 1024,
"num_audio_codebook": 8,

"audio_codebook_weights": [8, 8, 6, 6, 4, 4, 2, 2],
"drop_cond_ratio": 0.1,
"prompt_ratio_range": [0.0, 0.3],
"mask_ratio_range": [0.0, 1.0],
"language_ratio": 0.8,
"use_pinyin_ratio": 0.0,
"instruct_ratio": 0.0,
"only_instruct_ratio": 0.0,

"resume_from_checkpoint": null,
"init_from_checkpoint": "k2-fsa/OmniVoice",

"learning_rate": 1e-5,
"weight_decay": 0.01,
"max_grad_norm": 1.0,
"steps": 5000,
"seed": 42,
"warmup_type": "ratio",
"warmup_ratio": 0.01,
"warmup_steps": 0,

"batch_tokens": 8192,
"gradient_accumulation_steps": 1,
"num_workers": 2,

"mixed_precision": "bf16",
"allow_tf32": true,

"logging_steps": 50,
"eval_steps": 500,
"save_steps": 500,
"keep_last_n_checkpoints": -1

}

File: examples/config/train_config_finetune_lora.json Copy Chapter

{
"llm_name_or_path": "Qwen/Qwen3-0.6B",
"audio_vocab_size": 1025,
"audio_mask_id": 1024,
"num_audio_codebook": 8,

"audio_codebook_weights": [8, 8, 6, 6, 4, 4, 2, 2],
"drop_cond_ratio": 0.1,
"prompt_ratio_range": [0.0, 0.3],
"mask_ratio_range": [0.0, 1.0],
"language_ratio": 0.8,
"use_pinyin_ratio": 0.0,
"instruct_ratio": 0.0,
"only_instruct_ratio": 0.0,

"resume_from_checkpoint": null,
"init_from_checkpoint": "k2-fsa/OmniVoice",

"use_lora": true,
"lora_r": 16,
"lora_alpha": 32,
"lora_dropout": 0.05,
"lora_bias": "none",
"lora_target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
"lora_modules_to_save": ["audio_embeddings", "audio_heads"],

"learning_rate": 1e-4,
"weight_decay": 0.01,
"max_grad_norm": 1.0,
"steps": 5000,
"seed": 42,
"warmup_type": "ratio",
"warmup_ratio": 0.01,
"warmup_steps": 0,

"attn_implementation": "sdpa",
"max_sample_tokens": 2000,
"min_sample_tokens": 50,
"max_batch_size": 64,

"batch_tokens": 8192,
"gradient_accumulation_steps": 1,
"num_workers": 2,

"mixed_precision": "bf16",
"allow_tf32": true,

"logging_steps": 50,
"eval_steps": 500,
"save_steps": 500,
"keep_last_n_checkpoints": -1

}

File: examples/config/train_config_finetune_sdpa.json Copy Chapter

{
"llm_name_or_path": "Qwen/Qwen3-0.6B",
"audio_vocab_size": 1025,
"audio_mask_id": 1024,
"num_audio_codebook": 8,

"audio_codebook_weights": [8, 8, 6, 6, 4, 4, 2, 2],
"drop_cond_ratio": 0.1,
"prompt_ratio_range": [0.0, 0.3],
"mask_ratio_range": [0.0, 1.0],
"language_ratio": 0.8,
"use_pinyin_ratio": 0.0,
"instruct_ratio": 0.0,
"only_instruct_ratio": 0.0,

"resume_from_checkpoint": null,
"init_from_checkpoint": "k2-fsa/OmniVoice",

"learning_rate": 1e-5,
"weight_decay": 0.01,
"max_grad_norm": 1.0,
"steps": 5000,
"seed": 42,
"warmup_type": "ratio",
"warmup_ratio": 0.01,
"warmup_steps": 0,

"batch_tokens": 8192,
"gradient_accumulation_steps": 1,
"num_workers": 2,

"mixed_precision": "bf16",
"allow_tf32": true,
"attn_implementation": "sdpa",
"max_sample_tokens": 2000,
"min_sample_tokens": 50,
"max_batch_size": 64,

"logging_steps": 50,
"eval_steps": 500,
"save_steps": 500,
"keep_last_n_checkpoints": -1

}

File: examples/config/train_config_multilingual.json Copy Chapter

{
"llm_name_or_path": "Qwen/Qwen3-0.6B",
"audio_vocab_size": 1025,
"audio_mask_id": 1024,
"num_audio_codebook": 8,

"audio_codebook_weights": [8, 8, 6, 6, 4, 4, 2, 2],
"drop_cond_ratio": 0.1,
"prompt_ratio_range": [0.0, 0.3],
"mask_ratio_range": [0.0, 1.0],
"language_ratio": 0.8,
"use_pinyin_ratio": 0.3,
"instruct_ratio": 1.0,
"only_instruct_ratio": 0.5,

"resume_from_checkpoint": null,
"init_from_checkpoint": null,

"learning_rate": 1e-4,
"weight_decay": 0.01,
"max_grad_norm": 1.0,
"steps": 2000000,
"seed": 42,
"warmup_type": "ratio",
"warmup_ratio": 0.03,
"warmup_steps": 0,

"batch_tokens": 8192,
"gradient_accumulation_steps": 1,
"num_workers": 4,

"mixed_precision": "bf16",
"allow_tf32": true,

"logging_steps": 100,
"eval_steps": 1000,
"save_steps": 10000,
"keep_last_n_checkpoints": -1

}

File: .github/ISSUE_TEMPLATE/bug_report.yml Copy Chapter

name: "Bug Report"
Please provide as much details to help address the issue more efficiently, including input, output, logs and screenshots.
labels:

• bug
  body:
• type: checkboxes
  attributes:
  label: Checks
  options:
  - label: This template is only for bug reports, usage problems go with 'Help Wanted'.
  required: true
  - label: I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
  required: true
  - label: I have searched for existing issues, including closed ones, and couldn't find a solution.
  required: true
  - label: I am using English to submit this issue to facilitate community communication.
  required: true
• type: textarea
  attributes:
  label: Environment Details
  placeholder: e.g., Ubuntu 20.04.6 LTS, 4 * H20, Python 3.13, torch==2.8.0+cu128, cuda 12.8
  validations:
  required: true
• type: textarea
  attributes:
  label: Steps to Reproduce
  Include detailed steps, screenshots, and logs. Use the correct markdown syntax for code blocks.
  
  placeholder: |
  1. Clone the repo and install omnivoice with uv sync.
  2. Run the command: omnivoice-infer --text "This is a test." --ref_audio ref.wav --ref_text "Transcription." --output output.wav
  3. Got the following error message... (attach full logs).
  4. Upload relevant audio files (e.g., ref.wav, output.wav) as .wav or packed in .zip.
  
  validations:
  required: true
• type: textarea
  attributes:
  label: ✔️ Expected Behavior
  placeholder: Describe in detail what you expected to happen.
  validations:
  required: false
• type: textarea
  attributes:
  label: ❌ Actual Behavior
  placeholder: Describe in detail what actually happened.
  validations:
  required: false

File: .github/ISSUE_TEMPLATE/config.yml Copy Chapter

blank_issues_enabled: false

File: .github/ISSUE_TEMPLATE/feature_request.yml Copy Chapter

name: "Feature Request"
Some constructive suggestions and new ideas regarding current repo.
labels:

• enhancement
  body:

• type: checkboxes
  attributes:
  label: Checks
  options:
  - label: This template is only for feature request.
  required: true
  - label: I have thoroughly reviewed the project documentation but couldn't find any relevant information that meets my needs.
  required: true
  - label: I have searched for existing issues, including closed ones, and found not discussion yet.
  required: true
  - label: I am using English to submit this issue to facilitate community communication.
  required: true

• type: textarea
  attributes:
  label: 1. Is this request related to a challenge you're experiencing? Tell us your story.
  Describe the specific problem or scenario you're facing in detail. For example:
  "I was trying to use [feature] for [specific task], but encountered [issue]. This was frustrating because...."
  
  placeholder: Please describe the situation in as much detail as possible.
  validations:
  required: true

• type: textarea
  attributes:
  label: 2. What is your suggested solution?
  Provide a clear description of the feature or enhancement you'd like to propose.
  How would this feature solve your issue or improve the project?
  
  placeholder: Describe your idea or proposed solution here.
  validations:
  required: true

• type: textarea
  attributes:
  label: 3. Additional context or comments
  Any other relevant information, links, documents, or screenshots that provide clarity.
  Use this section for anything not covered above.
  
  placeholder: Add any extra details here.
  validations:
  required: false

• type: checkboxes
  attributes:
  label: 4. Can you help us with this feature?
  Let us know if you're interested in contributing. This is not a commitment but a way to express interest in collaboration.
  
  options:
  - label: I am interested in contributing to this feature.
  required: false

• type: markdown
  attributes:
  value: |
  Note: Please submit only one request per issue to keep discussions focused and manageable.

File: .github/ISSUE_TEMPLATE/help_wanted.yml Copy Chapter

name: "Help Wanted"
Please provide as much details to help address the issue more efficiently, including input, output, logs and screenshots.
labels:

• help wanted
  body:
• type: checkboxes
  attributes:
  label: Checks
  options:
  - label: This template is only for usage issues encountered.
  required: true
  - label: I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
  required: true
  - label: I have searched for existing issues, including closed ones, and couldn't find a solution.
  required: true
  - label: I am using English to submit this issue to facilitate community communication.
  required: true
• type: textarea
  attributes:
  label: Environment Details
  placeholder: |
  e.g., macOS 13.5, Python 3.13, torch==2.8.0
  If training or finetuning related, provide detailed configuration including GPU info and training setup.
  
  validations:
  required: true
• type: textarea
  attributes:
  label: Steps to Reproduce
  Include detailed steps, screenshots, and logs. Provide used prompt wav and text. Use the correct markdown syntax for code blocks.
  
  placeholder: |
  1. Clone the repo and install omnivoice with uv sync.
  2. Run the command: omnivoice-infer --text "This is a test." --ref_audio ref.wav --ref_text "Transcription." --output output.wav
  3. Stuck there with the following message... (attach logs and also error msg e.g. after ctrl-c).
  4. Upload relevant audio files (e.g., ref.wav, output.wav) as .wav or packed in .zip.
  
  validations:
  required: true
• type: textarea
  attributes:
  label: ✔️ Expected Behavior
  placeholder: Describe what you expected to happen in detail, e.g. output a generated audio.
  validations:
  required: false
• type: textarea
  attributes:
  label: ❌ Actual Behavior
  placeholder: Describe what actually happened in detail, failure messages, etc.
  validations:
  required: false

File: .github/ISSUE_TEMPLATE/question.yml Copy Chapter

name: "Question"
Research question or pure inquiry about the project, usage issue goes with "help wanted".
labels:

• question
  body:
• type: checkboxes
  attributes:
  label: Checks
  options:
  - label: This template is only for research question, not usage problems, feature requests or bug reports.
  required: true
  - label: I have thoroughly reviewed the project documentation and read the related paper(s).
  required: true
  - label: I have searched for existing issues, including closed ones, no similar questions.
  required: true
  - label: I am using English to submit this issue to facilitate community communication.
  required: true
• type: textarea
  attributes:
  label: Question details
  Question details, clearly stated using proper markdown syntax.
  
  validations:
  required: true

File: .github/workflows/publish.yml Copy Chapter

name: Publish to PyPI

on:
release:
types: [published]
workflow_dispatch:

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
    with:
      python-version: "3.x"

  - name: Install build dependencies
    run: pip install hatchling build twine

  - name: Build package
    run: python -m build

  - name: Validate distribution metadata
    run: twine check dist/*

  - uses: actions/upload-artifact@v4
    with:
      name: dist
      path: dist/

publish:
needs: build
runs-on: ubuntu-latest
environment: pypi
permissions:
id-token: write
steps:
- uses: actions/download-artifact@v4
with:
name: dist
path: dist/

- uses: pypa/gh-action-pypi-publish@release/v1

File: .github/workflows/stale.yml Copy Chapter

name: Close stale issues

on:
schedule:
- cron: "0 0 * * *"
workflow_dispatch:

permissions:
issues: write
contents: read

jobs:
stale:
runs-on: ubuntu-latest
steps:
- uses: actions/stale@v9
with:
days-before-issue-stale: 45
days-before-issue-close: 30
stale-issue-label: Stale
stale-issue-message: >
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs. Thank you
for your contributions.
close-issue-message: >
This issue is closed. Please re-open if needed.
days-before-pr-stale: -1
days-before-pr-close: -1