F5-TTS Documentation

Official code for "F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching"

15,411 stars Python Python 1 file · ~6,761 tokens
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy F5-TTS on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name f5-tts -p 8000:8000 --restart always soar97/triton-f5-tts:24.12
Configured Services & Ports
⚙️ tts
image: soar97/triton-f5-tts:24.12
🔌 Ports: 8000:8000, 8001:8001, 8002:8002
docker-compose.yml
yaml (docker-compose.yml) 
services:
  tts:
    image: soar97/triton-f5-tts:24.12
    shm_size: '1gb'
    ports:
      - "8000:8000"
      - "8001:8001"
      - "8002:8002"
    environment:
      - PYTHONIOENCODING=utf-8
      - MODEL_ID=${MODEL_ID}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    command: >
      /bin/bash -c "pip install vocos && rm -rf F5-TTS && git clone https://github.com/SWivid/F5-TTS.git && cd F5-TTS/src/f5_tts/runtime/triton_trtllm/ && bash run.sh 0 4 $MODEL"

Technical Documentation: SWivid/F5-TTS

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching








F5-TTS: Diffusion Transformer with ConvNeXt V2, faster trained and inference.

E2 TTS: Flat-UNet Transformer, closest reproduction from paper.

Sway Sampling: Inference-time flow step sampling strategy, greatly improves performance

Thanks to all the contributors !

News

Installation

Create a separate environment if needed

bash
# Create a conda env with python_version>=3.10  (you could also use virtualenv)
conda create -n f5-tts python=3.11
conda activate f5-tts

# Install FFmpeg if you haven't yet
conda install ffmpeg

Install PyTorch with matched device

NVIDIA GPU

bash
# Install pytorch with your CUDA version, e.g.
pip install torch==2.8.0+cu128 torchaudio==2.8.0+cu128 --extra-index-url https://download.pytorch.org/whl/cu128

# And also possible previous versions, e.g.
pip install torch==2.4.0+cu124 torchaudio==2.4.0+cu124 --extra-index-url https://download.pytorch.org/whl/cu124
# etc.

AMD GPU

bash
# Install pytorch with your ROCm version (Linux only), e.g.
pip install torch==2.9.1+rocm7.2 torchaudio==2.9.1+rocm7.2 --extra-index-url https://download.pytorch.org/whl/rocm7.2

# For older GPUs (RDNA1/2/3 only):
# pip install torch==2.5.1+rocm6.2 torchaudio==2.5.1+rocm6.2 --extra-index-url https://download.pytorch.org/whl/rocm6.2

Note: RDNA 3.5 and RDNA 4 GPUs (Radeon 8050S/8060S, RX 9060/9070 series) require
ROCm 7.x — these architectures (gfx1151/gfx1201) are not included in ROCm 6.x
(6.2 compatibility matrix vs
7.2 compatibility matrix).
Using ROCm 6.x on these GPUs causes HIP error: invalid device function (#1236).

Intel GPU

bash
# Install pytorch with your XPU version, e.g.
# Intel® Deep Learning Essentials or Intel® oneAPI Base Toolkit must be installed
pip install torch torchaudio --index-url https://download.pytorch.org/whl/test/xpu

# Intel GPU support is also available through IPEX (Intel® Extension for PyTorch)
# IPEX does not require the Intel® Deep Learning Essentials or Intel® oneAPI Base Toolkit
# See: https://pytorch-extension.intel.com/installation?request=platform

Apple Silicon

bash
# Install the stable pytorch, e.g.
pip install torch torchaudio

Then you can choose one from below:

1. As a pip package (if just for inference)

bash
pip install f5-tts

2. Local editable (if also do training, finetuning)

bash
git clone https://github.com/SWivid/F5-TTS.git
cd F5-TTS
# git submodule update --init --recursive  # (optional, if use bigvgan as vocoder)
pip install -e .

Docker usage also available

bash
# Build from Dockerfile
docker build -t f5tts:v1 .

# Run from GitHub Container Registry
docker container run --rm -it --gpus=all --mount 'type=volume,source=f5-tts,target=/root/.cache/huggingface/hub/' -p 7860:7860 ghcr.io/swivid/f5-tts:main

# Quickstart if you want to just run the web interface (not CLI)
docker container run --rm -it --gpus=all --mount 'type=volume,source=f5-tts,target=/root/.cache/huggingface/hub/' -p 7860:7860 ghcr.io/swivid/f5-tts:main f5-tts_infer-gradio --host 0.0.0.0

Runtime

Deployment solution with Triton and TensorRT-LLM.

Benchmark Results

Decoding on a single L20 GPU, using 26 different prompt_audio & target_text pairs, 16 NFE.

Model Concurrency Avg Latency RTF Mode
F5-TTS Base (Vocos) 2 253 ms 0.0394 Client-Server
F5-TTS Base (Vocos) 1 (Batch_size) - 0.0402 Offline TRT-LLM
F5-TTS Base (Vocos) 1 (Batch_size) - 0.1467 Offline Pytorch

See detailed instructions for more information.

Inference

  • In order to achieve desired performance, take a moment to read detailed guidance.
  • By properly searching the keywords of problem encountered, issues are very helpful.

1. Gradio App

Currently supported features:

bash
# Launch a Gradio app (web interface)
f5-tts_infer-gradio

# Specify the port/host
f5-tts_infer-gradio --port 7860 --host 0.0.0.0

# Launch a share link
f5-tts_infer-gradio --share

NVIDIA device docker compose file example

yaml
services:
  f5-tts:
    image: ghcr.io/swivid/f5-tts:main
    ports:
      - "7860:7860"
    environment:
      GRADIO_SERVER_PORT: 7860
    entrypoint: ["f5-tts_infer-gradio", "--port", "7860", "--host", "0.0.0.0"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  f5-tts:
    driver: local

2. CLI Inference

bash
# Run with flags
# Leave --ref_text "" will have ASR model transcribe (extra GPU memory usage)
f5-tts_infer-cli --model F5TTS_v1_Base \
--ref_audio "provide_prompt_wav_path_here.wav" \
--ref_text "The content, subtitle or transcription of reference audio." \
--gen_text "Some text you want TTS model generate for you."

# Run with default setting. src/f5_tts/infer/examples/basic/basic.toml
f5-tts_infer-cli
# Or with your own .toml file
f5-tts_infer-cli -c custom.toml

# Multi voice. See src/f5_tts/infer/README.md
f5-tts_infer-cli -c src/f5_tts/infer/examples/multi/story.toml

Training

1. With Hugging Face Accelerate

Refer to training & finetuning guidance for best practice.

2. With Gradio App

bash
# Quick start with Gradio web interface
f5-tts_finetune-gradio

Read training & finetuning guidance for more instructions.

[Evaluation](src/f5_tts/eval)

Development

Use pre-commit to ensure code quality (will run linters and formatters automatically):

bash
pip install pre-commit
pre-commit install

When making a pull request, before each commit, run:

bash
pre-commit run --all-files

Note: Some model components have linting exceptions for E722 to accommodate tensor notation.

Acknowledgements

Citation

If our work and codebase is useful for you, please cite as:

text
@article{chen-etal-2024-f5tts,
      title={F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching}, 
      author={Yushen Chen and Zhikang Niu and Ziyang Ma and Keqi Deng and Chunhui Wang and Jian Zhao and Kai Yu and Xie Chen},
      journal={arXiv preprint arXiv:2410.06885},
      year={2024},
}

License

Our code is released under MIT License. The pre-trained models are licensed under the CC-BY-NC license due to the training data Emilia, which is an in-the-wild dataset. Sorry for any inconvenience this may cause.

Overview (`ckpts/README.md`)

The pretrained model checkpoints can be reached at https://huggingface.co/SWivid/F5-TTS.

Scripts will automatically pull model checkpoints from Huggingface, by default to ~/.cache/huggingface/hub/.

Overview (`src/f5_tts/eval/README.md`)

Evaluation

Install packages for evaluation:

bash
pip install -e .[eval]

[!IMPORTANT]
For faster-whisper, for various compatibilities:
pip install ctranslate2==4.5.0 if CUDA 12 and cuDNN 9;
pip install ctranslate2==4.4.0 if CUDA 12 and cuDNN 8;
pip install ctranslate2==3.24.0 if CUDA 11 and cuDNN 8.

Generating Samples for Evaluation

Prepare Test Datasets

  1. Seed-TTS testset: Download from seed-tts-eval.
  2. LibriSpeech test-clean: Download from OpenSLR.
  3. Unzip the downloaded datasets and place them in the data/ directory.
  4. Our filtered LibriSpeech-PC 4-10s subset: data/librispeech_pc_test_clean_cross_sentence.lst

Batch Inference for Test Set

To run batch inference for evaluations, execute the following commands:

bash
# if not setup accelerate config yet
accelerate config

# if only perform inference
bash src/f5_tts/eval/eval_infer_batch.sh --infer-only

# if inference and with corresponding evaluation, setup the following tools first
bash src/f5_tts/eval/eval_infer_batch.sh

Objective Evaluation on Generated Results

Download Evaluation Model Checkpoints

  1. Chinese ASR Model: Paraformer-zh
  2. English ASR Model: Faster-Whisper
  3. WavLM Model: Download from Google Drive.

[!NOTE]
ASR model will be automatically downloaded if --local not set for evaluation scripts.
Otherwise, you should update the asr_ckpt_dir path values in eval_librispeech_test_clean.py or eval_seedtts_testset.py.

WavLM model must be downloaded and your wavlm_ckpt_dir path updated in eval_librispeech_test_clean.py and eval_seedtts_testset.py.

Objective Evaluation Examples

Update the path with your batch-inferenced results, and carry out WER / SIM / UTMOS evaluations:

bash
# Evaluation [WER] for Seed-TTS test [ZH] set
python src/f5_tts/eval/eval_seedtts_testset.py --eval_task wer --lang zh --gen_wav_dir <GEN_WAV_DIR> --gpu_nums 8

# Evaluation [SIM] for LibriSpeech-PC test-clean (cross-sentence)
python src/f5_tts/eval/eval_librispeech_test_clean.py --eval_task sim --gen_wav_dir <GEN_WAV_DIR> --librispeech_test_clean_path <TEST_CLEAN_PATH>

# Evaluation [UTMOS]. --ext: Audio extension
python src/f5_tts/eval/eval_utmos.py --audio_dir <WAV_DIR> --ext wav

[!NOTE]
Evaluation results can also be found in _*_results.jsonl files saved in <GEN_WAV_DIR>/<WAV_DIR>.

Overview (`src/f5_tts/infer/README.md`)

Inference

The pretrained model checkpoints can be reached at 🤗 Hugging Face and 🤖 Model Scope, or will be automatically downloaded when running inference scripts.

More checkpoints with whole community efforts can be found in SHARED.md, supporting more languages.

Currently support 30s for a single generation, which is the total length (same logic if fix_duration) including both prompt and output audio. However, infer_cli and infer_gradio will automatically do chunk generation for longer text. Long reference audio will be clip short to ~12s.

To avoid possible inference failures, make sure you have seen through the following instructions.

  • Use reference audio <12s and leave proper silence space (e.g. 1s) at the end. Otherwise there is a risk of truncating in the middle of word, leading to suboptimal generation.
  • Uppercased letters (best with form like K.F.C.) will be uttered letter by letter, and lowercased letters used for common words.
  • Add some spaces (blank: " ") or punctuations (e.g. "," ".") to explicitly introduce some pauses.
  • If English punctuation marks the end of a sentence, make sure there is a space " " after it. Otherwise not regarded as when chunk.
  • Preprocess numbers to Chinese letters if you want to have them read in Chinese, otherwise in English.
  • If the generation output is blank (pure silence), check for FFmpeg installation.
  • Try turn off use_ema if using an early-stage finetuned checkpoint (which goes just few updates).

Gradio App

Currently supported features:

The cli command f5-tts_infer-gradio equals to python src/f5_tts/infer/infer_gradio.py, which launches a Gradio APP (web interface) for inference.

The script will load model checkpoints from Huggingface. You can also manually download files and update the path to load_model() in infer_gradio.py. Currently only load TTS models first, will load ASR model to do transcription if ref_text not provided, will load LLM model if use Voice Chat.

More flags options:

bash
# Automatically launch the interface in the default web browser
f5-tts_infer-gradio --inbrowser

# Set the root path of the application, if it's not served from the root ("/") of the domain
# For example, if the application is served at "https://example.com/myapp"
f5-tts_infer-gradio --root_path "/myapp"

Could also be used as a component for larger application:

python
import gradio as gr
from f5_tts.infer.infer_gradio import app

with gr.Blocks() as main_app:
    gr.Markdown("# This is an example of using F5-TTS within a bigger Gradio app")

    # ... other Gradio components

    app.render()

main_app.launch()

CLI Inference

The cli command f5-tts_infer-cli equals to python src/f5_tts/infer/infer_cli.py, which is a command line tool for inference.

The script will load model checkpoints from Huggingface. You can also manually download files and use --ckpt_file to specify the model you want to load, or directly update in infer_cli.py.

For change vocab.txt use --vocab_file to provide your vocab.txt file.

Basically you can inference with flags:

bash
# Leave --ref_text "" will have ASR model transcribe (extra GPU memory usage)
f5-tts_infer-cli \
--model F5TTS_v1_Base \
--ref_audio "ref_audio.wav" \
--ref_text "The content, subtitle or transcription of reference audio." \
--gen_text "Some text you want TTS model generate for you."

# Use BigVGAN as vocoder. Currently only support F5TTS_Base. 
f5-tts_infer-cli --model F5TTS_Base --vocoder_name bigvgan --load_vocoder_from_local

# Use custom path checkpoint, e.g.
f5-tts_infer-cli --ckpt_file ckpts/F5TTS_v1_Base/model_1250000.safetensors

# More instructions
f5-tts_infer-cli --help

And a .toml file would help with more flexible usage.

bash
f5-tts_infer-cli -c custom.toml

For example, you can use .toml to pass in variables, refer to src/f5_tts/infer/examples/basic/basic.toml:

toml
# F5TTS_v1_Base | E2TTS_Base
model = "F5TTS_v1_Base"
ref_audio = "infer/examples/basic/basic_ref_en.wav"
# If an empty "", transcribes the reference audio automatically.
ref_text = "Some call me nature, others call me mother nature."
gen_text = "I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring."
# File with text to generate. Ignores the text above.
gen_file = ""
remove_silence = false
output_dir = "tests"

You can also leverage .toml file to do multi-style generation, refer to src/f5_tts/infer/examples/multi/story.toml.

toml
# F5TTS_v1_Base | E2TTS_Base
model = "F5TTS_v1_Base"
ref_audio = "infer/examples/multi/main.flac"
# If an empty "", transcribes the reference audio automatically.
ref_text = ""
gen_text = ""
# File with text to generate. Ignores the text above.
gen_file = "infer/examples/multi/story.txt"
remove_silence = true
output_dir = "tests"

[voices.town]
ref_audio = "infer/examples/multi/town.flac"
ref_text = ""

[voices.country]
ref_audio = "infer/examples/multi/country.flac"
ref_text = ""

You should mark the voice with [main] [town] [country] whenever you want to change voice, refer to src/f5_tts/infer/examples/multi/story.txt.

API Usage

python
from importlib.resources import files
from f5_tts.api import F5TTS

f5tts = F5TTS()
wav, sr, spec = f5tts.infer(
    ref_file=str(files("f5_tts").joinpath("infer/examples/basic/basic_ref_en.wav")),
    ref_text="some call me nature, others call me mother nature.",
    gen_text="""I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences.""",
    file_wave=str(files("f5_tts").joinpath("../../tests/api_out.wav")),
    file_spec=str(files("f5_tts").joinpath("../../tests/api_out.png")),
    seed=None,
)

Check api.py for more details.

TensorRT-LLM Deployment

See detailed instructions for more information.

Socket Real-time Service

Real-time voice output with chunk stream:

bash
# Start socket server
python src/f5_tts/socket_server.py

# If PyAudio not installed
sudo apt-get install portaudio19-dev
pip install pyaudio

# Communicate with socket client
python src/f5_tts/socket_client.py

Speech Editing

To test speech editing capabilities, use the following command:

bash
python src/f5_tts/infer/speech_edit.py

Overview (`src/f5_tts/train/README.md`)

Training

Check your FFmpeg installation:

bash
ffmpeg -version

If not found, install it first (or skip assuming you know of other backends available).

Prepare Dataset

Example data processing scripts, and you may tailor your own one along with a Dataset class in src/f5_tts/model/dataset.py.

1. Some specific Datasets preparing scripts

Download corresponding dataset first, and fill in the path in scripts.

bash
# Prepare the Emilia dataset
python src/f5_tts/train/datasets/prepare_emilia.py

# Prepare the Wenetspeech4TTS dataset
python src/f5_tts/train/datasets/prepare_wenetspeech4tts.py

# Prepare the LibriTTS dataset
python src/f5_tts/train/datasets/prepare_libritts.py

# Prepare the LJSpeech dataset
python src/f5_tts/train/datasets/prepare_ljspeech.py

2. Create custom dataset with CSV

Prepare a CSV with two columns using a required header: audio_file|text. Audio paths must be absolute.
Use guidance see #57 here.

bash
python src/f5_tts/train/datasets/prepare_csv_wavs.py /path/to/metadata.csv /path/to/output

Training & Finetuning

Once your datasets are prepared, you can start the training process.

1. Training script used for pretrained model

bash
# setup accelerate config, e.g. use multi-gpu ddp, fp16
# will be to: ~/.cache/huggingface/accelerate/default_config.yaml     
accelerate config

# .yaml files are under src/f5_tts/configs directory
accelerate launch src/f5_tts/train/train.py --config-name F5TTS_v1_Base.yaml

# possible to overwrite accelerate and hydra config
accelerate launch --mixed_precision=fp16 src/f5_tts/train/train.py --config-name F5TTS_v1_Base.yaml ++datasets.batch_size_per_gpu=19200

2. Finetuning practice

Discussion board for Finetuning #57.

Gradio UI training/finetuning with src/f5_tts/train/finetune_gradio.py see #143.

If want to finetune with a variant version e.g. F5TTS_v1_Base_no_zero_init, manually download pretrained checkpoint from model weight repository and fill in the path correspondingly on web interface.

If use tensorboard as logger, install it first with pip install tensorboard.

The use_ema = True might be harmful for early-stage finetuned checkpoints (which goes just few updates, thus ema weights still dominated by pretrained ones), try turn it off with finetune gradio option or load_model(..., use_ema=False), see if offer better results.

3. W&B Logging

The wandb/ dir will be created under path you run training/finetuning scripts.

By default, the training script does NOT use logging (assuming you didn't manually log in using wandb login).

To turn on wandb logging, you can either:

  1. Manually login with wandb login: Learn more here
  2. Automatically login programmatically by setting an environment variable: Get an API KEY at https://wandb.ai/authorize and set the environment variable as follows:

On Mac & Linux:

text
export WANDB_API_KEY=<YOUR WANDB API KEY>

On Windows:

text
set WANDB_API_KEY=<YOUR WANDB API KEY>

Moreover, if you couldn't access W&B and want to log metrics offline, you can set the environment variable as follows:

text
export WANDB_MODE=offline

Overview (`src/f5_tts/model/backbones/README.md`)

Backbones quick introduction

unett.py

  • flat unet transformer
  • structure same as in e2-tts & voicebox paper except using rotary pos emb
  • possible abs pos emb & convnextv2 blocks for embedded text before concat

dit.py

  • adaln-zero dit
  • embedded timestep as condition
  • concatted noised_input + masked_cond + embedded_text, linear proj in
  • possible abs pos emb & convnextv2 blocks for embedded text before concat
  • possible long skip connection (first layer to last layer)

mmdit.py

  • stable diffusion 3 block structure
  • timestep as condition
  • left stream: text embedded and applied a abs pos emb
  • right stream: masked_cond & noised_input concatted and with same conv pos emb as unett

Overview (`src/f5_tts/runtime/triton_trtllm/README.md`)

Triton Inference Serving Best Practice for F5-TTS

Setup

Option 1: Quick Start

sh
# Directly launch the service using docker compose
MODEL=F5TTS_v1_Base docker compose up

Option 2: Build from scratch

sh
# Build the docker image
docker build . -f Dockerfile.server -t soar97/triton-f5-tts:24.12

# Create Docker Container
your_mount_dir=/mnt:/mnt
docker run -it --name "f5-server" --gpus all --net host -v $your_mount_dir --shm-size=2g soar97/triton-f5-tts:24.12

Build TensorRT-LLM Engines and Launch Server

Inside docker container, we would follow the official guide of TensorRT-LLM to build qwen and whisper TensorRT-LLM engines. See here.

sh
# F5TTS_v1_Base | F5TTS_Base | F5TTS_v1_Small | F5TTS_Small
bash run.sh 0 4 F5TTS_v1_Base

[!NOTE]
If use custom checkpoint, set ckpt_file and vocab_file in run.sh.
Remember to used matched model version (F5TTS_v1_* for v1, F5TTS_* for v0).

If use checkpoint of different structure, see scripts/convert_checkpoint.py, and perform modification if necessary.

[!IMPORTANT]
If train or finetune with fp32, add --dtype float32 flag when converting checkpoint in run.sh phase 1.

HTTP Client

sh
python3 client_http.py

Benchmarking

Using Client-Server Mode

sh
# bash run.sh 5 5 F5TTS_v1_Base
num_task=2
python3 client_grpc.py --num-tasks $num_task --huggingface-dataset yuekai/seed_tts --split-name wenetspeech4tts

Using Offline TRT-LLM Mode

sh
# bash run.sh 7 7 F5TTS_v1_Base
batch_size=1
split_name=wenetspeech4tts
backend_type=trt
log_dir=./tests/benchmark_batch_size_${batch_size}_${split_name}_${backend_type}
rm -r $log_dir
torchrun --nproc_per_node=1 \
benchmark.py --output-dir $log_dir \
--batch-size $batch_size \
--enable-warmup \
--split-name $split_name \
--model-path $ckpt_file \
--vocab-file $vocab_file \
--vocoder-trt-engine-path $VOCODER_TRT_ENGINE_PATH \
--backend-type $backend_type \
--tllm-model-dir $TRTLLM_ENGINE_DIR || exit 1

Benchmark Results

Decoding on a single L20 GPU, using 26 different prompt_audio & target_text pairs, 16 NFE.

Model Concurrency Avg Latency RTF Mode
F5-TTS Base (Vocos) 2 253 ms 0.0394 Client-Server
F5-TTS Base (Vocos) 1 (Batch_size) - 0.0402 Offline TRT-LLM
F5-TTS Base (Vocos) 1 (Batch_size) - 0.1467 Offline Pytorch

Credits

  1. Yuekai Zhang
  2. F5-TTS-TRTLLM

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →