Repository: SWivid/F5-TTS

Stars: 14341

README.md Copy Chapter

F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

<!-- <img src="https://github.com/user-attachments/assets/12d7749c-071a-427c-81bf-b87b91def670" alt="Watermark" style="width: 40px; height: auto"> -->

F5-TTS: Diffusion Transformer with ConvNeXt V2, faster trained and inference.

E2 TTS: Flat-UNet Transformer, closest reproduction from paper.

Sway Sampling: Inference-time flow step sampling strategy, greatly improves performance

Thanks to all the contributors ! Copy Section

News Copy Chapter

Installation Copy Chapter

Create a separate environment if needed Copy Section

Create a conda env with python_version>=3.10  (you could also use virtualenv)

conda create -n f5-tts python=3.11
conda activate f5-tts
Install FFmpeg if you haven't yet

conda install ffmpeg

Install PyTorch with matched device Copy Section

<details>
<summary>NVIDIA GPU</summary>

``bash

`

</details>

<details>
<summary>AMD GPU</summary>

`bash

`

</details>

<details>
<summary>Intel GPU</summary>

`bash

`

</details>

<details>
<summary>Apple Silicon</summary>

`bash

`

</details>

Then you can choose one from below: Copy Section

### 1. As a pip package (if just for inference)

> `bash

`

> `bash

``

Docker usage also available Copy Section

Build from Dockerfile

docker build -t f5tts:v1 .
Run from GitHub Container Registry

docker container run --rm -it --gpus=all --mount 'type=volume,source=f5-tts,target=/root/.cache/huggingface/hub/' -p 7860:7860 ghcr.io/swivid/f5-tts:main
Quickstart if you want to just run the web interface (not CLI)

docker container run --rm -it --gpus=all --mount 'type=volume,source=f5-tts,target=/root/.cache/huggingface/hub/' -p 7860:7860 ghcr.io/swivid/f5-tts:main f5-tts_infer-gradio --host 0.0.0.0

Runtime Copy Section

Deployment solution with Triton and TensorRT-LLM.

#### Benchmark Results
Decoding on a single L20 GPU, using 26 different prompt_audio & target_text pairs, 16 NFE.

| Model | Concurrency | Avg Latency | RTF | Mode |
|---------------------|----------------|-------------|--------|-----------------|
| F5-TTS Base (Vocos) | 2 | 253 ms | 0.0394 | Client-Server |
| F5-TTS Base (Vocos) | 1 (Batch_size) | - | 0.0402 | Offline TRT-LLM |
| F5-TTS Base (Vocos) | 1 (Batch_size) | - | 0.1467 | Offline Pytorch |

See detailed instructions for more information.

Inference Copy Chapter

- In order to achieve desired performance, take a moment to read detailed guidance.
- By properly searching the keywords of problem encountered, issues are very helpful.

1. Gradio App Copy Section

Currently supported features:

- Basic TTS with Chunk Inference
- Multi-Style / Multi-Speaker Generation
- Voice Chat powered by Qwen2.5-3B-Instruct
- Custom inference with more language support

Launch a Gradio app (web interface)

f5-tts_infer-gradio
Specify the port/host

f5-tts_infer-gradio --port 7860 --host 0.0.0.0
Launch a share link

f5-tts_infer-gradio --share

<details>
<summary>NVIDIA device docker compose file example</summary>

services:
  f5-tts:
    image: ghcr.io/swivid/f5-tts:main
    ports:
      - "7860:7860"
    environment:
      GRADIO_SERVER_PORT: 7860
    entrypoint: ["f5-tts_infer-gradio", "--port", "7860", "--host", "0.0.0.0"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
volumes:
  f5-tts:
    driver: local

</details>

2. CLI Inference Copy Section

Run with flags

Leave --ref_text "" will have ASR model transcribe (extra GPU memory usage)

f5-tts_infer-cli --model F5TTS_v1_Base \
--ref_audio "provide_prompt_wav_path_here.wav" \
--ref_text "The content, subtitle or transcription of reference audio." \
--gen_text "Some text you want TTS model generate for you."
Run with default setting. src/f5_tts/infer/examples/basic/basic.toml

f5-tts_infer-cli
Or with your own .toml file

f5-tts_infer-cli -c custom.toml
Multi voice. See src/f5_tts/infer/README.md

f5-tts_infer-cli -c src/f5_tts/infer/examples/multi/story.toml

Training Copy Chapter

1. With Hugging Face Accelerate Copy Section

Refer to training & finetuning guidance for best practice.

2. With Gradio App Copy Section

Quick start with Gradio web interface

f5-tts_finetune-gradio

Read training & finetuning guidance for more instructions.

Evaluation Evaluation')" title="Copy chapter prompt for LLMs"> Copy Chapter

Development Copy Chapter

Use pre-commit to ensure code quality (will run linters and formatters automatically):

pip install pre-commit
pre-commit install

When making a pull request, before each commit, run:

pre-commit run --all-files

Note: Some model components have linting exceptions for E722 to accommodate tensor notation.

Acknowledgements Copy Chapter

- E2-TTS brilliant work, simple and effective
- Emilia, WenetSpeech4TTS, LibriTTS, LJSpeech valuable datasets
- lucidrains initial CFM structure with also bfs18 for discussion
- SD3 & Hugging Face diffusers DiT and MMDiT code structure
- torchdiffeq as ODE solver, Vocos and BigVGAN as vocoder
- FunASR, faster-whisper, UniSpeech, SpeechMOS for evaluation tools
- ctc-forced-aligner for speech edit test
- mrfakename huggingface space demo ~
- f5-tts-mlx Implementation with MLX framework by Lucas Newman
- F5-TTS-ONNX ONNX Runtime version by DakeQQ
- Yuekai Zhang Triton and TensorRT-LLM support ~

Citation Copy Chapter

@article{chen-etal-2024-f5tts,
      title={F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching}, 
      author={Yushen Chen and Zhikang Niu and Ziyang Ma and Keqi Deng and Chunhui Wang and Jian Zhao and Kai Yu and Xie Chen},
      journal={arXiv preprint arXiv:2410.06885},
      year={2024},
}

License Copy Chapter

Our code is released under MIT License. The pre-trained models are licensed under the CC-BY-NC license due to the training data Emilia, which is an in-the-wild dataset. Sorry for any inconvenience this may cause.