fish-speech Documentation

1. Project Overview & Quickstart (fishaudio/fish-speech) Copy Chapter

File: README.md Copy Chapter

[!IMPORTANT]
License Notice
This codebase and its associated model weights are released under FISH AUDIO RESEARCH LICENSE. Please refer to LICENSE for more details. We will take action against any violation of the license.

[!WARNING]
Legal Disclaimer
We do not hold any responsibility for any illegal usage of the codebase. Please refer to your local laws about DMCA and other related laws.

Quick Start Copy Chapter

For Human Copy Section

Here are the official documents for Fish Audio S2, follow the instructions to get started easily.

• Installation
• Command Line Inference
• WebUI Inference
• Server Inference
• Docker Setup

[!IMPORTANT]
For SGLang server, please read SGLang-Omni README.

For vLLM Omni server, please read vLLM-Omni Fish Speech S2 Pro Recipe and the User Guide.

For LLM Agent Copy Section

Install and configure Fish-Audio S2 by following the instructions here: https://speech.fish.audio/install/

Fish Audio S2 Pro Copy Chapter

State-of-the-art multilingual text-to-speech (TTS) system, redefining the boundaries of voice generation.

Fish Audio S2 Pro is the most advanced multimodal model developed by Fish Audio. Trained on over 10 million hours of audio data covering more than 80 languages, S2 Pro combines a Dual-Autoregressive (Dual-AR) architecture with reinforcement learning (RL) alignment to generate speech that is exceptionally natural, realistic, and emotionally rich, leading the competition among both open-source and closed-source systems.

The core strength of S2 Pro lies in its support for sub-word level fine-grained control of prosody and emotion using natural language tags (e.g., [whisper], [excited], [angry]), while natively supporting multi-speaker and multi-turn conversation generation.

Visit the Fish Audio website for a live playground, or read our technical report and blog post for more details.

Model Variants Copy Section

More details of the model can be found in the technical report.

Benchmark Results Copy Chapter

On Seed-TTS Eval, S2 achieves the lowest WER among all evaluated models including closed-source systems: Qwen3-TTS (0.77/1.24), MiniMax Speech-02 (0.99/1.90), Seed-TTS (1.12/2.25). On the Audio Turing Test, 0.515 surpasses Seed-TTS (0.417) by 24% and MiniMax-Speech (0.387) by 33%. On EmergentTTS-Eval, S2 achieves particularly strong results in paralinguistics (91.61% win rate), questions (84.41%), and syntactic complexity (83.39%).

Highlights Copy Chapter

Fine-Grained Inline Control via Natural Language Copy Section

S2 Pro brings unprecedented "soul" to speech. Using simple [tag] syntax, you can precisely embed emotional instructions at any position in the text.

• 15,000+ Unique Tags Supported: Not limited to fixed presets; S2 supports free-form text descriptions. Try [whisper in small voice], [professional broadcast tone], or [pitch up].
• Rich Emotion Library:
  [pause] [emphasis] [laughing] [inhale] [chuckle] [tsk] [singing] [excited] [laughing tone] [interrupting] [chuckling] [excited tone] [volume up] [echo] [angry] [low volume] [sigh] [low voice] [whisper] [screaming] [shouting] [loud] [surprised] [short pause] [exhale] [delight] [panting] [audience laughter] [with strong accent] [volume down] [clearing throat] [sad] [moaning] [shocked]

Innovative Dual-Autoregressive (Dual-AR) Architecture Copy Section

S2 Pro adopts a master-slave Dual-AR architecture consisting of a decoder-only transformer and an RVQ audio codec (10 codebooks, ~21 Hz):

• Slow AR (4B parameters): Operates along the time axis, predicting the primary semantic codebook.
• Fast AR (400M parameters): Generates the remaining 9 residual codebooks at each time step, reconstructing exquisite acoustic details.

This asymmetric design achieves peak audio fidelity while significantly boosting inference speed.

Reinforcement Learning (RL) Alignment Copy Section

S2 Pro utilizes Group Relative Policy Optimization (GRPO) for post-training alignment. We use the same model suite for data cleaning and annotation directly as Reward Models, perfectly resolving the distribution mismatch between pre-training data and post-training objectives.

• Multi-Dimensional Reward Signals: Comprehensively evaluates semantic accuracy, instruction adherence, acoustic preference scoring, and timbre similarity to ensure every second of generated speech feels intuitive to humans.

Extreme Streaming Performance (Powered by SGLang) Copy Section

As the Dual-AR architecture is structurally isomorphic to standard LLMs, S2 Pro natively supports all SGLang inference acceleration features, including Continuous Batching, Paged KV Cache, CUDA Graph, and RadixAttention-based Prefix Caching.

Performance on a single NVIDIA H200 GPU:

• Real-Time Factor (RTF): 0.195
• Time-to-First-Audio (TTFA): ~100 ms
• Extreme Throughput: 3,000+ acoustic tokens/s while maintaining RTF < 0.5

Robust Multilingual Support Copy Section

S2 Pro supports over 80 languages without requiring phonemes or language-specific preprocessing:

• Tier 1: Japanese (ja), English (en), Chinese (zh)
• Tier 2: Korean (ko), Spanish (es), Portuguese (pt), Arabic (ar), Russian (ru), French (fr), German (de)
• Global Coverage: sv, it, tr, no, nl, cy, eu, ca, da, gl, ta, hu, fi, pl, et, hi, la, ur, th, vi, jw, bn, yo, xsl, cs, sw, nn, he, ms, uk, id, kk, bg, lv, my, tl, sk, ne, fa, af, el, bo, hr, ro, sn, mi, yi, am, be, km, is, az, sd, br, sq, ps, mn, ht, ml, sr, sa, te, ka, bs, pa, lt, kn, si, hy, mr, as, gu, fo, etc.

Native Multi-Speaker Generation Copy Section

Fish Audio S2 allows users to upload reference audio containing multiple speakers, and the model processes each speaker's features via the <|speaker:i|> token. You can then control the model's performance via speaker ID tokens, enabling a single generation to include multiple speakers. There is no longer a need to upload separate reference audio for each individual speaker.

Multi-Turn Generation Copy Section

Thanks to the expansion of the model context, our model can now leverage previous information to improve the expressiveness of subsequent generated content, thereby increasing the naturalness of the dialogue.

Rapid Voice Cloning Copy Section

Fish Audio S2 supports accurate voice cloning using short reference samples (typically 10-30 seconds). The model captures timbre, speaking style, and emotional tendencies, producing realistic and consistent cloned voices without additional fine-tuning.
For SGLang Server usage, please refer to the SGLang-Omni README.

Credits Copy Chapter

• VITS2 (daniilrobnikov)
• Bert-VITS2
• GPT VITS
• MQTTS
• GPT Fast
• GPT-SoVITS
• Qwen3

Tech Report Copy Chapter

@misc{fish-speech-v1.4,
      title={Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis},
      author={Shijia Liao and Yuxuan Wang and Tianyu Li and Yifan Cheng and Ruoyi Zhang and Rongzhi Zhou and Yijin Xing},
      year={2024},
      eprint={2411.01156},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2411.01156},
}

@misc{liao2026fishaudios2technical,
      title={Fish Audio S2 Technical Report}, 
      author={Shijia Liao and Yuxuan Wang and Songting Liu and Yifan Cheng and Ruoyi Zhang and Tianyu Li and Shidong Li and Yisheng Zheng and Xingwei Liu and Qingzheng Wang and Zhizhuo Zhou and Jiahua Liu and Xin Chen and Dawei Han},
      year={2026},
      eprint={2603.08823},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2603.08823}, 
}

File: docs/en/finetune.md Copy Chapter

Fine-tuning

!!! warning
We highly do note recoomand users to do fine-tuning on an RL trained model. Fine-tuning a model after RL can shift the model distribution, which may lead to degraded performance.

In the current version, you only need to finetune the 'LLAMA' part.

Fine-tuning LLAMA Copy Chapter

1. Prepare the dataset Copy Section

.
├── SPK1
│   ├── 21.15-26.44.lab
│   ├── 21.15-26.44.mp3
│   ├── 27.51-29.98.lab
│   ├── 27.51-29.98.mp3
│   ├── 30.1-32.71.lab
│   └── 30.1-32.71.mp3
└── SPK2
    ├── 38.79-40.85.lab
    └── 38.79-40.85.mp3

You need to convert your dataset into the above format and place it under data. The audio file can have the extensions .mp3, .wav, or .flac, and the annotation file should have the extension .lab.

!!! info
The .lab annotation file only needs to contain the transcription of the audio, with no special formatting required. For example, if hi.mp3 says "Hello, goodbye," then the hi.lab file would contain a single line of text: "Hello, goodbye."

!!! warning
It's recommended to apply loudness normalization to the dataset. You can use fish-audio-preprocess to do this.

```bash
fap loudness-norm data-raw data --clean
```

2. Batch extraction of semantic tokens Copy Section

Make sure you have downloaded the VQGAN weights. If not, run the following command:

huggingface-cli download fishaudio/openaudio-s1-mini --local-dir checkpoints/openaudio-s1-mini

You can then run the following command to extract semantic tokens:

python tools/vqgan/extract_vq.py data \
    --num-workers 1 --batch-size 16 \
    --config-name "modded_dac_vq" \
    --checkpoint-path "checkpoints/openaudio-s1-mini/codec.pth"

!!! note
You can adjust --num-workers and --batch-size to increase extraction speed, but please make sure not to exceed your GPU memory limit.

This command will create .npy files in the data directory, as shown below:

.
├── SPK1
│   ├── 21.15-26.44.lab
│   ├── 21.15-26.44.mp3
│   ├── 21.15-26.44.npy
│   ├── 27.51-29.98.lab
│   ├── 27.51-29.98.mp3
│   ├── 27.51-29.98.npy
│   ├── 30.1-32.71.lab
│   ├── 30.1-32.71.mp3
│   └── 30.1-32.71.npy
└── SPK2
    ├── 38.79-40.85.lab
    ├── 38.79-40.85.mp3
    └── 38.79-40.85.npy

3. Pack the dataset into protobuf Copy Section

python tools/llama/build_dataset.py \
    --input "data" \
    --output "data/protos" \
    --text-extension .lab \
    --num-workers 16

After the command finishes executing, you should see the protos file in the data directory.

4. Finally, fine-tuning with LoRA Copy Section

Similarly, make sure you have downloaded the LLAMA weights. If not, run the following command:

huggingface-cli download fishaudio/openaudio-s1-mini --local-dir checkpoints/openaudio-s1-mini

Finally, you can start the fine-tuning by running the following command:

python fish_speech/train.py --config-name text2semantic_finetune \
    project=$project \
    [email protected]_config=r_8_alpha_16

!!! note
You can modify the training parameters such as batch_size, gradient_accumulation_steps, etc. to fit your GPU memory by modifying fish_speech/configs/text2semantic_finetune.yaml.

!!! note
For Windows users, you can use trainer.strategy.process_group_backend=gloo to avoid nccl issues.

After training is complete, you can refer to the inference section to test your model.

!!! info
By default, the model will only learn the speaker's speech patterns and not the timbre. You still need to use prompts to ensure timbre stability.
If you want to learn the timbre, you can increase the number of training steps, but this may lead to overfitting.

After training, you need to convert the LoRA weights to regular weights before performing inference.

python tools/llama/merge_lora.py \
	--lora-config r_8_alpha_16 \
	--base-weight checkpoints/openaudio-s1-mini \
	--lora-weight results/$project/checkpoints/step_000000010.ckpt \
	--output checkpoints/openaudio-s1-mini-yth-lora/

!!! note
You may also try other checkpoints. We suggest using the earliest checkpoint that meets your requirements, as they often perform better on out-of-distribution (OOD) data.

File: docs/en/index.md Copy Chapter

!!! info "License Notice"
This codebase and its associated model weights are released under FISH AUDIO RESEARCH LICENSE. Please refer to LICENSE for more details. We will take action against any violation of the license.

!!! warning "Legal Disclaimer"
We do not hold any responsibility for any illegal usage of the codebase. Please refer to your local laws about DMCA and other related laws.

Quick Start Copy Chapter

For Human Copy Section

Here are the official documents for Fish Audio S2, follow the instructions to get started easily.

• Installation
• Command Line Inference
• WebUI Inference
• Server Inference
• Docker Setup

[!IMPORTANT]
For SGLang server, please read SGLang-Omni README.

For LLM Agent Copy Section

Install and configure Fish-Audio S2 by following the instructions here: https://speech.fish.audio/install/

Fish Audio S2 Copy Chapter

Best text-to-speech system among both open source and closed source

Fish Audio S2 is the latest model developed by Fish Audio. Trained on over 10 million hours of audio across approximately 50 languages, S2 combines reinforcement learning alignment with a Dual-Autoregressive architecture to generate speech that sounds natural, realistic, and emotionally rich.

S2 supports fine-grained inline control of prosody and emotion using natural-language tags like [laugh], [whispers], and [super happy], as well as native multi-speaker and multi-turn generation.

Visit the Fish Audio website for live playground. Read the blog post and technical report for more details.

Model Variants Copy Section

More details of the model can be found in the technical report.

Benchmark Results Copy Chapter

On Seed-TTS Eval, S2 achieves the lowest WER among all evaluated models including closed-source systems: Qwen3-TTS (0.77/1.24), MiniMax Speech-02 (0.99/1.90), Seed-TTS (1.12/2.25). On the Audio Turing Test, 0.515 surpasses Seed-TTS (0.417) by 24% and MiniMax-Speech (0.387) by 33%. On EmergentTTS-Eval, S2 achieves particularly strong results in paralinguistics (91.61% win rate), questions (84.41%), and syntactic complexity (83.39%).

Highlights Copy Chapter

Fine-Grained Inline Control via Natural Language Copy Section

S2 enables localized control over speech generation by embedding natural-language instructions directly at specific word or phrase positions within the text. Rather than relying on a fixed set of predefined tags, S2 accepts free-form textual descriptions — such as [whisper in small voice], [professional broadcast tone], or [pitch up] — allowing open-ended expression control at the word level.

Dual-Autoregressive Architecture Copy Section

S2 builds on a decoder-only transformer combined with an RVQ-based audio codec (10 codebooks, ~21 Hz frame rate). The Dual-AR architecture splits generation into two stages:

• Slow AR operates along the time axis and predicts the primary semantic codebook.
• Fast AR generates the remaining 9 residual codebooks at each time step, reconstructing fine-grained acoustic detail.

This asymmetric design — 4B parameters along the time axis, 400M parameters along the depth axis — keeps inference efficient while preserving audio fidelity.

Reinforcement Learning Alignment Copy Section

S2 uses Group Relative Policy Optimization (GRPO) for post-training alignment. The same models used to filter and annotate training data are directly reused as reward models during RL — eliminating distribution mismatch between pre-training data and post-training objectives. The reward signal combines semantic accuracy, instruction adherence, acoustic preference scoring, and timbre similarity.

Production Streaming via SGLang Copy Section

Because the Dual-AR architecture is structurally isomorphic to standard autoregressive LLMs, S2 directly inherits all LLM-native serving optimizations from SGLang — including continuous batching, paged KV cache, CUDA graph replay, and RadixAttention-based prefix caching.

On a single NVIDIA H200 GPU:

• Real-Time Factor (RTF): 0.195
• Time-to-first-audio: ~100 ms
• Throughput: 3,000+ acoustic tokens/s while maintaining RTF below 0.5

Multilingual Support Copy Section

S2 supports high-quality multilingual text-to-speech without requiring phonemes or language-specific preprocessing. Including:

English, Chinese, Japanese, Korean, Arabics, German, French...

AND MORE!

The list is constantly expanding, check Fish Audio for the latest releases.

Native Multi-Speaker Generation Copy Section

Fish Audio S2 allows users to upload reference audio with multi-speaker, the model will deal with every speaker's feature via <|speaker:i|> token. Then you can control the model's performance with the speaker id token, allowing a single generation to include multiple speakers. You no longer need to upload reference audio separately for each speaker.

Multi-Turn Generation Copy Section

Thanks to the expansion of the model context, our model can now use previous information to improve the expressiveness of subsequent generated content, thereby increasing the naturalness of the content.

Rapid Voice Cloning Copy Section

Fish Audio S2 supports accurate voice cloning using a short reference sample (typically 10–30 seconds). The model captures timbre, speaking style, and emotional tendencies, producing realistic and consistent cloned voices without additional fine-tuning. Please refer to [SGLang-Omni README](https://github.com/sgl-project/sglang-omni/blob/main/sglang_omni/models/fishaudio_s2_pro/README.md) to use the SGLang server. Copy Chapter

Credits Copy Chapter

• VITS2 (daniilrobnikov)
• Bert-VITS2
• GPT VITS
• MQTTS
• GPT Fast
• GPT-SoVITS
• Qwen3

Tech Report Copy Chapter

@misc{fish-speech-v1.4,
      title={Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis},
      author={Shijia Liao and Yuxuan Wang and Tianyu Li and Yifan Cheng and Ruoyi Zhang and Rongzhi Zhou and Yijin Xing},
      year={2024},
      eprint={2411.01156},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2411.01156},
}

@misc{liao2026fishaudios2technical,
      title={Fish Audio S2 Technical Report}, 
      author={Shijia Liao and Yuxuan Wang and Songting Liu and Yifan Cheng and Ruoyi Zhang and Tianyu Li and Shidong Li and Yisheng Zheng and Xingwei Liu and Qingzheng Wang and Zhizhuo Zhou and Jiahua Liu and Xin Chen and Dawei Han},
      year={2026},
      eprint={2603.08823},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2603.08823}, 
}

File: docs/en/inference.md Copy Chapter

Inference

The Fish Audio S2 model requires a large amount of VRAM. We recommend using a GPU with at least 24GB for inference.

Download Weights Copy Chapter

First, you need to download the model weights:

hf download fishaudio/s2-pro --local-dir checkpoints/s2-pro

Command Line Inference Copy Chapter

!!! note
If you plan to let the model randomly choose a voice timbre, you can skip this step.

1. Get VQ tokens from reference audio Copy Section

python fish_speech/models/dac/inference.py \
    -i "test.wav" \
    --checkpoint-path "checkpoints/s2-pro/codec.pth"

You should get a fake.npy and a fake.wav.

2. Generate Semantic tokens from text: Copy Section

python fish_speech/models/text2semantic/inference.py \
    --text "The text you want to convert" \
    --prompt-text "Your reference text" \
    --prompt-tokens "fake.npy" \
    # --compile

This command will create a codes_N file in the working directory, where N is an integer starting from 0.

!!! note
You may want to use --compile to fuse CUDA kernels for faster inference. However, we recommend using our sglang inference acceleration optimization.
Correspondingly, if you do not plan to use acceleration, you can comment out the --compile parameter.

!!! info
For GPUs that do not support bf16, you may need to use the --half parameter.

3. Generate vocals from semantic tokens: Copy Section

python fish_speech/models/dac/inference.py \
    -i "codes_0.npy" \

After that, you will get a fake.wav file.

WebUI Inference Copy Chapter

1. Gradio WebUI Copy Section

For compatibility, we still maintain the Gradio WebUI.

python tools/run_webui.py # --compile if you need acceleration

2. Awesome WebUI Copy Section

Awesome WebUI is a modernized Web interface built with TypeScript, offering richer features and a better user experience.

Build WebUI:

You need to have Node.js and npm installed on your local machine or server.

1. Enter the awesome_webui directory:
   bash Copy

   cd awesome_webui

2. Install dependencies:
   bash Copy

   npm install

3. Build the WebUI:
   bash Copy

   npm run build

Start Backend Server:

After building the WebUI, return to the project root and start the API server:

python tools/api_server.py --listen 0.0.0.0:8888 --compile

Access:

Once the server is running, you can access it via your browser:
http://localhost:8888/ui

File: docs/ja/finetune.md Copy Chapter

ファインチューニング

このページを開いたということは、明らかに、事前学習済みモデルのゼロショット性能に満足していないということでしょう。データセットでより良い性能を発揮するようにモデルをファインチューニングしたいとお考えのはずです。

現在のバージョンでは、「LLAMA」部分のみをファインチューニングする必要があります。

LLAMA のファインチューニング Copy Chapter

1. データセットの準備 Copy Section

.
├── SPK1
│   ├── 21.15-26.44.lab
│   ├── 21.15-26.44.mp3
│   ├── 27.51-29.98.lab
│   ├── 27.51-29.98.mp3
│   ├── 30.1-32.71.lab
│   └── 30.1-32.71.mp3
└── SPK2
    ├── 38.79-40.85.lab
    └── 38.79-40.85.mp3

データセットを上記の形式に変換し、data ディレクトリに配置する必要があります。音声ファイルの拡張子は .mp3、.wav、または .flac が使用でき、注釈ファイルの拡張子は .lab にすることを推奨します。

!!! info
.lab 注釈ファイルには、音声の書き起こしテキストのみを含める必要があり、特別なフォーマット要件はありません。たとえば、hi.mp3 の内容が「こんにちは、さようなら。」である場合、hi.lab ファイルには「こんにちは、さようなら。」という一行のテキストのみが含まれます。

!!! warning
データセットにラウドネス正規化を適用することをお勧めします。これには fish-audio-preprocess を使用できます。
bash fap loudness-norm data-raw data --clean

2. セマンティックトークンの一括抽出 Copy Section

VQGANの重みをダウンロードしていることを確認してください。まだの場合は、次のコマンドを実行してください。

huggingface-cli download fishaudio/openaudio-s1-mini --local-dir checkpoints/openaudio-s1-mini

その後、次のコマンドを実行してセマンティックトークンを抽出できます。

python tools/vqgan/extract_vq.py data \
    --num-workers 1 --batch-size 16 \
    --config-name "modded_dac_vq" \
    --checkpoint-path "checkpoints/openaudio-s1-mini/codec.pth"

!!! note
--num-workers と --batch-size を調整して抽出速度を向上させることができますが、GPUメモリの制限を超えないように注意してください。

このコマンドは data ディレクトリに .npy ファイルを作成します。以下のようになります。

.
├── SPK1
│   ├── 21.15-26.44.lab
│   ├── 21.15-26.44.mp3
│   ├── 21.15-26.44.npy
│   ├── 27.51-29.98.lab
│   ├── 27.51-29.98.mp3
│   ├── 27.51-29.98.npy
│   ├── 30.1-32.71.lab
│   ├── 30.1-32.71.mp3
│   └── 30.1-32.71.npy
└── SPK2
    ├── 38.79-40.85.lab
    ├── 38.79-40.85.mp3
    └── 38.79-40.85.npy```

### 3. データセットを protobuf にパックする

```bash
python tools/llama/build_dataset.py \
    --input "data" \
    --output "data/protos" \
    --text-extension .lab \
    --num-workers 16

コマンドの実行が完了すると、data ディレクトリに protos ファイルが表示されるはずです。

4. 最後に LoRA でファインチューニング Copy Section

同様に、LLAMA の重みをダウンロードしていることを確認してください。まだの場合は、次のコマンドを実行してください。

huggingface-cli download fishaudio/openaudio-s1-mini --local-dir checkpoints/openaudio-s1-mini

最後に、次のコマンドを実行してファインチューニングを開始できます。

python fish_speech/train.py --config-name text2semantic_finetune \
    project=$project \
    [email protected]_config=r_8_alpha_16

!!! note
fish_speech/configs/text2semantic_finetune.yaml を変更することで、batch_size や gradient_accumulation_steps などのトレーニングパラメータをGPUメモリに合わせて変更できます。

!!! note
Windows ユーザーの場合、trainer.strategy.process_group_backend=gloo を使用して nccl の問題を回避できます。

トレーニングが完了したら、推論 のセクションを参照してモデルをテストできます。

!!! info
デフォルト設定では、モデルは話者の発音方法のみを学習し、音色は学習しません。音色の安定性を確保するためには、依然としてプロンプトを使用する必要があります。
音色を学習させたい場合は、トレーニングステップ数を増やしてください。ただし、これにより過学習が発生する可能性があります。

トレーニング後、推論を行う前に LoRA の重みを通常の重みに変換する必要があります。

python tools/llama/merge_lora.py \
	--lora-config r_8_alpha_16 \
	--base-weight checkpoints/openaudio-s1-mini \
	--lora-weight results/$project/checkpoints/step_000000010.ckpt \
	--output checkpoints/openaudio-s1-mini-yth-lora/

!!! note
他のチェックポイントを試すこともできます。要件を満たす最も早いチェックポイントを使用することをお勧めします。これらは通常、OOD（分布外）データに対してより良いパフォーマンスを発揮します。

File: docs/ja/index.md Copy Chapter

!!! info "ライセンス通知"
このコードベースおよび関連するモデルの重みは FISH AUDIO RESEARCH LICENSE の下でリリースされています。詳細は LICENSE を参照してください。

!!! warning "法的免責事項"
私たちは、コードベースのいかなる違法な使用に対しても責任を負いません。DMCA およびその他の関連法に関する現地の規制を参照してください。

クイックスタート Copy Chapter

まずはドキュメントから Copy Section

Fish Audio S2 の公式ドキュメントです。以下からすぐに始められます。

• インストール
• コマンドライン推論
• WebUI 推論
• サーバー推論
• Docker セットアップ

[!IMPORTANT]
SGLang サーバーについては SGLang-Omni README を参照してください。

LLM Agent 向け Copy Section

https://speech.fish.audio/ja/install/ の手順に従って、Fish Audio S2 をインストール・設定してください。

Fish Audio S2 Copy Chapter

オープンソースおよびクローズドソースの中で最も優れたテキスト読み上げシステム

Fish Audio S2 は Fish Audio が開発した最新モデルです。約 50 言語・1,000 万時間超の音声データで学習され、強化学習アラインメントと Dual-Autoregressive アーキテクチャを組み合わせることで、自然でリアルかつ感情表現豊かな音声を生成します。

S2 は [laugh]、[whispers]、[super happy] といった自然言語タグで、韻律や感情を文中の任意位置で細かく制御できます。さらに、マルチスピーカー生成とマルチターン生成にもネイティブ対応しています。

ライブデモは Fish Audio ウェブサイト から、詳細は ブログ記事 と 技術レポート をご覧ください。

モデルバリアント Copy Section

モデルの詳細は技術レポートをご参照ください。

ベンチマーク結果 Copy Chapter

Seed-TTS Eval では、S2 はクローズドソースを含む全評価モデルの中で最小 WER を達成しました：Qwen3-TTS（0.77/1.24）、MiniMax Speech-02（0.99/1.90）、Seed-TTS（1.12/2.25）。Audio Turing Test では 0.515 を記録し、Seed-TTS（0.417）比で 24%、MiniMax-Speech（0.387）比で 33% 上回りました。EmergentTTS-Eval では、副言語情報（91.61%）、疑問文（84.41%）、統語的複雑性（83.39%）で特に高い成績を示しています。

ハイライト Copy Chapter

自然言語による細粒度インライン制御 Copy Section

Fish Audio S2 では、テキスト内の特定の単語やフレーズ位置に自然言語の指示を直接埋め込むことで、音声生成を局所的に制御できます。固定の事前定義タグに依存するのではなく、S2 は [whisper in small voice]、[professional broadcast tone]、[pitch up] のような自由形式のテキスト記述を受け付け、単語レベルで表現をオープンエンドに制御できます。

二重自己回帰（Dual-Autoregressive）アーキテクチャ Copy Section

S2 はデコーダー専用 Transformer と RVQ ベースの音声コーデック（10 codebooks、約 21 Hz）を組み合わせています。Dual-AR は生成を 2 段階に分割します。

• Slow AR は時間軸方向に動作し、主となる semantic codebook を予測。
• Fast AR は各時刻で残り 9 個の residual codebook を生成し、細かな音響ディテールを復元。

この非対称設計（時間軸 4B パラメータ、深さ軸 400M パラメータ）により、音質を保ちながら推論効率を高めています。

強化学習アラインメント Copy Section

S2 は後学習アラインメントに Group Relative Policy Optimization（GRPO）を採用しています。学習データのフィルタリングとアノテーションに使った同一モデル群を、そのまま RL の報酬モデルとして再利用することで、事前学習データ分布と事後学習目的のミスマッチを抑制しています。報酬信号には、意味的正確性、指示追従性、音響的選好スコア、音色類似度が含まれます。

SGLang による本番向けストリーミング Copy Section

Dual-AR は構造的に標準的な自己回帰 LLM と同型のため、S2 は SGLang の LLM 向け最適化をそのまま活用できます。たとえば continuous batching、paged KV cache、CUDA graph replay、RadixAttention ベースの prefix caching です。

単一の NVIDIA H200 GPU での実測:

• RTF（Real-Time Factor）: 0.195
• 初回音声出力までの時間: 約 100 ms
• スループット: RTF 0.5 未満を維持しつつ 3,000+ acoustic tokens/s

多言語サポート Copy Section

Fish Audio S2 は、音素や言語固有の前処理を必要とせずに、高品質な多言語テキスト読み上げをサポートします。以下を含みます：

英語、中国語、日本語、韓国語、アラビア語、ドイツ語、フランス語...

さらに多く！

リストは常に拡大しています。最新のリリースについては Fish Audio を確認してください。

ネイティブなマルチスピーカー生成 Copy Section

Fish Audio S2 では、ユーザーが複数のスピーカーを含む参照オーディオをアップロードでき、モデルは <|speaker:i|> トークンを介して各スピーカーの特徴を処理します。その後、スピーカーIDトークンを使用してモデルのパフォーマンスを制御し、1回の生成で複数のスピーカーを含めることができます。以前のように各スピーカーに対して個別に参照オーディオをアップロードして音声を生成する必要はもうありません。

マルチターン対話生成 Copy Section

モデルのコンテキストの拡張により、以前の情報を使用して後続の生成されたコンテンツの表現力を向上させ、コンテンツの自然さを高めることができるようになりました。

高速音声クローニング Copy Section

Fish Audio S2 は、短い参照サンプル（通常10〜30秒）を使用した正確な音声クローニングをサポートしています。モデルは音色、話し方、感情的な傾向を捉え、追加の微調整なしでリアルで一貫したクローン音声を生成します。
SGLang サーバーの利用については SGLang-Omni README を参照してください。

クレジット Copy Chapter

• VITS2 (daniilrobnikov)
• Bert-VITS2
• GPT VITS
• MQTTS
• GPT Fast
• GPT-SoVITS
• Qwen3

技術レポート Copy Chapter

@misc{fish-speech-v1.4,
      title={Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis},
      author={Shijia Liao and Yuxuan Wang and Tianyu Li and Yifan Cheng and Ruoyi Zhang and Rongzhi Zhou and Yijin Xing},
      year={2024},
      eprint={2411.01156},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2411.01156},
}

@misc{liao2026fishaudios2technical,
      title={Fish Audio S2 Technical Report}, 
      author={Shijia Liao and Yuxuan Wang and Songting Liu and Yifan Cheng and Ruoyi Zhang and Tianyu Li and Shidong Li and Yisheng Zheng and Xingwei Liu and Qingzheng Wang and Zhizhuo Zhou and Jiahua Liu and Xin Chen and Dawei Han},
      year={2026},
      eprint={2603.08823},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2603.08823}, 
}

File: docs/ja/inference.md Copy Chapter

推論

Fish Audio S2 モデルは大きなビデオメモリを必要とします。推論には少なくとも 24GB の GPU を使用することをお勧めします。

重みのダウンロード Copy Chapter

まず、モデルの重みをダウンロードする必要があります：

hf download fishaudio/s2-pro --local-dir checkpoints/s2-pro

コマンドライン推論 Copy Chapter

!!! note
モデルに音声をランダムに選択させる場合は、このステップをスキップできます。

1. リファレンスオーディオから VQ トークンを取得する Copy Section

python fish_speech/models/dac/inference.py \
    -i "test.wav" \
    --checkpoint-path "checkpoints/s2-pro/codec.pth"

fake.npy と fake.wav が生成されるはずです。

2. テキストから Semantic トークンを生成する： Copy Section

python fish_speech/models/text2semantic/inference.py \
    --text "変換したいテキスト" \
    --prompt-text "リファレンステキスト" \
    --prompt-tokens "fake.npy" \
    # --compile

このコマンドは、作業ディレクトリに codes_N ファイルを作成します。ここで N は 0 から始まる整数です。

!!! note
より高速な推論のために CUDA カーネルを融合する --compile を使用したい場合がありますが、私たちの sglang 推論加速最適化を使用することをお勧めします。
同様に、加速を使用する予定がない場合は、--compile パラメータをコメントアウトしてください。

!!! info
bf16 をサポートしていない GPU の場合、--half パラメータを使用する必要があるかもしれません。

3. セマンティックトークンから音声を生成する： Copy Section

python fish_speech/models/dac/inference.py \
    -i "codes_0.npy" \

その後、fake.wav ファイルが取得できます。

WebUI 推論 Copy Chapter

1. Gradio WebUI Copy Section

互換性を維持するため、以前の Gradio WebUI も引き続き利用可能です。

python tools/run_webui.py # 加速が必要な場合は --compile

2. Awesome WebUI Copy Section

Awesome WebUI は TypeScript で開発された、より豊富な機能と優れたユーザー体験を提供する最新の Web インターフェースです。

WebUI のビルド：

ローカルまたはサーバーに Node.js と npm がインストールされている必要があります。

1. awesome_webui ディレクトリに移動します：
   bash Copy

   cd awesome_webui

2. 依存関係をインストールします：
   bash Copy

   npm install

3. WebUI をビルドします：
   bash Copy

   npm run build

バックエンドサーバーの起動：

WebUI のビルドが完了したら、プロジェクトのルートに戻り、API サーバーを起動します：

python tools/api_server.py --listen 0.0.0.0:8888 --compile

アクセス：

サーバーが起動したら、ブラウザから以下のアドレスにアクセスして体験できます：
http://localhost:8888/ui

2. Official Technical Reference & Guides (fishaudio/docs) Copy Chapter

Fish Audio

Canonical documentation index for Fish Audio APIs, SDKs, models, voice cloning, real-time streaming, and self-hosting. This custom index prioritizes the lowest-noise, highest-value entry points for AI agents.

Start Here Copy Chapter

• Coding Assistant Quickstart: Minimal-noise entry point for AI coding assistants with retrieval order, canonical URLs, and task routing.
• Quick Start: Generate your first AI voice with Fish Audio in under 5 minutes.
• AI Coding Assistants: Install the Fish Audio skill so your coding agent writes correct SDK and API code.

API Specs Copy Chapter

• OpenAPI: REST API schema for text-to-speech, speech-to-text, wallet, and voice model endpoints.
• AsyncAPI: WebSocket schema for real-time text-to-speech streaming.

Core REST API Copy Chapter

• API Introduction: How to use the Fish Audio API.
• Text to Speech Endpoint: Convert text to speech.
• Speech to Text Endpoint: Transcribe audio to text.
• Voice Design Endpoint: Generate candidate voices from a prompt.
• List Models: Get a list of all models.
• Create Model: Create a new voice model.
• Get Model: Get details of a specific model.
• Update Model: Update an existing model.
• Delete Model: Delete an existing model.
• WebSocket TTS Streaming: Real-time text-to-speech streaming via WebSocket.

SDKs Copy Chapter

• Python SDK Overview: Official Python SDK overview and setup.
• Python Authentication: Configure API authentication for the Fish Audio Python SDK.
• Python Text-to-Speech: Generate natural-sounding speech with the Fish Audio Python SDK.
• Python Voice Cloning: Clone voices and create custom voice models with the Fish Audio Python SDK.
• Python WebSocket Streaming: Stream text-to-speech in real-time with WebSocket connections.
• JavaScript Installation: Install and set up the Fish Audio JavaScript SDK.
• JavaScript Authentication: Manage API keys and client setup in the Fish Audio JavaScript SDK.
• JavaScript Text-to-Speech: Convert text to natural speech with Fish Audio JavaScript SDK.
• JavaScript Voice Cloning: Clone voices using reference audio with Fish Audio JavaScript SDK.
• JavaScript WebSocket: Real-time streaming with Fish Audio JavaScript SDK.

Product Guides Copy Chapter

• Text to Speech Guide: Convert text to natural-sounding speech with Fish Audio.
• Speech to Text Guide: Convert audio recordings into accurate text transcriptions.
• Voice Design Guide: Generate candidate voices from natural-language prompts.
• Creating Voice Models: Learn how to create custom voice models with Fish Audio.
• Emotion Control: Add natural emotions and expressions to your AI-generated speech.
• Fine-grained Control: Advanced control over speech generation.
• English Phoneme Control: Control English pronunciation with CMU Arpabet.
• Chinese Phoneme Control: Control Chinese pronunciation with tone-number pinyin.
• Japanese Phoneme Control: Control Japanese pronunciation with romaji phonemes and pitch accent markers.
• Voice Cloning Best Practices: Improve voice cloning quality and consistency.
• Real-time Voice Streaming: Stream voice generation in real time for interactive applications.

Voice Agents Copy Chapter

• Agents Overview: The Fish Audio voice-agent platform — build agents in the console, integrate via REST API and web SDK.
• Agents Quickstart: Create, test, and publish your first voice agent via the dashboard or the API.
• Agent Concepts: Workspaces, agents, draft and published versions, and sessions.
• Agent Configuration: System prompt, first message modes, and draft autosave.
• Voice & Language: Choose the agent's voice profile and speaking language.
• Time & Timezone: Agents know the current date and time by default; timezone resolution and opt-out.
• Knowledge Base: Attach documents so the agent can answer from your content.
• Tools Overview: Webhook, client, and system tools for voice agents.
• Webhook Tools: Let the agent call your HTTP endpoints, with templating and testing.
• Client Tools: Tools your app executes in the browser through the SDK.
• System Tools: Built-in capabilities like hanging up the call.
• Dynamic Variables: Personalize each session with template variables supplied at creation time.
• Custom LLM: Serve your agent's replies from your own OpenAI-compatible model endpoint.
• Versions & Publishing: Draft, publish, clone, and restore agent configurations.
• Deployment Overview: Pick a deployment channel — widget, public agents, authenticated sessions, or a phone number.
• Authentication & Session Tokens: Public agent IDs vs server-minted session tokens; POST /v1/agent/sessions.
• Public Agents: Credential-free browser access with an origin allowlist.
• Web SDK: @fishaudio/agent-client — events, modes, transcripts, audio controls, and errors.
• React SDK: React hooks and components for voice-agent UIs.
• Widget: Embed a complete voice and chat UI on any website with two lines of HTML — no build step.
• Wire Protocol: The realtime message protocol for non-SDK consumers.
• Phone Numbers: Manage phone numbers and bind them to agents.
• Inbound Calls: Answer phone calls with an agent and track caller attribution.
• Call Transfers: Let the agent hand a phone call to a human — cold carrier handoff or warm transfer with a private briefing.
• Preview Calls: Test your draft agent by voice inside the Builder.
• Agent Tests: Scripted next-reply tests scored by an LLM judge.
• Conversation History: Query sessions, transcripts, tool timelines, and recordings via the API.
• Post-call Analysis: Automatic summaries, data extraction, and success criteria.
• Agent Webhooks: call.ended and call.analyzed events with HMAC signature verification.

Operational Docs Copy Chapter

• Models Overview: Explore Fish Audio's voice generation models and their capabilities.
• Choosing a Model: Select the right Fish Audio model for your use case and requirements.
• Pricing And Rate Limits: Understand pricing, usage costs, and API rate limits.
• Model Deprecations: Track deprecated models and migration timelines.
• Migration Guide: Switch from ElevenLabs, OpenAI, or other TTS providers to Fish Audio.
• Changelog: Complete release history and version updates for Fish Audio products.