# Technical Documentation: Huanshere/VideoLingo

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/Huanshere/VideoLingo)

### Overview (`README.md`)

<div align="center">

<img src="/docs/logo.png" alt="VideoLingo Logo" height="140">

# Connect the World, Frame by Frame

<a href="https://trendshift.io/repositories/12200" target="_blank"><img src="https://trendshift.io/api/badge/repositories/12200" alt="Huanshere%2FVideoLingo | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>

[**English**](/README.md)｜[**简体中文**](/translations/README.zh.md)｜[**繁體中文**](/translations/README.zh-TW.md)｜[**日本語**](/translations/README.ja.md)｜[**Español**](/translations/README.es.md)｜[**Русский**](/translations/README.ru.md)｜[**Français**](/translations/README.fr.md)

</div>

## 🌟 Overview ([Try VL Now!](https://videolingo.io))

VideoLingo combines speech recognition, subtitle translation, segmentation and dubbing in a Streamlit interface. It produces subtitle files and optionally subtitled or dubbed videos. Translation quality depends on the source audio, language and chosen models.

Key features:
- 🎥 YouTube video download via yt-dlp

- Word-level speech recognition and alignment with Qwen3-ASR + Qwen3-ForcedAligner

- **📝 NLP and AI-powered subtitle segmentation**

- **📚 Custom + AI-generated terminology for coherent translation**

- Direct translation with optional reflection and natural rewriting

- Subtitle segmentation with configurable length limits

- **🗣️ Dubbing with GPT-SoVITS, OpenAI, Edge TTS, and more**

- 🚀 One-click startup and processing in Streamlit

- 🌍 Multi-language support in Streamlit UI

- 📝 Detailed logging with progress resumption

- 🔍 Model searchbox with API auto-fetch — search and filter from your provider's full model list

- ⏯️ Task control — pause, resume, or stop processing at any step

The workflow combines transcription, translation, subtitle layout and dubbing in one project.

## 🎥 Demo

<table>
<tr>
<td width="33%">

### Dual Subtitles
---
https://github.com/user-attachments/assets/a5c3d8d1-2b29-4ba9-b0d0-25896829d951

</td>
<td width="33%">

### Cosy2 Voice Clone
---
https://github.com/user-attachments/assets/e065fe4c-3694-477f-b4d6-316917df7c0a

</td>
<td width="33%">

### GPT-SoVITS with my voice
---
https://github.com/user-attachments/assets/47d965b2-b4ab-4a0b-9d08-b49a7bf3508c

</td>
</tr>
</table>

### Language Support

**Input languages:**

🇺🇸 English 🤩 | 🇷🇺 Russian 😊 | 🇫🇷 French 🤩 | 🇩🇪 German 🤩 | 🇮🇹 Italian 🤩 | 🇪🇸 Spanish 🤩 | 🇯🇵 Japanese 😊 | 🇨🇳 Chinese 🤩

Dubbing languages depend on the selected TTS method.

## Installation

VideoLingo supports Windows, macOS (Apple Silicon / Intel), and Linux.

### Ask your local AI agent 🤖

If you use an AI agent that can operate your computer, send it this prompt:

> Install and launch GitHub's Huanshere/VideoLingo on my computer.

### Windows: one-click install 🎉

1. Download **Source code (zip)** from the [latest Release](https://github.com/Huanshere/VideoLingo/releases/latest), extract it to your Desktop or another folder, and open the folder.
2. Double-click `OneKeyStart.bat` and keep the window open. On the first run, it automatically installs uv, Python 3.12, the app dependencies, and FFmpeg. An internet connection is required.
3. After installation, VideoLingo opens automatically in your browser. Enter your API URL, key, and model in the sidebar to start using it.

### Install from source (Windows, macOS, Linux)

```bash
git clone https://github.com/Huanshere/VideoLingo.git && cd VideoLingo
uv run start.py
```

To start it later, run `uv run start.py` again from the VideoLingo folder. Apple Silicon uses MLX; Intel Macs use CPU recognition. By default, Intel Mac dubbing uses the new voice without the original background sound.

#### Docker (optional)

For a Linux NVIDIA container deployment, install Docker, a compatible GPU driver and the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). The image uses the same Python 3.12 setup and application dependencies, with CUDA 12.8.1/cu128 by default. See [Docker docs](/docs/pages/docs/docker.en-US.md) for the matched CUDA 12.6 alternative and persistence settings.

```bash
docker build -t videolingo .
docker run -d -p 8501:8501 --gpus all videolingo
```

## HTTP API (replaces Excel batch mode)

For agents and scripts, use the local HTTP API instead of the former Excel batch mode.
It shares the Streamlit pipeline and processes one operation at a time using `output/`.
Configure `config.yaml` and start it from the project root. The command also installs missing dependencies on first use:

```bash
uv run start.py --api
```

See the **[HTTP API guide](docs/api.md)** for input, processing, progress, downloads,
retries and serial batch processing. Interactive endpoint docs: [localhost:8000/docs](http://localhost:8000/docs).

## LLM, ASR and TTS providers
VideoLingo supports OpenAI-Like API format and various TTS interfaces:
- LLM: choose an OpenAI-compatible Chat Completions provider and model that can return the structured JSON required by the workflow. [OpenLux](https://www.openlux.ai/register?aff=wKYu) is recommended; set the API URL to `https://api.openlux.ai/v1`. Prefer GPT-6 Luna with model ID `gpt-6-luna` for best value, GPT-6 Sol with `gpt-6-sol` for better quality, or Claude Opus 5.5 with `claude-opus-5-5` for best quality. OpenLux relay rates are in the install docs. Configure the API URL, key and model in the sidebar.
- Speech recognition: run Qwen3-ASR + ForcedAligner locally (default), or choose ElevenLabs or MAI-Transcribe-2 in the sidebar. MAI supports an Azure Speech key or an OpenRouter key entered in the sidebar; audio is sent to the selected provider and may incur charges. WhisperX is not installed by the installer; to use it as a backend, follow [WhisperX (manual install)](docs/pages/docs/whisperx-manual.en-US.md).
- TTS: OpenAI, Fish Audio, SiliconFlow Fish/CosyVoice2, GPT-SoVITS, Edge TTS, F5-TTS and a custom adapter in `core/tts_backend/custom_tts.py`.

For detailed installation, LLM configuration, and usage instructions, please refer to the documentation: [English](/docs/pages/docs/start.en-US.md) | [中文](/docs/pages/docs/start.zh-CN.md)

## Current Limitations

1. Background noise and language-specific alignment models affect recognition and word timestamps. Vocal separation may help. Numbers and symbols may lack reliable word timings; inspect the resulting subtitles.

2. LLM output must satisfy the workflow's JSON structure. For failures, inspect `output/gpt_log/error.json`. Existing successful response caches and completed outputs can be reused on retry; changing the model alone does not regenerate every completed stage. Do not delete all output as the first troubleshooting step.

3. Dubbing quality and timing depend on translation, the TTS service and speech rate. Speed adjustment does not guarantee natural delivery or perfect synchronization.

4. Local recognition uses one primary recognition/alignment language per audio segment. Mixed-language speech is not guaranteed to retain accurate text and timing in every language.

5. The dubbing workflow does not automatically assign a separate voice to each speaker.

## 📄 License

This project is licensed under the Apache 2.0 License. Special thanks to the following open source projects for their contributions:

[Qwen3-ASR](https://github.com/Qwen/Qwen3-ASR), [MLX Audio](https://github.com/Blaizzy/mlx-audio), [whisperX](https://github.com/m-bain/whisperX), [yt-dlp](https://github.com/yt-dlp/yt-dlp), [json_repair](https://github.com/mangiucugna/json_repair), [BELLE](https://github.com/LianjiaTech/BELLE)

## 📬 Contact Me

- Submit [Issues](https://github.com/Huanshere/VideoLingo/issues) or [Pull Requests](https://github.com/Huanshere/VideoLingo/pulls) on GitHub
- DM me on Twitter: [@Huanshere](https://twitter.com/Huanshere)
- Email me at: team@videolingo.io

## ⭐ Star History

[](https://star-history.com/#Huanshere/VideoLingo&Timeline)

---

<p align="center">If you find VideoLingo helpful, please give me a ⭐️!</p>

---

### Api (`docs/api.md`)

# Local HTTP API

The API replaces Excel batch mode. It uses the **same pipeline as Streamlit**, the existing
`config.yaml`, fixed `output/`, and `history/`. There is one operation at a time, no queue,
no database and no task directories. Use either the UI or API for a working directory.

## Start

Configure your LLM and other settings in `config.yaml`, then run from the repository root.
The same command installs any missing dependencies on first use:

```bash
uv run start.py --api
```

The server listens on `127.0.0.1:8000`. Interactive documentation: http://127.0.0.1:8000/docs;
OpenAPI schema: http://127.0.0.1:8000/openapi.json. This is a trusted local API with no
authentication; keep it on localhost. Run one server process, without multiple workers.

## Process a video

Send JSON with `Content-Type: application/json`:

```http
POST /input
{"source": "/absolute/path/video.mp4"}
```

On Windows use a JSON path such as `"C:/Videos/video.mp4"`. HTTP(S) video URLs are also
accepted. Local files are copied into `output/`; source files are untouched.
`202` means accepted, not finished. Poll `GET /status` until `active` is false and
`state` is `completed`, then start processing:

```http
POST /run
{"stage": "all", "target_language": "zh", "dubbing": true}
```

`stage` is `transcribe`, `subtitles`, `dubbing`, or `all` (default). `dubbing` defaults to false and
only controls whether `all` includes dubbing. `stage: dubbing` requires existing subtitles.
`stage: transcribe` stops after the source subtitles: it writes `src.srt` only, without
translation or video. A later `subtitles` run reuses the transcription.
On Intel Macs, dubbing uses the new voice without mixing in the original background sound.
Audio-only input supports subtitles, matching the UI. Optional `source_language` and
`target_language` are saved to `config.yaml`; other settings use that file directly.
Do not edit configuration while processing.

Poll `GET /status` again. It returns `state`, `active`, `step`, `step_index` (zero-based),
`total_steps`, `progress`, `error`, and top-level output `files`. Progress counts completed
step groups, not elapsed time. States: `idle`, `running`, `paused`, `stopping`, `stopped`,
`completed`, `error`. With `pause_before_translate: true` in `config.yaml` the task pauses
after the terminology is extracted: `state` is `paused` and `pause_message` says why. Edit
`output/log/terminology.json` if needed, then continue with `POST /resume`.
`pause_after_translate: true` pauses in the same way after the translation: edit the
`Translation` column of `output/log/translation_results.xlsx`, keep the `Source` column and
the rows as they are, then continue with `POST /resume`.

```http
GET /files/trans.srt
POST /archive
```

## Existing subtitles

To translate subtitles that you already have, pass an SRT file (UTF-8). They are used instead
of the recognition, with their own lines and times:

```http
POST /input
{"source": "/absolute/path/video.mp4", "subtitles": "/absolute/path/video.srt"}
```

An SRT file as `source` translates the subtitles without a video: the result is subtitle files
only, as for audio-only input. The language of the subtitles can not be detected, so set
`source_language` (or `whisper.language` in `config.yaml`) to their language, not to `auto`.
The file is kept in `output/input/`.

Output files are also available directly in `output/`. File downloads only serve top-level
files inside this directory. `/archive` runs in the background: wait for completion before
preparing the next video. It retains the existing history behavior (same-name archives may
be overwritten); move results elsewhere first if you want to keep multiple versions.

## Serial processing, retries and stopping

- A busy `/input`, `/run`, or `/archive` returns `409`. Wait before submitting the next operation.
- `/input` rejects nonempty `output/` by default. Explicitly pass `"existing": "archive"`
  or `"existing": "replace"` to archive or delete old output before preparing the next input.
- Failed operations expose the error and step through `/status`. Fix the cause, then repeat
  `/run` to reuse existing intermediate files. No automatic whole-step retry is added.
- Changing parameters does **not** invalidate intermediate files. To regenerate with different
  settings, prepare the source again with `existing: archive` or `replace`, then run.
- `POST /stop` requests cooperative cancellation. It may wait for the current model/network call
  to finish. Wait for `active: false` before retrying or replacing input.
- Status is in memory. After a server restart it is `idle`; files remain, and `/run` can retry.
- No CLI, Excel task file, persistent job history, or service queue is needed. An agent can loop:
  input → wait → run → wait → collect results → archive → wait → next input.

## Tests (no model downloads)

These commands use the project `.venv`. If you use a shared environment, replace `.venv`
with its path in both commands.

```bash
uv pip install --python .venv pytest httpx
uv run --python .venv pytest tests/test_api.py
```

---

### Audio Extract Quality (`docs/audio-extract-quality.md`)

# 原始音频提取质量与克隆参考音频（关联 #533）

范围有限：本改动只回升 3.0.0 重构时降低的三处音频参数，并给归一化加峰值保护。
它不解决 #533 报告的全部症状，也未做真实 TTS 听测。成片响度已由 #610 单独处理。

## 改了什么

| 位置 | 3.0.0 至 #610 | 本改动 |
| --- | --- | --- |
| `audio_preprocess.convert_video_to_audio` / `prepare_audio_for_asr` | `raw.mp3` 16 kHz / 32 kbps | 32 kHz / 128 kbps，`config.yaml` 的 `audio.raw_sample_rate` / `audio.raw_bitrate` 可覆盖；缺键回落默认；低于 16000 报错 |
| `audio_preprocess.normalize_audio_volume` | 目标 -20 dBFS，pydub 在 int16 上加增益，稀疏语音会削波 | 正增益封顶在峰值 -1 dBFS，衰减不受限；静音输入不再产生 `inf` |
| `sf_fishtts.merge_audio` | 合并参考导出 16 kHz | 44.1 kHz（`REFER_SAMPLE_RATE`），与切片源一致 |
| `config_utils.load_key_or` | 无 | 读取可选键，旧 `config.yaml` 无需改动 |

`aresample=async=1:first_pts=0` 与 PCM 回退保持不变（见 `docs/audio-timeline.md`）。

## 为什么

`raw.mp3` 不只是识别输入。本地识别（默认的 Qwen3-ASR，以及可选的 WhisperX）内部都固定解码到 16 kHz，不受影响；但 Demucs 的人声、
`_9_refer_audio` 按字幕切出的 `refers/*.wav`、以及 `sf_fish_tts` custom/dynamic、`sf_cosyvoice2`、
`gpt_sovits`、`f5tts`、`fish_audio` 的克隆参考，全部继承提取时的带宽。

本机测量（ffmpeg 7.1，白噪声，48 kHz 源，8192 点 FFT，取相对 1 kHz 频段衰减 10 dB 处）：

| 提取参数 | -10 dB 截止 | 8 kHz 相对电平 | 体积 / 10 s |
| --- | --- | --- | --- |
| v2.2.1：32 kHz / 128 kbps | 约 15.3 kHz | 0 dB | 157 KB |
| 3.0.0 至 #610：16 kHz / 32 kbps | 约 7.2 kHz | -104 dB | 39 KB |
| 16 kHz 源再经 44.1 kHz / 128 kbps 重存（模拟 Demucs 输出） | 约 7.2 kHz | -87 dB | 157 KB |

3.0.0 降到 16 kHz 的原因是 302.ai WhisperX 云端 25 MB 上传限制，该服务已在 #609 移除；
ElevenLabs 路径按段解码到 16 kHz 再上传，与 `raw.mp3` 采样率无关。SiliconFlow 文档建议参考
音频用 192 kbps 以上 MP3 或 8 至 44.1 kHz WAV。

## 不覆盖什么

- 「一句一句念」「句间不连贯」来自逐行合成、硬静音拼接与逐块 `atempo`，2.2.1 与 3.0 代码相同，本改动不动。
- 同一视频内句间响度差异不处理；成片整体响度由 #610 的门限响度归一化处理。
- 克隆音色是否因此变好，没有用真实 TTS 验证；能保证的只是可测的信号属性。

## 副作用

| 项目 | 变化 |
| --- | --- |
| `raw.mp3` 体积 | 每小时约 14 MB 变 58 MB；PCM 回退每小时约 115 MB 变 230 MB |
| `split_audio` 内存 | pydub 整段载入，翻倍：1 小时约 230 MB，3 小时约 690 MB |
| ASR | 识别仍在 16 kHz；转录缓存键不含提取参数，缓存继续命中 |
| Demucs | 运行时间不变，内部固定 44.1 kHz |
| SiliconFlow 上传 | 10 秒参考的 base64 约从 320 KB 增到 1.2 MB |
| 归一化 | 稀疏且峰值尖的人声轨，`vocal.mp3` 可能低于 -20 dBFS |

已有项目不会重新提取：`convert_video_to_audio` 见到 `output/audio/raw.mp3` 就跳过。要让改动生效，
需删除 `output/audio/` 下的 `raw.mp3`、`vocal.mp3`、`background.mp3`、`refers/`、`tmp/`、`segs/`，
或整个 `output/`；用 `sf_fish_tts` custom 模式还要把 `sf_fish_tts.custom_name` 置空以重建音色。

## 验证

```sh
python -m pytest -q tests/test_audio_extract_quality.py tests/test_audio_timeline.py
```

用例覆盖：提取命令参数（含旧配置缺键、PCM 回退）、真实 FFmpeg 的 9 至 14 kHz 频段保留断言、
归一化峰值上限与静音输入、参考合并采样率、`load_key_or` 回落、`config.yaml` 默认值。
未覆盖：真实 Demucs、真实 TTS 与克隆 API、听感、内存与耗时。

---

### Audio Timeline (`docs/audio-timeline.md`)

# Preserve source audio timing during extraction

## Defect

The ASR input must follow the video's presentation clock. Decoding audio packets
into contiguous samples is not sufficient when packet timestamps contain gaps
or overlap. A compressed frame may decode to 1,024 samples while its container
duration is shorter. Concatenating every decoded sample expands the recognition
clock, so correctly aligned words produce increasingly late subtitles.

This is an existing extraction limitation, not a translation prompt issue.
The FFmpeg extraction command at upstream commit `6e85833` (2024-11-14,
`core/all_whisper_methods/whisperX_utils.py`) already resampled to 16 kHz without
timestamp reconciliation. The recent dependency update did not change that
extraction command. Some inputs with continuous timestamps are unaffected.

## Fix

Apply `aresample=async=1:first_pts=0` before encoding the recognition input,
including MP3 and PCM fallback paths for video and audio-only uploads. FFmpeg
reconciles sample output against timestamps rather than blindly concatenating
frames. No model, subtitle timing heuristic, or translation change is required.

## Validation

```sh
python -m pytest -q tests/test_audio_timeline.py
```

The tests exercise real extraction function bodies without importing ML models.
Synthetic PCM media covers a continuous clock, repeated overlaps and repeated
gaps. The real FFmpeg test compares decoded output duration to source frame
presentation times and demonstrates the old contiguous decode differs by more
than 0.8 seconds on discontinuous inputs. It requires FFmpeg and ffprobe and makes
no network calls. Mocked tests cover all four encoder/input combinations.

## Existing results

Existing recognition audio is retained by the resume mechanism. This fix applies
to newly extracted audio; restarting the application alone does not replace cached
audio or regenerate subtitles. Existing outputs require explicit regeneration or
an independently validated timestamp repair. Source recording/remuxing defects
may also require correction in their producing tool; this change prevents those
timestamp discontinuities from silently shifting VideoLingo's recognition clock.

## Demucs stems start late (fixed in BUILDER-305)

With vocal separation on, word timestamps came out about 60 ms late, because
`vocal.mp3` started 953 samples (16 kHz) after `raw.mp3`. Two MP3 encoder delays
were being counted as audio:

- Demucs' `separate_audio_file()` reads with sphn first. sphn keeps the LAME
  priming of `raw.mp3` (1,105 samples at 32 kHz, about 35 ms).
- `demucs.audio.save_audio` encodes MP3 with lameenc, which writes no LAME/Xing
  header, so decoders cannot trim its own priming (1,105 samples at 44.1 kHz,
  about 25 ms).

`demucs_vl` now decodes `raw.mp3` with FFmpeg (`demucs.audio.AudioFile`, which trims
the delay) and writes each stem through FFmpeg's libmp3lame (its header lets
FFmpeg, pydub and soundfile trim the delay). Both stems start sample-aligned with
`raw.mp3`. This also applies to the WhisperX backend, `background.mp3` in the
dubbing mix and the reference clips cut from `vocal.mp3`. `qwen_asr_local.match_length`
still pads or trims the tail, whose decoded length can differ by a frame.

`tests/test_demucs_vl.py` runs the real FFmpeg and Demucs I/O with the separation
model faked. It checks a lag of at most one 16 kHz sample for both stems, after
the pydub re-export in `_2_asr`, and through soundfile. The same checks against
the previous code measure 953 samples. Stems already in `output/audio/` are reused
as they are, so delete `vocal.mp3` and `background.mp3` (or start a new task) to
regenerate them. The ASR cache schema was bumped to 2, so results transcribed
from the old stems are not reused.

---

### Components.Json (`docs/components.json`)

{
  "$schema": "https://ui.shadcn.com/schema.json",
  "style": "default",
  "rsc": false,
  "tsx": true,
  "tailwind": {
    "config": "tailwind.config.js",
    "css": "pages/globals.css",
    "baseColor": "slate",
    "cssVariables": true,
    "prefix": ""
  },
  "aliases": {
    "components": "@/components",
    "utils": "@/lib/utils",
    "ui": "@/components/ui",
    "magicui": "@/components/magicui"
  }
}

---

### Deployment Versions (`docs/deployment-versions.md`)

# Deployment dependency audit

Reviewed against upstream `c5f8fe7` on 2026-09-15. Git trees, not PR descriptions
or commit timestamps alone, determine the effective dependency versions.

## History

| Baseline/change | Effective dependency changes |
| --- | --- |
| `v2.2.1` | NVIDIA: Torch/torchaudio 2.0.0 from cu118; CPU: 2.1.2. WhisperX Git revision `7307306`, Demucs Git development extra, CTranslate2 4.4.0, NumPy 1.26.4, Transformers 4.39.3, spaCy 3.7.4, Streamlit 1.38.0, librosa 0.10.2.post1. |
| `v3.0.0` | Retains that dependency family; adds xmltodict. The Dockerfile independently pins Torch 2.0.0/cu118. |
| #531, included in `v3.0.1` | Intermediate Torch 2.6/cu124 and WhisperX 3.4 work is superseded within the PR by Torch 2.8, WhisperX >=3.8.1, pyannote-audio >=4, CTranslate2 >=4.5/cuDNN 9, NumPy >=2, librosa 0.11, spaCy 3.8.11, Streamlit 1.49.1 and Lightning 2.6.1. Demucs installed separately with --no-deps. Docker documentation changed but Dockerfile did not. |
| `301d561`, also ancestor of `v3.0.1` | Adds cu129 to the old install.py selection. This is historical, not a pending future change. |
| #537 | Adds uv/Python 3.10 setup and project-venv batch launcher; dependency installation remains in install.py. |
| #576/#577 | Shared/local environments, staged installer.py, legacy install.py wrapper, bounded spaCy 3.8, separate WhisperX/Demucs stages. Docker and Colab were not migrated. |
| #599 | New uv environments use Python 3.13; existing environments supported on 3.10–3.13. Torch/torchaudio 2.8.0, torchvision 0.23.0, WhisperX >=3.8.6,<3.9, TorchCodec >=0.7,<0.8, Transformers <5, Hub <1. Normal PyPI Demucs >=4.1,<5 replaces Git/--no-deps. CUDA selection becomes cu128/cu126, CPU index explicit. Removes MoviePy, Replicate, direct resampy and duplicate Lightning pins. |
| #602/#610/#611 | Runtime/cache/logging and audio-processing changes; no replacement of the #599 dependency family. Audio sample rates and bitrates are media settings, not CUDA dependency versions. |
| BUILDER-305 (unreleased) | Default local ASR becomes Qwen3-ASR + Qwen3-ForcedAligner. Removes whisperx, pyannote-audio, ctranslate2 and torchcodec from `requirements.txt` and the installer's WhisperX stage (installer now has 6 stages). Adds marker-split engines: `qwen-asr==0.0.6` with Transformers 4.57/Hub <1 outside Apple Silicon; `mlx-audio>=0.5.5,<0.6` with Transformers 5/Hub 1 plus nagisa/soynlp on Apple Silicon (macOS ≥14). Torch 2.8 family unchanged. WhisperX is not an installer option; install it yourself from `docs/pages/docs/whisperx-manual.*.md`. On Apple Silicon the installer removes the WhisperX stack (whisperx, torchcodec, faster-whisper, ctranslate2, pyannote-*) (hub <1 conflict) unless another installed package still needs one, and does not install WhisperX again. It re-registers stale project metadata before syncing so pip reports no conflict against the old requirements. The installer stops early on Intel Macs and on Apple Silicon below macOS 14. |
| #624 (pending) | Setup uses Python 3.12. Intel Macs use qwen-asr on CPU with Torch/torchaudio 2.2.2, torchvision 0.17.2, NumPy 1.26 and Numba <0.63; optional Demucs is skipped because sphn has no Intel macOS wheel. Apple Silicon still uses MLX on macOS 14+. Windows/Linux retain Torch 2.8. |

Full current application bounds are in `requirements.txt`; optional Demucs is in
`installer.py`. Do not copy an older PR's intermediate version list into setup.

## Deployment alignment

- Host uv setup: `setup_env.py` creates Python 3.12 and delegates to `installer.py`.
  Auto selection uses the driver's `nvidia-smi` CUDA capability: >=12.8 selects
  cu128, otherwise cu126 when NVIDIA is detected. An unparseable capability also
  falls back to cu126; this fallback is not a compatibility guarantee. No NVIDIA
  selects CPU. Existing compatible builds can be reused without replacement.
- Docker: CUDA 12.8.1/cuDNN runtime on Ubuntu 24.04, Python 3.12 through the same
  setup script, explicit cu128. `CUDA_VERSION=12.6.3` selects the matching cu126
  variant. Explicit selection is necessary because image builds normally cannot
  see the runtime GPU. Host driver and NVIDIA Container Toolkit remain external.
  Source comes from the build context, not a fresh upstream clone. Build performs
  the common installer check and pip check; no model is downloaded at build time.
- Colab: application runs from its own Python 3.12 venv and the same installer.
  Colab's kernel Python remains platform-managed; pyngrok runs there. The install
  cell does not launch Streamlit, and the final cell uses the application venv.
  Embedded output logs predate this change and are explicitly labeled historical.
- Windows batch: shared venv, project venv, then existing legacy environment, with
  the common version check. The uv-named entry point delegates to the same launcher.
- Existing `install.py` delegates to `installer.py`; it is a compatibility wrapper,
  not another version policy. Direct `pip install -e .` uses requirements.txt but
  does not select GPU wheels or install optional Demucs. Use the staged installer.

CUDA Toolkit, PyTorch build tags and the driver's CUDA capability are different
layers. The installer selects Python wheels; it does not install a system Toolkit.
CUDA 13-capable drivers may run CUDA 12 builds. The default Qwen3-ASR path uses the
CUDA runtime bundled with the PyTorch wheels. For a manually installed WhisperX backend,
CTranslate2 4.5's release explicitly migrates to cuDNN 9; its generic installation page still mentions cuDNN 8, so use
the release-specific requirement and faster-whisper's current GPU instructions.

## Verification boundary

BUILDER-305 (2026-09-24): only offline checks. `uv pip compile` resolved the new
requirements for Linux x86_64, Windows x86_64 and macOS arm64 (target 14.0), and
the offline test suite passed. No image was rebuilt and no Qwen3-ASR inference
was run; the records below predate the migration and describe the WhisperX-based
environment.

Actual validation on 2026-09-15:

- Linux amd64, Docker Engine 24.0.2: default CUDA 12.8.1 image built successfully
  (`7471025af15a`, approximately 19 GB). Build-time and separate container
  `pip check` passed. Imported Torch/torchaudio 2.8.0+cu128, torchvision
  0.23.0+cu128, WhisperX ASR, TorchCodec 0.7.0, CTranslate2 4.8.2, Demucs API,
  Streamlit, spaCy 3.8.16, librosa 1.0.0 and OpenCV 5.0.0.
- In that image, FFmpeg generated a one-second WAV and TorchCodec decoded all
  44,100 samples. FFmpeg also rendered text using the installed Noto CJK font.
- Actual execution exposed a false font warning: `fc-match` needs the family
  `Noto Sans CJK SC`, not the filename stem `NotoSansCJK-Regular`. The corrected
  installer was mounted into the built image and its full environment check
  passed without that warning. The full image was not rebuilt after this fix.
- Windows driver 610.88 reports `CUDA UMD Version: 13.3`. Detection now accepts
  that format as well as `CUDA Version`, rather than silently selecting cu126.
- Windows fresh Python 3.13 environment completed installation including Demucs
  4.1.0; pip check passed. CUDA matrix multiplication on an RTX 4000 Ada passed,
  and CTranslate2 detected one CUDA device. Initial TorchCodec import failed with
  system FFmpeg 9.0.1. A downloaded BtbN FFmpeg 7.1 shared build plus explicit
  Windows DLL-directory registration fixed it; all major imports and decoding
  a one-second WAV to 44,100 samples passed. The installer now probes TorchCodec
  in a subprocess and rejects the incompatible FFmpeg 9 environment instead of
  reporting metadata-only success. `runtime_libraries.py` also configures DLL
  lookup before the application's core imports. These final changes were verified
  on Windows, not included in the earlier Docker image build.
- Twenty focused mocked installer tests passed, with six unrelated tests excluded.
  Python syntax, notebook JSON/code and seven-language README commands also passed.

The Docker host has no NVIDIA GPU: this verifies the CUDA image can be built on
a GPU-less host, not GPU inference. The cu126 image variant and an actual Colab
session have not been run. Component import and media decoding are not a complete
translation/dubbing quality test. Production environments and services are preserved.

---

### Download Network (`docs/download-network.md`)

# Download network configuration

Video downloads use yt-dlp's native proxy support. No proxy application, local
port, JavaScript runtime, or browser impersonation profile is forced.

Set `youtube.proxy` in `config.yaml`:

```yaml
youtube:
  cookies_path: ''
  proxy: null
```

- Missing or `null`: leave yt-dlp's proxy option unset, preserving its existing
  system/environment discovery. With no discovered proxy, downloads connect directly.
- `''`: explicitly connect directly, even if the environment has a proxy.
- A URL such as `http://proxy.example.com:8080` or
  `socks5://proxy.example.com:1080`: use that proxy for yt-dlp downloads.

This setting applies to yt-dlp only, not translation, model downloads, TTS, or the
existing pip update operation preceding a download. It does not alter system or
process environment settings. Credentials in a proxy URL are private configuration:
never include them in commits or shared diagnostic logs.

Verification: `python -m pytest -q tests/test_download_proxy.py`. Tests replace
network downloads and package updates, and check the installed yt-dlp's actual
proxy selection without sending requests. Live site access still depends on the
user's network, cookies and the site's requirements.

---

### Dubbing Loudness (`docs/dubbing-loudness.md`)

# Dubbing loudness: bounded improvement for issue 533

Issue 533 reports changing perceived distance and disconnected sentences after an
upgrade. Its previous version, exact Fish backend, source artifacts and historical
provider outputs are unknown. This change addresses a reproduced local mechanism,
not a confirmed complete explanation of that report.

## Historical findings

- v2.0.x used the direct Fish endpoint; v2.1.0 switched to the 302 Fish endpoint.
- v2.2.1 applied the configured dub_volume multiplier (default 1.5). Commit eb8ee4c
  replaced it with whole-file RMS normalization, including silence in the average.
- v2.2.1 to v3.0.0 also reduced final video AAC bitrate from 192k to 96k.
- Fish requests and dubbing assembly did not substantively change between v3.0.0
  and v3.0.1. Synthetic processing produced identical decoded samples for those tags.

## Change

Final video assembly measures the dub with FFmpeg's gated integrated loudness
measurement (`loudnorm`). Quiet/silent blocks are excluded by the standard gate,
so silence does not dilute the average as it did with whole-file RMS. Target is
-20 LUFS, with gain capped to keep the measured true peak at or below -1 dBTP.
Peak headroom takes priority over reaching the loudness target.

A second pass applies only a fixed gain, not dynamic compression. Speech dynamics,
silences, sample rate, channel count and timing are preserved. Unmeasurable silence
or very short audio is passed through with zero gain. The temporary mix input is
24-bit PCM; final video AAC uses 192k again. FFmpeg failure now propagates rather
than printing a success message. Analysis requires one additional full audio pass.

This is scoped to the dubbed track during final video assembly. Original ASR/F5
reference normalization is unchanged. Standalone dub.mp3 remains the original
assembly output. The existing no-burn placeholder and audio-only paths are unchanged.
Background gain and mixing are unchanged. This does not equalize separate TTS
sentences, remove reverberation, join sentence requests, restore lost bandwidth or
reproduce an older remote Fish model. More natural prosody still needs actual speech
comparisons before modifying TTS request segmentation.

## Validation

`python -m pytest tests/test_dubbing_loudness.py -q`

Seven tests use real local FFmpeg and synthetic audio: silence-padding stability
(less than 0.3 dB active-level difference), preserved 12 dB dynamics, peak headroom,
silence, short stereo input, error propagation and actual six-second video assembly
with subtitles/background/dubbing. Frame counts, sample rates and channels are
checked, plus final audio/video durations. Network/model calls are not made. This
establishes signal-processing behavior, not subjective speech quality improvement.

---

### Ffmpeg Runtime (`docs/ffmpeg-runtime.md`)

# Managed FFmpeg runtime

The default Qwen installation requires no manual FFmpeg download, package-manager
command, administrator permission, or system PATH change. Run `uv run start.py` as
documented in the README. Its `installer.py` stage installs `static-ffmpeg==3.0`
and fetches FFmpeg and ffprobe automatically. The binaries live inside that
Python environment's `static_ffmpeg/bin` directory.

The provider offers Windows x86_64, Linux x86_64/aarch64 and macOS Intel/Apple
Silicon binaries. VideoLingo's own platform requirements still apply (notably
macOS 14+ Apple Silicon for the default MLX stack). This is automatic provisioning,
not removal of the FFmpeg engine. Setup needs access to PyPI and the provider's
GitHub download URLs. A download failure fails setup with a retry instruction;
rerun `uv run start.py` once connectivity is restored. Partial downloads
without both tools are fetched again, including a read/execute-only executable
left by an interrupted extraction. Normal imports, app launch and `--check`
never download, so installed media tools work offline.

`runtime_libraries.configure_ffmpeg()` prepends the managed directory to the
current process PATH before application dependencies are imported. Child
processes inherit it. This covers Qwen decoding, Demucs' AudioFile, pydub
(including ffprobe), yt-dlp, subtitle rendering, speed adjustment and mixing.
The installer checks both programs, required filters, and MP3/AAC/H.264/PCM
encoders. NVENC remains optional and also requires compatible hardware/drivers.

## Overrides and WhisperX

For a custom deployment, set `VIDEOLINGO_FFMPEG_DIR` to a directory containing
both executables before launch. This explicit override disables managed downloads
and is checked by the same validation. A random system FFmpeg on PATH never
silently overrides the application's managed tools.

Both Qwen and VideoLingo's WhisperX pipeline use the managed CLI. WhisperX
passes decoded waveforms to pyannote and alignment, so it does not need
TorchCodec's FFmpeg shared-library decoder. `installer.py --check` exercises
WhisperX CLI decoding and pyannote waveform handling only when that backend is
selected. It downloads no models. See the [WhisperX guide](pages/docs/whisperx-manual.en-US.md#ffmpeg-runtime)
for the distinction from custom code that passes filenames directly to pyannote.
The optional Windows shared-DLL override remains available for custom usage.

Do not test installation by running `ffmpeg -version` in a new system terminal:
the application intentionally does not modify that terminal's PATH. Run
`uv run start.py --check` instead. It checks the environment used by the launcher.

## Validation

The Windows experiment used a fresh Python 3.13 environment and downloaded
FFmpeg 8.0.1 automatically through static-ffmpeg 3.0. `installer.check_ffmpeg()`
was also run with an empty PATH and a fresh provider cache: download, extraction
and validation all passed. A real one-second H.264 NVENC encode passed on an
RTX 5060 Laptop GPU with the managed executable and no system tools on PATH.
The automated runtime test
starts a child Python with an empty PATH and disables Python socket connections.
It verifies that neither tool exists before initialization, then checks managed
tool discovery, pydub MP3 export/read, ffprobe metadata, atempo conversion,
Qwen's actual audio decoder and yt-dlp's FFmpeg/ffprobe discovery.
The media regression suite additionally covers extraction quality, discontinuous
timestamps, loudness and final subtitle/audio/video synthesis.

Windows validation on 2026-09-27 (Python 3.13.5, Torch 2.8.0+cu128,
qwen-asr 0.0.6, Demucs 4.1.0):

- Full suite with system PATH cleared before managed-runtime initialization:
  **349 passed, 4 skipped, 6 subtests passed**. Skips: two opt-in paid pipeline
  tests, live-server test, and the PowerShell-dependent test with no system PATH.
- `uv pip check`: all installed packages compatible. The real installer health
  check passed with Demucs required and cu128 selected; TorchCodec was absent.
- Real Qwen3-ASR **0.6B** + ForcedAligner-0.6B inference on the public
  [Qwen English sample](https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav):
  15.05125 s input, 36 timestamped words, 17.83 s elapsed including model loading
  in the ASR/aligner calls. Models came from the local Hugging Face cache with
  Hub/Transformers offline mode; system PATH was empty before initialization.
  Word starts were monotonic and all word intervals stayed within the input.
  This validates the real decoder/model/aligner path, not translation APIs or
  subjective transcription accuracy. The 1.7B model was not exercised here.

Run the targeted offline checks after setup:

```bash
python -m pytest tests/test_ffmpeg_runtime.py tests/test_dependencies.py tests/test_qwen_asr_local.py tests/test_audio_extract_quality.py tests/test_audio_timeline.py tests/test_dubbing_loudness.py
```

macOS validation on 2026-09-27 used an Apple Silicon arm64 Mac (macOS 26.6.2,
16 GiB) with a new Python 3.13.15 environment. The documented `setup_env.py`
flow, using `--path` to keep the environment in a separate verification directory,
installed Torch 2.8.0, mlx-audio 0.5.6, Demucs 4.1.0 and static-ffmpeg 3.0.
`installer.py` downloaded and extracted both tools without Homebrew or a manual
FFmpeg installation. The macOS provider archive supplied **FFmpeg 7.0** (the
provider's URL contains `v8.0`, but the binaries report 7.0); `file` and `lipo`
identified both as native Mach-O arm64, and they executed without changing macOS
security settings. With a child process PATH limited to system command directories,
neither tool was discoverable before `configure_ffmpeg()`. Afterwards, both resolved
inside this environment's `static_ffmpeg/bin/darwin_arm64` directory. The separate
WhisperX environment downloaded its own managed pair to the same package-relative
location. `installer.py --check --require-demucs` passed with network access denied.

The same network-denied process exported and read MP3 through pydub, read metadata
with ffprobe, ran atempo, loudnorm and amix, encoded H.264/AAC, and burned an ASS
subtitle. libass selected `/System/Library/Fonts/Helvetica.ttc`; the subtitle was
visible in the encoded frame. yt-dlp discovered both managed tools. Demucs' real
audio reader/writer path passed the stem alignment tests (separation model mocked).
The full default-environment suite with system FFmpeg excluded passed **354 tests,
6 skipped, 6 subtests passed**. Skips covered two opt-in API pipeline tests, a
live-server test, the optional WhisperX test in the default MLX environment, and
two Windows-only checks. `pip check` found no broken requirements.

The default **MLX Qwen3-ASR 1.7B** model and ForcedAligner-0.6B, reused from a
complete local Hugging Face cache, processed the 15.051 s public English sample
with network access denied: 37 words with valid monotonic timestamps, 7.95 s for
the backend call. The managed FFmpeg CLI decoded the sample. The separate stable
WhisperX 3.8.6 environment used Torch 2.8.0, pyannote-audio 4.0.7 and TorchCodec
0.7.0. Importing TorchCodec's native decoder failed because `libavutil.59.dylib`
was absent, while `check_whisperx_runtime()` passed with an in-memory waveform.
The optional-backend regression test also passed in that separate environment
with TorchCodec imports deliberately blocked.
The real tiny.en CPU backend then ran its default pyannote VAD and English alignment
on that sample: 35 timestamped words. Once tiny.en and the alignment model were
cached in the separate verification directory, a network-denied repeat passed in
1.49 s. Replacing two unrelated candidate-version transitive packages with
stable tokenizers 0.22.2 and safetensors 0.8.0 still passed in 1.57 s. WhisperX
remains a manual **Python dependency** in a separate Apple
Silicon environment because its Hugging Face Hub constraint conflicts with MLX;
it did not require a manually installed FFmpeg CLI or shared libraries for this
pipeline. No paid translation/TTS API, live YouTube download, NVENC or larger
WhisperX model was exercised. Linux binaries remain untested here.

The macOS run also exposed and fixed an interrupted-cache edge case: the provider's
read/execute-only `ffmpeg` could survive without `ffprobe`, and extraction could
not overwrite it. Setup now removes the incomplete managed pair before fetching
again. A regression test covers this, and a temporary-cache exercise verified
missing-tool and download-failure messages plus successful re-extraction. Only
the new verification cache was modified.

Provider: https://github.com/zackees/static_ffmpeg (version 3.0).
The alternative ffmpeg-binaries 1.1.0 was inspected but not adopted: its Windows
wheel contained FFmpeg 6.0, while static-ffmpeg supplied 8.0.1 in this experiment.

### WhisperX follow-up validation

With WhisperX 3.8.6, pyannote-audio 4.0.7 and TorchCodec 0.7.0 installed in the
same isolated Windows environment, system PATH was cleared before initialization.
Importing `torchcodec.decoders` raised RuntimeError, confirming that no working
shared-library decoder was available. The actual `whisperX_local.transcribe_audio`
entry point then decoded the same 15.05125 s sample using managed FFmpeg 8.0.1,
ran tiny.en on CPU with the default pyannote VAD, and aligned 35 words with
WAV2VEC2_ASR_BASE_960H. ASR took 0.65 s and alignment 1.26 s; the complete call
including model acquisition/loading took 24.68 s. All timestamped word intervals
stayed within the input. The experiment overrode only model/language/device and
configuration persistence, not VAD, decoding, recognition or alignment.
Large/Belle models, WhisperX GPU execution and other operating systems were not
validated in this follow-up. The automated optional-backend regression also
blocks TorchCodec imports explicitly and verifies the new waveform health check.
After this change, the full suite with system PATH cleared passed **350 tests,
4 skipped, 6 subtests passed**. `uv pip check` reported all 238 installed packages
compatible. The real installer health check also passed with WhisperX selected,
Demucs required and the cu128 Torch build.
After rebasing onto #621 at `bf6e4e4`, the same full-suite run passed **356 tests,
4 skipped, 6 subtests passed**.

### Windows verification of the macOS recovery fix

After pulling `814bd78` and `f2e556b`, the interrupted-cache regression failed on
Windows with `PermissionError: [WinError 5]` when deleting the read-only surviving
executable. Recovery now retries Windows permission failures after adding the
owner-write bit, preserving other mode bits. This applies only to the managed
pair and its installation marker during incomplete-cache recovery; the macOS
and Linux unlink path is unchanged.

All six combinations of a surviving ffmpeg/ffprobe/neither and a writable or
read-only marker passed. A separate temporary-cache experiment seeded a
read-only `ffmpeg.exe` and marker, then ran the actual provider download,
extraction and codec/filter validation successfully with system PATH cleared
(FFmpeg 8.0.1). The temporary cache was removed afterward. The full Windows
suite passed **361 tests, 4 skipped, 6 subtests passed**.

---

### Package.Json (`docs/package.json`)

{
  "name": "docs",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "dev": "next",
    "build": "next build",
    "start": "next start"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@next/third-parties": "^14.2.13",
    "@radix-ui/react-accordion": "^1.2.0",
    "@radix-ui/react-icons": "^1.3.0",
    "@radix-ui/react-slot": "^1.1.0",
    "@radix-ui/react-tooltip": "^1.1.3",
    "class-variance-authority": "^0.7.0",
    "clsx": "^2.1.1",
    "framer-motion": "^11.6.0",
    "lucide-react": "^0.446.0",
    "next": "^14.2.13",
    "nextra": "^2.13.4",
    "nextra-theme-docs": "^2.13.4",
    "react": "^18.3.1",
    "react-dom": "^18.3.1",
    "tailwind-merge": "^2.5.2",
    "tailwindcss-animate": "^1.0.7"
  },
  "devDependencies": {
    "@types/node": "22.7.0",
    "autoprefixer": "^10.4.20",
    "postcss": "^8.4.47",
    "tailwindcss": "^3.4.13",
    "typescript": "^5.6.2"
  }
}

---

### Runtime Compatibility (`docs/runtime-compatibility.md`)

# Runtime compatibility follow-ups

This change addresses distinct compatibility gaps discussed in PR #590 without
hardcoded personal network settings or unrelated provider/translation changes.

## Whisper model cache

This section applies to the manual WhisperX backend (`whisper.backend: whisperx`).
The default Qwen3-ASR backend uses a complete local copy at
`model_dir/<last part of the repo id>` (a directory containing `config.json`, e.g.
`_model_cache/Qwen3-ASR-1.7B`) when present, and otherwise the standard Hugging Face
cache/Hub resolution of qwen-asr or mlx-audio, which honors `HF_ENDPOINT`.

For WhisperX: resolve complete explicit local directories, then the configured project Hub
cache and the global Hub cache with `local_files_only=True`. Pass the resulting
directory to WhisperX. If neither cache has a complete snapshot, fetch missing
files with the installed loader's standard endpoint configuration. Preserve model
aliases by letting faster-whisper resolve them rather than constructing repository
names. An incomplete explicitly selected directory fails clearly.

Remove per-segment ICMP mirror races, late global endpoint mutation, and the fixed
obsolete Torch warning. Missing project cache is normal when global cache is
complete. This does not keep models resident in GPU memory or guarantee that
other WhisperX components (such as alignment models) need no downloads.

## Launcher logs

Initialize UTF-8 without BOM and stream subsequent lines through the same encoding
with Windows PowerShell 5.1. The logger reads stdin incrementally and flushes each
line. It uses the LOGFILE environment variable so paths are not injected into
PowerShell expressions. Both Conda and shared-venv launch paths use it. Existing
mixed-encoding log files are not rewritten.

## GPT-SoVITS startup

Use the read-only FastAPI schema endpoint instead of `/ping`, which is absent in
upstream api_v2.py. Require synthesis and both model-selection routes. An occupied
port alone, an unrelated 200 response, a redirect, or an HTTP error is insufficient.
Requests bypass environment proxies for loopback, have explicit timeouts, and do
not call control or synthesis endpoints. Detect a child process that exits during
startup. Use Popen's cwd rather than changing the entire application's directory.

Schema availability proves API shape, not successful synthesis. Custom builds
with OpenAPI disabled are not accepted automatically. The initial wait remains
50 seconds with bounded probes (requests timeouts are per socket operation, not
a strict whole-operation deadline). No existing service is restarted by checks.

## Legacy task tables

Parse task-cell Python literals with ast.literal_eval, accepting only finite
numeric `np.float32(...)` and `np.float64(...)` wrappers from older NumPy-generated
tables. Other function calls, imports and expressions are rejected. New tables
continue to contain plain floats as fixed in PR #599. Both dubbing generation
and final audio assembly readers use the same parser. No historical files are
migrated or overwritten.

## Verification

`python -m pytest -q tests/test_runtime_compatibility.py tests/test_download_proxy.py`

Tests cover cache selection without models/network, proxy discovery/overrides,
legacy literals and rejected executable expressions, SoVITS schema/status/child
exit behavior, and an actual Windows PowerShell UTF-8 stdin/console/file roundtrip
using the launcher's header command. They do not start Streamlit, SoVITS, download
media, install dependencies or invoke paid APIs. Real synthesis and live site
downloads remain runtime checks for users of those optional integrations.

---

### Runtime Dependencies (`docs/runtime-dependencies.md`)

# Runtime dependency refresh

> BUILDER-305 update: the default local ASR is now Qwen3-ASR + Qwen3-ForcedAligner.
> WhisperX is not an installer option and is not in `requirements.txt`. See "Default ASR
> dependencies" below; the WhisperX notes in the rest of this page describe the
> #599 refresh and apply only after you install WhisperX yourself.

This change retains the existing WhisperX, Streamlit, subtitle and dubbing
architecture. It updates application dependencies within compatible API ranges,
without vendored libraries, package-metadata overrides or a CUDA 13 requirement.

## Installation and updates

`uv run setup_env.py --shared` creates/reuses the shared environment using Python
3.12. The installer checks for Python 3.12. Recreating an environment with a
different Python version requires confirmation unless `--yes` is supplied.

Use `uv run --python .venv installer.py --upgrade` for the project environment or
`uv run setup_env.py --shared --upgrade` for a shared environment to refresh dependencies within the declared
ranges. Normal startup checks do not unconditionally upgrade dependencies.
`OneKeyStart.bat --check-only` reports health without repairing or launching the app.

### Why the GPU stack remains matched (#599; WhisperX only)

This section records the #599 policy for the WhisperX stack. Since BUILDER-305 it
applies only when you have manually installed WhisperX on Windows/Linux; the
default Qwen3-ASR stack uses Transformers 5 on Apple Silicon (see below).

WhisperX 3.8.6 requires Torch/torchaudio 2.8, torchvision 0.23, TorchCodec 0.6-0.7
and huggingface-hub below 1. Transformers 5 requires hub 1 or newer. Consequently,
this refresh retains Transformers 4 and the matched Torch stack rather than
forcing incompatible latest releases.

The installer selects CUDA 12.8 wheels on drivers supporting CUDA 12.8 or newer,
including CUDA 13-capable drivers, and CUDA 12.6 otherwise. With WhisperX installed,
CTranslate2's Windows build also needs CUDA 12 cuBLAS. Drivers can be newer than the runtime used by the app.
CPU wheels are explicitly selected on non-NVIDIA Windows/Linux systems.
FFmpeg and ffprobe are installed automatically by `installer.py` using
`static-ffmpeg==3.0`. Setup downloads the platform's static binaries once and
checks required filters and encoders. Startup and `--check` never download.
The runtime prepends the managed directory to the current process PATH, so
pydub, yt-dlp, Qwen, Demucs and child processes use the same tools.
No system FFmpeg installation or permanent PATH change is needed.
See [managed FFmpeg](ffmpeg-runtime.md) for troubleshooting and validation.
WhisperX also uses managed CLI decoding and passes in-memory waveforms to pyannote;
see the [WhisperX guide](pages/docs/whisperx-manual.en-US.md#ffmpeg-runtime).

### Default ASR dependencies (BUILDER-305)

`requirements.txt` selects the Qwen3-ASR engine with environment markers:

| Platform | Packages | Transformers / Hub |
| --- | --- | --- |
| Apple Silicon (`darwin` + `arm64`) | `mlx-audio>=0.5.5,<0.6`, plus `nagisa==0.2.11` and `soynlp==0.0.493` | `transformers>=5.14,<6`, `huggingface-hub>=1,<2` |
| Windows, Linux and Intel Mac | `qwen-asr==0.0.6` | `transformers>=4.57.6,<5`, `huggingface-hub>=0.36.2,<1` |

Qwen uses the bundled CPU model from `silero-vad>=6.2.3,<6.3` to trim non-speech
edges from its existing windows before transcription and alignment. Both tracks
keep identical sample boundaries and original time offsets. One second of padding
protects short or quiet opening words; internal pauses remain intact. Windows
without detected speech are retained rather than silently discarded. The model
ships in the Python package, so this step needs no separate model download.

Intel Macs use the last macOS x86_64 PyTorch wheel family: Torch/torchaudio 2.2.2,
torchvision 0.17.2, NumPy 1.26 and Numba below 0.63, with Qwen3-ASR on CPU.
The optional Demucs package is skipped on Intel Macs because its `sphn` dependency
has no x86_64 macOS wheel. Dubbing still works with reference clips from the original audio, but the final video does not retain the original background sound unless separation is installed manually.
Other platforms retain the PyTorch 2.8 family. Installation still stops on
Apple Silicon below macOS 14, where mlx has no wheels. On Apple Silicon the
installer uninstalls the WhisperX stack (whisperx, torchcodec, faster-whisper, ctranslate2, pyannote-*) before syncing requirements,
because WhisperX 3.8's `huggingface-hub<1` conflicts with mlx-audio's `>=1`.

Why the split: every mlx-audio release that includes Qwen3-ASR requires
Transformers 5, and 0.4.2+ also requires huggingface-hub 1. qwen-asr 0.0.6 pins
`transformers==4.57.6` and `accelerate==1.12.0` exactly (and pulls in gradio and
flask), so Transformers stays on 4.x wherever qwen-asr is used. mlx only publishes
macOS ≥14 arm64 wheels, so Apple Silicon needs macOS 14 or newer. mlx-audio's
forced aligner imports nagisa (Japanese) and soynlp (Korean) lazily without
declaring them, so they are listed explicitly with qwen-asr's pins.

WhisperX, pyannote-audio, CTranslate2 and TorchCodec are no longer default
requirements. WhisperX retains `huggingface-hub<1` and TorchCodec 0.7 package
constraints, but VideoLingo's waveform path does not need TorchCodec's native
file decoder or FFmpeg shared libraries. `installer.py --check` probes CLI audio
decoding and pyannote waveform handling when local WhisperX is selected.
WhisperX is not an installer option. Install steps are in
[WhisperX (manual install)](pages/docs/whisperx-manual.en-US.md); on Apple Silicon
stable WhisperX 3.8.6 (hub <1) cannot share the MLX environment (hub ≥1).

Verified offline (`uv pip compile`, Python 3.13, 2026-09-24):

- Linux x86_64 and Windows x86_64: qwen-asr 0.0.6, transformers 4.57.6,
  huggingface-hub 0.36.2, accelerate 1.12.0, torch 2.8.0.
- macOS arm64 with `MACOSX_DEPLOYMENT_TARGET=14.0`: mlx-audio 0.5.5, mlx 0.32.2,
  transformers 5.17.0, huggingface-hub 1.33.0, nagisa 0.2.11, soynlp 0.0.493.
  The default macOS 13 target fails because mlx has no wheel for it.
- Linux/Windows defaults plus the four WhisperX packages resolve (whisperx 3.8.6,
  torchcodec 0.7.0, pyannote-audio 4.0.7, transformers 4.57.6, hub 0.36.2). On
  macOS arm64, uv only resolves by selecting the pre-release whisperx 3.8.7rc1.

At the time of the BUILDER-305 review, not verified: installing these environments and running Qwen3-ASR inference
(transformers on CUDA/CPU, MLX on Apple Silicon), GPU memory use and speed, and
the Docker image build with the new requirements.

Intel macOS x86_64 verification on 2026-09-27 used Python 3.12 under Rosetta:
the complete requirements installed, the default installer completed with
managed FFmpeg, `pip check` passed, and Qwen3-ASR/ForcedAligner, Streamlit and
spaCy imported. Engine auto-selection chose transformers on CPU. The quick check
took 0.14 seconds in this environment. This does not cover model-weight download,
an actual transcription, or timing on Intel hardware. Demucs was skipped as
described above.

### Reduced installation complexity

- Remove unused MoviePy and Replicate dependencies.
- Remove the direct resampy requirement; remaining libraries manage their own
  transitive dependencies.
- Let pyannote-audio/WhisperX resolve Lightning instead of separately pinning
  both Lightning distributions.
- Use maintained PyPI Demucs 4.1 instead of a Git development snapshot and
  `--no-deps` workaround. Training extras are not needed for inference.
- Import Demucs only when separation is requested by the transcription stage.
- Use one FFmpeg WAV slicing helper for the two cloud ASR backends instead of
  decoding the entire audio through librosa and re-encoding separately.
- Launch Edge TTS with the active Python interpreter, not a possibly unrelated
  executable on the system PATH.

## Validation

Run `python -m pytest tests/test_dependencies.py` for targeted regression checks.
The suite requires the app dependencies, pytest and FFmpeg.

`tests/test_pipeline_integration.py` is explicitly opt-in and spends API credits.
Set `VIDEOLINGO_TEST_AUDIO` to an English WAV and `VIDEOLINGO_TEST_API_KEY` to a
valid OpenAI-compatible key. Optionally set `VIDEOLINGO_TEST_BASE_URL` and
`VIDEOLINGO_TEST_MODEL`. It tests real translation, subtitle burn-in and Edge TTS
dubbing through the existing background TaskRunner, plus audio-only subtitles.
Generated outputs use pytest temporary directories and the temporary credential
configuration is removed at teardown.

#599 recorded a Windows 11, NVIDIA RTX 4000 Ada (20 GB), Python 3.13.15
environment with FFmpeg 9.0.1 shared libraries and a complete translation/dubbing
pipeline. That FFmpeg 9.0.1 note is superseded: later TorchCodec 0.7 runtime
probing showed FFmpeg 9 can fail to load even when `pip check` is clean.
Custom TorchCodec 0.7 file decoding requires FFmpeg 4–7 shared libraries.
VideoLingo's WhisperX waveform pipeline does not need this decoder; the newer
experiment is recorded in [managed FFmpeg validation](ffmpeg-runtime.md).
The #599 resolved environment had no dependency conflicts (`uv pip check`).
Representative versions: WhisperX 3.8.6, Torch 2.8.0+cu128, CTranslate2 4.8.2,
pyannote-audio 4.0.7, TorchCodec 0.7.0, spaCy 3.8.16, Streamlit 1.63.0,
pandas 3.0.5, NumPy 2.5.3, OpenAI client 3.13.0 and Demucs 4.1.0.

Real integration uses a roughly 15-second public English speech sample,
OpenLux's OpenAI-compatible endpoint with gpt-5.5 for Simplified Chinese
translation, and Edge TTS for Chinese dubbing. It checks generated subtitle,
audio and video artifacts, not subjective translation or dubbing quality.
The public sample is available at
https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav.
With `VIDEOLINGO_TEST_SERVER=1` and the integration variables supplied, the final
suite passed all 15 checks, including a real HTTP/WebSocket first-page render.

Not claimed as live-verified: Linux/macOS runtime, other ASR languages, YouTube
availability, paid 302.ai/ElevenLabs/other TTS accounts, or GPT-SoVITS services.
Cloud ASR slicing is tested locally without claiming successful cloud requests.

---

### Transcription Cache (`docs/transcription-cache.md`)

# Persistent transcription cache

ASR results are cached under `.cache/asr/`, outside `output/` and `history/`.
Reuploading an identical media file reuses recognition even after the working output
has been cleared. Renaming a file does not change its identity. Successful segments
are saved individually so a failed later segment can be retried without repeating
earlier recognition. A complete hit skips recognition model loading. Audio preparation
is retained because later alignment and dubbing stages still require those files.

Identity includes the source file MD5, recognition language, runtime, model setting,
local backend (`qwen` or `whisperx`), Qwen3-ASR model size (`1.7b`/`0.6b`) and the
resolved Qwen engine (`mlx` or `transformers`, never `auto`), vocal separation,
installed ASR package versions (whisperx, faster-whisper, qwen-asr, mlx-audio,
transformers, demucs) and a cache schema version. WhisperX and Qwen results, the two
Qwen sizes and the two engines are therefore never reused for each other. Adding
these fields changed every key once, so entries written before the Qwen migration
are no longer hit and recognition runs again on first use. Schema 2 (Demucs stems
no longer start ~60 ms late, see `docs/audio-timeline.md`) invalidates them again. The
source bytes are hashed, not only the filename or duration. A re-encode is a miss,
even if it sounds identical. Credentials and translation settings are not stored.
Cached results contain transcript text and timestamps: keep this local directory
private. It is already ignored by Git.

Language metadata is restored on every hit. Failed requests, cancellation and
incomplete JSON writes are not reused. Legacy cloud logs named only by time range
are not trusted as cache entries because they do not identify their source media.
Existing `cleaned_chunks.xlsx` retains the normal in-progress step-skipping behavior.
Old archives are not automatically imported into the new cache.

To bypass persistent caching, add `cache: false` under `whisper` in config.yaml.
To reclaim disk space or force rebuilding after changing a custom model in-place,
remove `.cache/asr/` while no task is running. No automatic eviction is performed.
Remote providers can change a model behind the same name: bypass/clear the cache
when a fresh provider result is required. No translation or TTS result caching is
added here; existing LLM response caching is independent.

---

### Tsconfig.Json (`docs/tsconfig.json`)

{
    "compilerOptions": {
      "target": "es5",
      "lib": ["dom", "dom.iterable", "esnext"],
      "allowJs": true,
      "skipLibCheck": true,
      "strict": false,
      "forceConsistentCasingInFileNames": true,
      "noEmit": true,
      "incremental": true,
      "esModuleInterop": true,
      "module": "esnext",
      "moduleResolution": "node",
      "resolveJsonModule": true,
      "isolatedModules": true,
      "jsx": "preserve",
      "baseUrl": ".",
      "paths": {
        "@/*": ["./*"]
      },
      "plugins": [
        {
          "name": "next"
        }
      ],
      "strictNullChecks": true
    },
    "include": [
      "next-env.d.ts",
      "types.d.ts",
      "**/*.ts",
      "**/*.tsx",
      "tailwind.config.js",
      "pages/_meta.js",
      ".next/types/**/*.ts"
    ],
    "exclude": ["node_modules", "components/magicui", "components/ui"]
  }

---

### Pages  App (`docs/pages/_app.mdx`)

import './globals.css'
import { GoogleAnalytics } from '@next/third-parties/google'

export default function App({ Component, pageProps }) {
  return (
    <>
      <Component {...pageProps} />
      <GoogleAnalytics gaId="G-XY5W5DM87Y" />
    </>
  )
}

---

### Pages  Meta.En US.Json (`docs/pages/_meta.en-US.json`)

{
    "index": {
      "type": "page",
      "title": "VideoLingo",
      "display": "hidden",
      "theme": {
        "layout": "raw"
      }
    },
    "docs": {
      "type": "page",
      "title": "Documents"
    }
  }

---

### Pages  Meta.Ja.Json (`docs/pages/_meta.ja.json`)

{
    "index": {
      "type": "page",
      "title": "VideoLingo",
      "display": "hidden",
      "theme": {
        "layout": "raw"
      }
    },
    "docs": {
      "type": "page",
      "title": "ドキュメント"
    }
  }

---

### Pages  Meta.Zh CN.Json (`docs/pages/_meta.zh-CN.json`)

{
    "index": {
      "type": "page",
      "title": "VideoLingo",
      "display": "hidden",
      "theme": {
        "layout": "raw"
      }
    },
    "docs": {
      "type": "page",
      "title": "文档"
    }
  }

---

### Pages Index.En US (`docs/pages/index.en-US.mdx`)

---
title: VideoLingo
---

import Landing from '@/components/landing'

export const getStaticProps = ({ params }) => {
    return Promise.all([
        fetch(`https://api.github.com/repos/Huanshere/VideoLingo`).then(res => res.json()),
        fetch(`https://api.github.com/repos/Huanshere/VideoLingo/contributors?per_page=16`).then(res => res.json())
    ]).then(([repo, stargazers]) => ({
        props: {
            ssg: {
                stars: repo.stargazers_count,
                recentStargazers: stargazers
            }
        },
        revalidate: 60
    }))
}

export default function Component() {
    const landingData = {
        hero: {
            title: "VideoLingo: Connecting Every Frame Across the World",
            description: "Netflix-level subtitle cutting, translation, alignment, and even dubbing - one-click fully automated video localization AI subtitle team",
            videoSrc: "/videos/demo.mp4"
        },
        features: {
            title: "Powerful Features, Unleash Creativity",
            items: [
                {
                    title: 'Intelligent Subtitle Segmentation',
                    description: 'Using NLP and LLM technologies to accurately segment subtitles based on sentence meaning, ensuring each phrase is just right.',
                    icon: 'CheckCircle',
                },
                {
                    title: 'Context-Aware Translation',
                    description: 'GPT summarizes and extracts terminology knowledge base, achieving context-coherent translation, making every sentence natural and fluent.',
                    icon: 'ArrowRight',
                },
                {
                    title: 'Three-Step Translation Process',
                    description: 'Direct translation - Reflection - Paraphrasing, multiple safeguards, rivaling the quality of professional subtitle team translations.',
                    icon: 'CheckCircle',
                },
                {
                    title: 'Precise Subtitle Alignment',
                    description: 'Using Qwen3-ASR with Qwen3-ForcedAligner for word-level timeline subtitle recognition, ensuring every word is accurately synchronized.',
                    icon: 'ArrowRight',
                },
                {
                    title: 'High-Quality Dubbing',
                    description: 'Supports various TTS solutions, including high-quality personalized dubbing with GPT-SoVITS technology, making videos more appealing.',
                    icon: 'CheckCircle',
                },
                {
                    title: 'Developer-Friendly',
                    description: 'Structured file design, convenient for developers to customize and extend functionality. Supports multiple deployment methods.',
                    icon: 'ArrowRight',
                },
            ]
        },
        comments: {
            title: "They're All Using VideoLingo",
            items: [
                {
                    content: "What used to take a whole day now gets done in an hour!",
                    author: "k",
                    title: "Bilibili creator with 300k followers"
                },
                {
                    content: "This dubbing is even more accurate than my own speech, I suddenly have so many fun ideas 🤩",
                    author: "Ah Biao",
                    title: "Xiaohongshu Cantonese creator with 100k followers"
                },
                {
                    content: "I just posted it for fun after work, didn't expect it to blow up so quickly 😂",
                    author: "X",
                    title: "Douyin creator gaining 7k followers daily"
                }
            ]
        },
        faq: {
            title: "Frequently Asked Questions",
            items: [
                {
                    question: "How is the translation quality?",
                    description: "We strictly adhere to Netflix subtitle standards, using the most advanced Claude 3.5 model for multi-step translation."
                },
                {
                    question: "How long does it take to process a video?",
                    answer: "Processing time depends on the length of the video and the selected services. Typically, a 60-minute video takes about 40 minutes to complete translation and dubbing."
                },
                {
                    question: "How is it priced?",
                    answer: "VideoLingo is an open-source project that has already gained 3k+ stars on Github. A commercial version with more features is coming soon~"
                },
            ]
        }
    }

    return <Landing data={landingData} />
}

---

### Pages Index.Ja (`docs/pages/index.ja.mdx`)

---
title: VideoLingo
---

import Landing from '@/components/landing'

export const getStaticProps = ({ params }) => {
    return Promise.all([
        fetch(`https://api.github.com/repos/Huanshere/VideoLingo`).then(res => res.json()),
        fetch(`https://api.github.com/repos/Huanshere/VideoLingo/contributors?per_page=16`).then(res => res.json())
    ]).then(([repo, stargazers]) => ({
        props: {
            ssg: {
                stars: repo.stargazers_count,
                recentStargazers: stargazers
            }
        },
        revalidate: 60
    }))
}

export default function Component() {
    const landingData = {
        hero: {
            title: "VideoLingo: 世界の一コマ一コマをつなぐ",
            description: "Netflixレベルの字幕分割、翻訳、同期、さらに吹き替えまで、ワンクリックで全自動の動画翻訳AIチーム",
            videoSrc: "/videos/demo.mp4"
        },
        features: {
            title: "強力な機能で創造性を解き放つ",
            items: [
                {
                    title: 'インテリジェントな字幕分割',
                    description: 'NLPとLLM技術を使用し、文意に基づいて字幕を正確に分割し、各フレーズが適切であることを保証します。',
                    icon: 'CheckCircle',
                },
                {
                    title: 'コンテキスト認識翻訳',
                    description: 'GPTで用語知識ベースを要約・抽出し、文脈に沿った一貫性のある翻訳を実現。自然で流暢な翻訳を提供します。',
                    icon: 'ArrowRight',
                },
                {
                    title: '3ステップ翻訳プロセス',
                    description: '直接翻訳 - 反省 - 意訳の多重保証で、プロの字幕翻訳チームに匹敵する品質を実現。',
                    icon: 'CheckCircle',
                },
                {
                    title: '精密な字幕同期',
                    description: 'Qwen3-ASR と Qwen3-ForcedAligner を使用して単語レベルのタイムライン字幕認識を行い、すべての単語を正確に同期させます。',
                    icon: 'ArrowRight',
                },
                {
                    title: '高品質な吹き替え',
                    description: 'GPT-SoVITS技術を含む複数のTTSソリューションをサポートし、高品質でパーソナライズされた吹き替えで動画をより魅力的に。',
                    icon: 'CheckCircle',
                },
                {
                    title: '開発者フレンドリー',
                    description: '構造化されたファイル設計で、開発者によるカスタマイズと機能拡張が容易。複数のデプロイメント方法をサポート。',
                    icon: 'ArrowRight',
                },
            ]
        },
        comments: {
            title: "VideoLingoユーザーの声",
            items: [
                {
                    content: "以前は1日かかっていたことが、今では1時間で完了します！",
                    author: "k",
                    title: "ビリビリ動画30万フォロワーの配信者"
                },
                {
                    content: "この吹き替えは私の話し方よりも正確で、たくさんの面白いアイデアが浮かびました🤩",
                    author: "アービョウ",
                    title: "小紅書10万フォロワーの広東語配信者"
                },
                {
                    content: "仕事帰りに単純に遊びで投稿したら、思わぬ大ヒットになりました😂",
                    author: "X",
                    title: "Douyin（TikTok中国版）で1日7000フォロワー増の配信者"
                }
            ]
        },
        faq: {
            title: "よくある質問",
            items: [
                {
                    question: "翻訳の品質はどうですか？",
                    description: "Netflixの字幕基準に厳密に従い、最先端のClaude 3.5モデルを使用して多段階の翻訳を行っています。"
                },
                {
                    question: "1つの動画の処理にどのくらい時間がかかりますか？",
                    answer: "処理時間は動画の長さと選択したサービスによって異なります。通常、60分の動画の翻訳と吹き替えには約40分かかります。"
                },
                {
                    question: "料金はどうなっていますか？",
                    answer: "VideoLingoはオープンソースプロジェクトで、すでにGitHubで3k以上のスターを獲得しています。商用版の発表が間もなく予定されており、さらに多くの機能が追加される予定です。"
                },
            ]
        }
    }

    return <Landing data={landingData} />
}

---

### Pages Index.Zh CN (`docs/pages/index.zh-CN.mdx`)

---
title: VideoLingo
---

import Landing from '@/components/landing'

export const getStaticProps = async ({ params }) => {
    const [repo, stargazers] = await Promise.all([
        fetch(`https://api.github.com/repos/Huanshere/VideoLingo`).then(res => res.json()),
        fetch(`https://api.github.com/repos/Huanshere/VideoLingo/contributors?per_page=16`).then(res => res.json())
    ]);

    return {
        props: {
            ssg: {
                stars: repo.stargazers_count !== undefined ? repo.stargazers_count : null,
                recentStargazers: stargazers
            }
        },
        revalidate: 60
    };
}

export default function Component() {
    const landingData = {
        hero: {
            title: "VideoLingo: 连接世界的每一帧",
            description: "Netflix级字幕切割、翻译、对齐、甚至加上配音，一键全自动视频搬运AI字幕组",
            videoSrc: "/videos/demo.mp4"
        },
        features: {
            title: "强大功能，释放创意",
            items: [
                {
                    title: '智能字幕分割',
                    description: '使用 NLP 和 LLM 技术，根据句意精确分割字幕，确保每一句话都恰到好处。',
                    icon: 'CheckCircle',
                },
                {
                    title: '上下文感知翻译',
                    description: 'GPT 总结提取术语知识库，实现上下文连贯翻译，让每一句翻译都自然流畅。',
                    icon: 'ArrowRight',
                },
                {
                    title: '三步翻译过程',
                    description: '直接翻译 - 反思 - 意译，多重保障，媲美字幕组精翻效果。',
                    icon: 'CheckCircle',
                },
                {
                    title: '精确字幕对齐',
                    description: '使用 Qwen3-ASR 与 Qwen3-ForcedAligner 进行单词级时间轴字幕识别，让每一个字都准确同步。',
                    icon: 'ArrowRight',
                },
                {
                    title: '高质量配音',
                    description: '支持多种 TTS 方案，包括 GPT-SoVITS 技术的高质量个性化配音，让视频更具魅力。',
                    icon: 'CheckCircle',
                },
                {
                    title: '开发者友好',
                    description: '结构化文件设计，方便开发者自定义和扩展功能。支持多种部署方式。',
                    icon: 'ArrowRight',
                },
            ]
        },
        comments: {
            title: "他们都在用 VideoLingo",
            items: [
                {
                    content: "之前要弄一整天，现在一个小时就弄完了！",
                    author: "k",
                    title: "B站30w粉up"
                },
                {
                    content: "这个配音比我说得还准，我一下就有好多好玩的想法了🤩",
                    author: "阿标",
                    title: "小红书10w粉粤语up"
                },
                {
                    content: "下班单纯发着玩，没想到一下就爆了😂",
                    author: "X",
                    title: "抖音日涨7k粉up"
                }
            ]
        },
        faq: {
            title: "常见问题",
            items: [
                {
                    question: "翻译的质量如何？",
                    description: "我们严格参照 Netflix 字幕标准，使用最先进的 Claude 3.5 模型进行多步骤翻译。"
                },
                {
                    question: "处理一个视频需要多长时间？",
                    answer: "处理时间取决于视频的长度和所选的服务。通常，一个 60 分钟的视频完成翻译和配音大约需要 40 分钟。"
                },
                {
                    question: "如何收费呢？",
                    answer: "VideoLingo是开源项目，已经在Github上获得3k+ stars，即将推出商业版，会带来更多功能~"
                },
            ]
        }
    }

    return <Landing data={landingData} />
}

---

### Pages Docs  Meta.En US.Json (`docs/pages/docs/_meta.en-US.json`)

{ 
    "introduction": {
      "title": "Introduction"
    },
    "start": {
      "title": "Start"
    },
    "whisperx-manual": {
      "title": "WhisperX (manual install)"
    },
    "tech": {
      "title": "Tech"
    },
    "docker": {
      "title": "Docker Installation"
    }
  }

---

### Pages Docs  Meta.Ja.Json (`docs/pages/docs/_meta.ja.json`)

{
    "introduction": {
      "title": "紹介"
    },
    "start": {
      "title": "スタート"
    },
    "tech": {
      "title": "技術"
    }
  }

---

### Pages Docs  Meta.Zh CN.Json (`docs/pages/docs/_meta.zh-CN.json`)

{
    "introduction": {
      "title": "介绍"
    },
    "start": {
      "title": "使用文档"
    },
    "whisperx-manual": {
      "title": "WhisperX（手动安装）"
    },
    "tech": {
      "title": "技术文档"
    },
    "docker": {
      "title": "Docker 文档"
    }
  }

---

### Pages Docs Docker.En US (`docs/pages/docs/docker.en-US.md`)

# Docker installation

## Requirements

Use a Linux NVIDIA GPU host with Docker, a compatible driver and the
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
The host does not need the CUDA Toolkit installed separately for this image.
The driver must support the selected image runtime and the GPU itself.

## Build the checked-out source

Run in a clean public checkout before adding credentials to `config.yaml`.
The Dockerfile copies this checkout, including its configuration, rather than
cloning another revision from GitHub. `.dockerignore` excludes local caches,
outputs and private notes, but does not sanitize a modified configuration file.

```bash
docker build -t videolingo .
```

Default: `nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04`, Python 3.12 and PyTorch
cu128. For the matched CUDA 12.6 variant:

```bash
docker build --build-arg CUDA_VERSION=12.6.3 -t videolingo:cu126 .
```

That selects `12.6.3-cudnn-runtime-ubuntu24.04` and cu126 together. No GPU is
required at build time: `setup_env.py` passes an explicit build choice to the
same `installer.py` used on hosts. Other CUDA_VERSION values are rejected.

Both variants use Torch/torchaudio 2.8.0, torchvision 0.23.0 and the same
`requirements.txt` bounds. The image is Linux, so local recognition uses the default
Qwen3-ASR + ForcedAligner through the official qwen-asr package (Transformers 4.57,
Hub <1) on CUDA. WhisperX is not installed in the image and is not an installer option.
To add it yourself (for example in a derived image), follow
[WhisperX (manual install)](whisperx-manual.en-US.md). Demucs 4.1 uses normal dependency resolution.
The installer automatically downloads FFmpeg and ffprobe during the image build.
Ubuntu supplies Noto CJK fonts and image runtime libraries. Extra FFmpeg shared
libraries are needed only if you add WhisperX yourself.

## Run and preserve data

```bash
docker run -d --name videolingo --gpus all -p 127.0.0.1:8501:8501 -v videolingo-output:/app/output -v videolingo-history:/app/history -v videolingo-models:/app/_model_cache -v videolingo-cache:/app/.cache -v videolingo-hf:/root/.cache/huggingface videolingo
```

Open `http://localhost:8501`. Named volumes preserve output, history and model/ASR
caches when the container is replaced. Mount a local `config.yaml` and
`custom_terms.xlsx` separately if those settings must also persist; the files
must exist before mounting, and configuration needs write access for sidebar edits.
For cu126, use `videolingo:cu126` instead of `videolingo`.

Models (Qwen3-ASR 1.7B/0.6B and the ForcedAligner, several GB) are downloaded to
the Hugging Face cache volume on first use, not bundled at build time.
Stop the container with `docker stop videolingo`. Port binding above is local-only;
remote access requires an intentionally configured listening address and access controls.

## Verification scope

The Dockerfile runs the shared installation checks and `pip check` during a build.
Source-level checks do not prove a successful image build or GPU processing.
Third-party prebuilt images are not guaranteed to match this checkout's dependencies.

---

### Pages Docs Docker.Zh CN (`docs/pages/docs/docker.zh-CN.md`)

# Docker 安装

## 环境要求

使用 Linux NVIDIA GPU 主机，安装 Docker、兼容的显卡驱动和
[NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)。
本镜像不要求主机另外安装 CUDA Toolkit，驱动需支持所选镜像运行库和显卡本身。

## 构建当前代码

在填写 `config.yaml` 凭据之前，从干净的公开代码目录构建。
Dockerfile 复制当前目录及配置，不会另行从 GitHub 克隆不同版本。
`.dockerignore` 排除本地缓存、输出和私人笔记，但不会清除已修改配置里的凭据。

```bash
docker build -t videolingo .
```

默认使用 `nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04`、Python 3.12 和 PyTorch
cu128。匹配的 CUDA 12.6 方案：

```bash
docker build --build-arg CUDA_VERSION=12.6.3 -t videolingo:cu126 .
```

这会同时选择 `12.6.3-cudnn-runtime-ubuntu24.04` 和 cu126。构建时不需要 GPU：
`setup_env.py` 把明确的构建选择传给与主机共用的 `installer.py`。
其他 CUDA_VERSION 值会被拒绝。

两个方案均采用 Torch/torchaudio 2.8.0、torchvision 0.23.0 和相同的
`requirements.txt` 约束。镜像是 Linux，本地识别默认使用 Qwen3-ASR + ForcedAligner，
通过官方 qwen-asr 包（Transformers 4.57、Hub <1）在 CUDA 上运行。镜像内不安装
WhisperX，它也不是安装器选项。若要自己追加（例如在派生镜像中），见
[WhisperX（手动安装）](whisperx-manual.zh-CN.md)。Demucs 4.1 使用正常依赖解析。
安装器在镜像构建时自动下载 FFmpeg 和 ffprobe。Ubuntu 提供 Noto CJK 字体和图像运行库。
只有自行追加 WhisperX 时才需要额外安装 FFmpeg 共享库。

## 启动并保留数据

```bash
docker run -d --name videolingo --gpus all -p 127.0.0.1:8501:8501 -v videolingo-output:/app/output -v videolingo-history:/app/history -v videolingo-models:/app/_model_cache -v videolingo-cache:/app/.cache -v videolingo-hf:/root/.cache/huggingface videolingo
```

打开 `http://localhost:8501`。命名卷让输出、历史和模型/识别缓存在替换容器后保留。
如需保留配置和术语表，另行挂载本地 `config.yaml` 和 `custom_terms.xlsx`。
挂载前文件必须存在，侧栏修改配置需要写权限。使用 cu126 时，将镜像名替换为
`videolingo:cu126`。

模型（Qwen3-ASR 1.7B/0.6B 与 ForcedAligner，共数 GB）在首次识别时下载到
Hugging Face 缓存卷，不包含在构建镜像内。执行 `docker stop videolingo` 停止。
上述端口仅监听本机，远程访问需要明确配置监听地址及访问控制。

## 验证范围

Dockerfile 在构建时执行共用安装检查和 `pip check`。源码检查不等于已经成功构建
镜像或完成 GPU 处理。第三方预构建镜像不保证与当前代码的依赖一致。

---

### Pages Docs Introduction.En US (`docs/pages/docs/introduction.en-US.md`)

# VideoLingo: Connecting the World, Frame by Frame

## 🌟 Overview ([Try VideoLingo Now!](https://videolingo.io))

VideoLingo is an all-in-one video translation, localization, and dubbing tool aimed at generating Netflix-quality subtitles. It eliminates stiff machine translations and multi-line subtitles while adding high-quality dubbing, enabling global knowledge sharing across language barriers.

Key features:
- 🎥 YouTube video download via yt-dlp

- **🎙️ Word-level subtitle recognition with Qwen3-ASR + ForcedAligner**

- **📝 NLP and GPT-based subtitle segmentation**

- **📚 GPT-generated terminology for coherent translation**

- **🔄 3-step direct translation, reflection, and adaptation for professional-level quality**

- **✅ Netflix-standard single-line subtitles only**

- **🗣️ Dubbing alignment with GPT-SoVITS and other methods**

- 🚀 One-click startup and output in Streamlit

- 📝 Detailed logging with progress resumption

Difference from similar projects: **Single-line subtitles only, superior translation quality, seamless dubbing experience**

## 🎥 Demo

<table>
<tr>
<td width="50%">

### Russian Translation
---
https://github.com/user-attachments/assets/25264b5b-6931-4d39-948c-5a1e4ce42fa7

</td>
<td width="50%">

### GPT-SoVITS Dubbing
---
https://github.com/user-attachments/assets/47d965b2-b4ab-4a0b-9d08-b49a7bf3508c

</td>
</tr>
</table>

### Language Support

**Input Language Support(more to come):**

🇺🇸 English 🤩 | 🇷🇺 Russian 😊 | 🇫🇷 French 🤩 | 🇩🇪 German 🤩 | 🇮🇹 Italian 🤩 | 🇪🇸 Spanish 🤩 | 🇯🇵 Japanese 😐 | 🇨🇳 Chinese* 😊

> *Local recognition uses Qwen3-ASR (1.7B by default, 0.6B selectable).

**Translation supports all languages, while dubbing language depends on the chosen TTS method.**

## Installation

VideoLingo supports Windows, macOS (Apple Silicon / Intel), and Linux.

### Ask your local AI agent 🤖

If your AI agent can operate your computer, tell it:

> Install and launch GitHub's Huanshere/VideoLingo on my computer.

### Windows: double-click to install 🎉

1. Download **Source code (zip)** from the [latest Release](https://github.com/Huanshere/VideoLingo/releases/latest), extract it to your Desktop or another folder, and open the folder.
2. Double-click `OneKeyStart.bat` and keep the window open. On the first run, it automatically installs uv, Python 3.12, app dependencies, and FFmpeg. An internet connection is required.
3. After installation, VideoLingo opens automatically in your browser. Enter your API URL, key, and model in the sidebar to start using it.

### Install from source (Windows, macOS, Linux)

```bash
git clone https://github.com/Huanshere/VideoLingo.git && cd VideoLingo
uv run start.py
```

To start it later, run `uv run start.py` again from the VideoLingo folder.

### Docker
For Linux NVIDIA containers, use Docker with a compatible driver and NVIDIA Container Toolkit. The image uses the same Python 3.12 setup; see [Docker docs](/docs/pages/docs/docker.en-US.md):

```bash
docker build -t videolingo .
docker run -d -p 8501:8501 --gpus all videolingo
```

## API
The project supports OpenAI-Like API format and various dubbing interfaces:
- Choose an OpenAI-compatible Chat Completions provider and a model capable of returning structured JSON. Set the API URL, key and model in the sidebar.
- `edge-tts`, `openai-tts`, `siliconflow-fishtts`, `fish-tts`, `GPT-SoVITS`

For detailed installation, LLM configuration, and usage instructions, please refer to the documentation: [English](/docs/pages/docs/start.en-US.md) | [中文](/docs/pages/docs/start.zh-CN.md)

## Current Limitations

1. Transcription and word timestamps may be affected by background noise. For videos with loud background music, enable Voice Separation Enhancement: Qwen3-ASR still transcribes the original audio, while the ForcedAligner aligns against the separated vocals. Punctuation is re-attached to aligned words heuristically; in rare cases a word may lose its punctuation.

2. Using weaker models can lead to errors during intermediate processes due to strict JSON format requirements for responses. If this error occurs, please delete the `output` folder and retry with a different LLM, otherwise repeated execution will read the previous erroneous response causing the same error.

3. The dubbing feature may not be 100% perfect due to differences in speech rates and intonation between languages, as well as the impact of the translation step. However, this project has implemented extensive engineering processing for speech rates to ensure the best possible dubbing results.

4. **Multilingual video transcription recognition is only reliable for the main language**. Alignment runs with one language per audio window, so text and timing in other languages are not guaranteed.

5. **Cannot dub multiple characters separately**; recognition does not distinguish speakers.

## 📄 License

This project is licensed under the Apache 2.0 License. Special thanks to the following open source projects for their contributions:

[Qwen3-ASR](https://github.com/Qwen/Qwen3-ASR), [MLX Audio](https://github.com/Blaizzy/mlx-audio), [whisperX](https://github.com/m-bain/whisperX), [yt-dlp](https://github.com/yt-dlp/yt-dlp), [json_repair](https://github.com/mangiucugna/json_repair), [BELLE](https://github.com/LianjiaTech/BELLE)

## 📬 Contact Us

- Join our Discord: https://discord.gg/9F2G92CWPp
- Submit [Issues](https://github.com/Huanshere/VideoLingo/issues) or [Pull Requests](https://github.com/Huanshere/VideoLingo/pulls) on GitHub
- Follow me on Twitter: [@Huanshere](https://twitter.com/Huanshere)
- Email me at: team@videolingo.io

## ⭐ Star History

[](https://star-history.com/#Huanshere/VideoLingo&Timeline)

---

<p align="center">If you find VideoLingo helpful, please give us a ⭐️!</p>

---

### Pages Docs Introduction.Zh CN (`docs/pages/docs/introduction.zh-CN.md`)

# VideoLingo: 连接世界的每一帧

**QQ 群：875297969**

## 🌟 简介（[在线体验！](https://videolingo.io)）

VideoLingo 是一站式视频翻译本地化配音工具，能够一键生成 Netflix 级别的高质量字幕，告别生硬机翻，告别多行字幕，还能加上高质量的克隆配音，让全世界的知识能够跨越语言的障碍共享。

主要特点和功能：
- 🎥 使用 yt-dlp 从 Youtube 链接下载视频

- **🎙️ 使用 Qwen3-ASR + ForcedAligner 进行单词级时间轴字幕识别**

- **📝 使用 NLP 和 GPT 根据句意进行字幕分割**

- **📚 GPT 总结提取术语知识库，上下文连贯翻译**

- **🔄 三步直译、反思、意译，媲美字幕组精翻效果**

- **✅ 按照 Netflix 标准检查单行长度，绝无双行字幕**

- **🗣️ 使用 GPT-SoVITS 等方法对齐克隆配音**

- 🚀 整合包一键启动，在 streamlit 中一键出片

- 📝 详细记录每步操作日志，支持随时中断和恢复进度

与同类项目相比的优势：**绝无多行字幕，最佳的翻译质量，无缝的配音体验**

## 🎥 效果演示

<table>
<tr>
<td width="50%">

### 俄语翻译
---
https://github.com/user-attachments/assets/25264b5b-6931-4d39-948c-5a1e4ce42fa7

</td>
<td width="50%">

### GPT-SoVITS配音
---
https://github.com/user-attachments/assets/47d965b2-b4ab-4a0b-9d08-b49a7bf3508c

</td>
</tr>
</table>

### 语言支持

**输入语言支持：**

🇺🇸 英语 🤩  |  🇷🇺 俄语 😊  |  🇫🇷 法语 🤩  |  🇩🇪 德语 🤩  |  🇮🇹 意大利语 🤩  |  🇪🇸 西班牙语 🤩  |  🇯🇵 日语 😐  |  🇨🇳 中文* 😊

> *本地识别使用 Qwen3-ASR（默认 1.7B，可选 0.6B）。

**翻译语言支持所有语言，配音语言取决于选取的TTS。**

## 安装

VideoLingo 支持 Windows、macOS（Apple Silicon / Intel）和 Linux。

### 让本地 AI Agent 帮你安装 🤖

如果你的 AI Agent 可以操作这台电脑，直接告诉它：

> `帮我安装 GitHub 上的 Huanshere/VideoLingo，并启动它。`

### Windows：双击安装 🎉

1. 从[最新版本页面](https://github.com/Huanshere/VideoLingo/releases/latest)下载 **Source code (zip)**，解压到桌面等方便找到的位置，并打开文件夹。
2. 双击 `OneKeyStart.bat`，保持窗口打开。首次运行会自动安装 uv、Python 3.12、应用依赖和 FFmpeg，需要联网。
3. 安装完成后，VideoLingo 会自动在浏览器中打开。在侧栏填写 API 地址、密钥和模型，就可以开始使用了。

### 从源码安装（Windows、macOS、Linux）

```bash
git clone https://github.com/Huanshere/VideoLingo.git && cd VideoLingo
uv run start.py
```

以后在 VideoLingo 文件夹中运行 `uv run start.py` 即可启动。

### Docker
Linux NVIDIA 容器使用 Docker、兼容驱动和 NVIDIA Container Toolkit。镜像采用相同的 Python 3.12 安装流程，详见 [Docker 文档](/docs/pages/docs/docker.zh-CN.md)：

```bash
docker build -t videolingo .
docker run -d -p 8501:8501 --gpus all videolingo
```

## API
本项目支持 OpenAI-Like 格式的 api 和多种配音接口：
- 自行选择兼容 OpenAI Chat Completions、能返回结构化 JSON 的服务和模型，在侧栏配置 API 地址、密钥和模型。
- `edge-tts`, `openai-tts`, `siliconflow-fishtts`, `fish-tts`, `GPT-SoVITS`

详细的安装、 LLM 配置和使用说明可以参见文档：[English](/docs/pages/docs/start.en-US.md) | [简体中文](/docs/pages/docs/start.zh-CN.md)

## 当前限制
1. 转录和词级时间轴可能受到视频背景声影响。对于背景音乐较大的视频，请开启人声分离增强：Qwen3-ASR 仍对原始音频转写，ForcedAligner 用分离出的人声对齐。对齐后的词会按启发式规则贴回标点，少数情况下个别词可能丢失标点。

2. 使用较弱模型时容易在中间过程报错，这是因为对响应的 json 格式要求较为严格。如果出现此错误，请删除 `output` 文件夹后更换 llm 重试，否则重复执行会读取上次错误的响应导致同样错误。

3. 配音功能由于不同语言的语速和语调差异，还受到翻译步骤的影响，可能不能 100% 完美，但本项目做了非常多的语速上的工程处理，尽可能保证配音效果。

4. **多语言视频转录识别只对主要语言可靠**，对齐时每个音频窗口只使用一种语言，其他语言的文字和时间轴不保证准确。

5. **无法多角色分别配音**，识别环节不区分说话人。

## 📄 许可证

本项目采用 Apache 2.0 许可证，衷心感谢以下开源项目的贡献：

[Qwen3-ASR](https://github.com/Qwen/Qwen3-ASR), [MLX Audio](https://github.com/Blaizzy/mlx-audio), [whisperX](https://github.com/m-bain/whisperX), [yt-dlp](https://github.com/yt-dlp/yt-dlp), [json_repair](https://github.com/mangiucugna/json_repair), [BELLE](https://github.com/LianjiaTech/BELLE)

## 📬 联系我们

- 加入我们的 QQ 群寻求解答：875297969
- 在 GitHub 上提交 [Issues](https://github.com/Huanshere/VideoLingo/issues) 或 [Pull Requests](https://github.com/Huanshere/VideoLingo/pulls)
- 关注我的 Twitter：[@Huanshere](https://twitter.com/Huanshere)
- 联系邮箱：team@videolingo.io

## ⭐ Star History

[](https://star-history.com/#Huanshere/VideoLingo&Timeline)

---

### Pages Docs Start.En US (`docs/pages/docs/start.en-US.md`)

# 🚀 Getting Started

## 📋 API Configuration
VideoLingo uses an LLM for translation. TTS is optional and only needed for dubbing. Choose your own provider and model.

### 1. **Get API_KEY for LLM**:

Set the API URL, key and model in the sidebar. The client uses OpenAI-compatible
Chat Completions, with structured JSON required by several processing steps.
Use a model supported by your endpoint; parameter count alone does not establish
translation quality or JSON reliability. Enable JSON mode only if supported.

[OpenLux](https://www.openlux.ai/register?aff=wKYu) is the recommended relay. Set the API base URL to `https://api.openlux.ai/v1`. Pick one of the three models below. Prices are approximate OpenLux auto-routing group rates measured on the gateway (USD per 1M tokens; they vary by routing group):

| Recommendation | Model ID | Input | Output |
|:---------------|:---------|------:|-------:|
| Default, best value | `gpt-6-luna` | $0.006 | $0.029 |
| Better quality | `gpt-6-sol` | $0.074 | $0.37 |
| Best quality | `claude-opus-5-5` | $0.71 | $3.53 |

Other OpenAI-compatible providers also work, for example OpenRouter at `https://openrouter.ai/api/v1`, or a local compatible server. The application requires a non-empty key field even when that server does not authenticate requests; use a placeholder only for a server that explicitly ignores the key. Edge TTS requires network access and is not an offline synthesizer.

### 2. **TTS API**
VideoLingo provides multiple TTS integration methods. Here's a comparison (skip if only using translation without dubbing)

| TTS Solution | Provider | Pros | Cons | Chinese Effect | Non-Chinese Effect |
|:---------|:---------|:-----|:-----|:---------|:-----------|
| 🎙️ OpenAI TTS | [OpenLux](https://www.openlux.ai/register?aff=wKYu) | Realistic emotions | Chinese sounds foreign | 😕 | 🤩 |
| 🎙️ Fish Audio | [Fish Audio](https://fish.audio) | Clones the voice of the video, or uses any voice of fish.audio | Paid API | 🤩 | 😃 |
| 🎙️ SiliconFlow FishTTS | [SiliconFlow](https://cloud.siliconflow.cn/i/ttKDEsxE) | Voice Clone | Unstable cloning effect | 😃 | 😃 |
| Edge TTS | Online service | No separate API key in this adapter | Requires network access | — | — |
| 🗣️ GPT-SoVITS | Local | Best voice cloning | Only supports Chinese/English, requires local inference, complex setup | 🏆 | 🚫 |

- For SiliconFlow FishTTS, get key from [SiliconFlow](https://cloud.siliconflow.cn/i/ttKDEsxE), note that cloning feature requires paid credits;
- For OpenAI TTS, use [OpenLux](https://www.openlux.ai/register?aff=wKYu) with the same key as for the LLM. Another service with the speech endpoint of OpenAI works with `openai_tts.base_url` and `openai_tts.model` in `config.yaml`;
- For Fish Audio, get the key at [fish.audio](https://fish.audio/app/api-keys/). The API is billed by usage and needs credit in the account; cloning costs nothing extra;
- For F5-TTS, use [302AI](https://gpt302.saaslink.net/C2oHR9). It clones the voice of the video;
> For a custom TTS adapter, edit `core/tts_backend/custom_tts.py`.

<details>
<summary>SiliconFlow FishTTS Tutorial</summary>

Currently supports 3 modes:

1. `preset`: Uses fixed voice, can preview on [Official Playground](https://cloud.siliconflow.cn/playground/text-to-speech/17885302608), default is `anna`.
2. `clone(stable)`: API mode `custom`; combines eligible reference segments from the task list, subject to text-length and duration limits, and uploads a reusable voice. This is not necessarily the first ten seconds of the video.
3. `clone(dynamic)`: API mode `dynamic`; uses the current sentence's reference clip. Voice consistency and quality depend on the reference and service, with no guaranteed improvement over custom mode.

</details>

<details>
<summary>How to choose OpenAI voices?</summary>

Voice list can be found on the [official website](https://platform.openai.com/docs/guides/text-to-speech/voice-options), such as `alloy`, `echo`, `nova`, etc. Modify `openai_tts.voice` in `config.yaml`. The default model is `gpt-4o-mini-tts`.

</details>
<details>
<summary>How to choose Fish Audio voices?</summary>

Select the provider `Fish Audio` in the page. There are two modes:

- **Voice of the video (cloned)**: the default. The first 15-30 seconds of speech of the video are the reference of all lines. Nothing is stored in your Fish Audio account. Clone only voices that you are allowed to use.
- **Fixed voice**: the list has two voices, `语彤 Yutong` and `浩然 Haoran`. For any other voice, select `Another voice of fish.audio`, find a voice that you like on [fish.audio](https://fish.audio), open it and paste the address of its page or its ID.

To have your favourite voices in the list, add them with a name and their ID to `fish_audio.voices` in `config.yaml`. The model is `fish_audio.model`, by default `s2.1-pro`.

</details>

<details>
<summary>How to choose Edge TTS voices?</summary>

Edge TTS is the default and needs no API key. Run `edge-tts --list-voices` to list the voices, e.g. `zh-CN-XiaoxiaoNeural` or `en-US-JennyNeural`, and select one of the target language. Modify `edge_tts.voice` in `config.yaml` or in the sidebar.

</details>


<details>
<summary>GPT-SoVITS-v2 Tutorial</summary>

1. Check requirements and download the package from [official Yuque docs](https://www.yuque.com/baicaigongchang1145haoyuangong/ib3g1e/dkxgpiy9zb96hob4#KTvnO).

2. Place `GPT-SoVITS-v2-xxx` and `VideoLingo` in the same directory. **Note they should be parallel folders.**

3. Choose one of the following ways to configure the model:

   a. Self-trained model:
   - After training, `tts_infer.yaml` under `GPT-SoVITS-v2-xxx\GPT_SoVITS\configs` will have your model path auto-filled. Copy and rename it to `your_preferred_english_character_name.yaml`
   - In the same directory as the `yaml` file, place reference audio named `your_preferred_english_character_name_reference_audio_text.wav` or `.mp3`, e.g. `Huanyuv2_Hello, this is a test audio.wav`
   - In VideoLingo's sidebar, set `GPT-SoVITS Character` to `your_preferred_english_character_name`.

   b. Use pre-trained model:
   - Download my model from [here](https://vip.123pan.cn/1817874751/8137723), extract and overwrite to `GPT-SoVITS-v2-xxx`.
   - Set `GPT-SoVITS Character` to `Huanyuv2`.

   c. Use other trained models:
   - Place `xxx.ckpt` in `GPT_weights_v2` folder and `xxx.pth` in `SoVITS_weights_v2` folder.
   - Following method a, rename `tts_infer.yaml` and modify `t2s_weights_path` and `vits_weights_path` under `custom` to point to your models, e.g.:
  
      ```yaml
      # Example config for method b:
      t2s_weights_path: GPT_weights_v2/Huanyu_v2-e10.ckpt
      version: v2
      vits_weights_path: SoVITS_weights_v2/Huanyu_v2_e10_s150.pth
      ```
   - Following method a, place reference audio in the same directory as the `yaml` file, named `your_preferred_english_character_name_reference_audio_text.wav` or `.mp3`, e.g. `Huanyuv2_Hello, this is a test audio.wav`. The program will auto-detect and use it.
   - ⚠️ Warning: **Please use English for `character_name`** to avoid errors. `reference_audio_text` can be in Chinese. Currently in beta, may produce errors.


   ```
   # Expected directory structure:
   .
   ├── VideoLingo
   │   └── ...
   └── GPT-SoVITS-v2-xxx
       ├── GPT_SoVITS
       │   └── configs
       │       ├── tts_infer.yaml
       │       ├── your_preferred_english_character_name.yaml
       │       └── your_preferred_english_character_name_reference_audio_text.wav
       ├── GPT_weights_v2
       │   └── [your GPT model file]
       └── SoVITS_weights_v2
           └── [your SoVITS model file]
   ```
        
After configuration, select `Reference Audio Mode` in the sidebar (see Yuque docs for details). During dubbing, VideoLingo will automatically open GPT-SoVITS inference API port in the command line, which can be closed manually after completion. Note that stability depends on the base model chosen.</details>

## 🛠️ Quick Start

VideoLingo supports Windows, macOS (Apple Silicon / Intel), and Linux.

### Ask your local AI agent 🤖

If your AI agent can operate your computer, tell it:

> Install and launch GitHub's Huanshere/VideoLingo on my computer.

### Windows: double-click to install 🎉

1. Download **Source code (zip)** from the [latest Release](https://github.com/Huanshere/VideoLingo/releases/latest), extract it to your Desktop or another folder, and open the folder.
2. Double-click `OneKeyStart.bat` and keep the window open. On the first run, it automatically installs uv, Python 3.12, app dependencies, and FFmpeg. An internet connection is required.
3. After installation, VideoLingo opens automatically in your browser. Enter your API URL, key, and model in the sidebar to start using it.

If you use an NVIDIA GPU, install a compatible driver first; see [GPU runtime](#gpu-runtime).

### Install from source (Windows, macOS, Linux)

```bash
git clone https://github.com/Huanshere/VideoLingo.git && cd VideoLingo
uv run start.py
```

To start it later, run `uv run start.py` again from the VideoLingo folder. Apple Silicon uses MLX; Intel Macs use CPU recognition. By default, Intel Mac dubbing uses the new voice without the original background sound.



<a id="asr-runtime"></a>
### Speech recognition (Qwen3-ASR + ForcedAligner)

Local recognition transcribes with **Qwen3-ASR** and then produces word timestamps with **Qwen3-ForcedAligner-0.6B**. With vocal separation enabled, transcription uses the original audio and alignment uses the separated vocals.

- **Model size**: `whisper.qwen_model` in `config.yaml` (not in the sidebar). `1.7b` (default) is more accurate; `0.6b` is faster and uses less memory. The aligner is always ForcedAligner-0.6B.
- **Engine**: `whisper.qwen_engine: auto` selects it automatically; you normally do not need to change it.

| Platform | Engine | Models | Notes |
|:---------|:-------|:-------|:------|
| Apple Silicon Mac | MLX (mlx-audio) | `mlx-community/Qwen3-ASR-{1.7B,0.6B}-8bit`, `mlx-community/Qwen3-ForcedAligner-0.6B-8bit` | Requires **macOS 14 or newer** (mlx only ships macOS ≥14 arm64 wheels) |
| Windows / Linux + NVIDIA | Official qwen-asr (transformers) | `Qwen/Qwen3-ASR-{1.7B,0.6B}`, `Qwen/Qwen3-ForcedAligner-0.6B` | `cuda:0`, bf16 when the GPU supports it, otherwise fp16 |
| Windows / Linux without NVIDIA | Same | Same | CPU fp32 works but is **slow**; prefer 0.6B or the ElevenLabs runtime |
| Intel Mac | Official qwen-asr (transformers) | `Qwen/Qwen3-ASR-{1.7B,0.6B}`, `Qwen/Qwen3-ForcedAligner-0.6B` | CPU fp32 with PyTorch 2.2.2 installed automatically; recognition is slow, so prefer 0.6B. Optional vocal separation is not installed automatically |

- On Apple Silicon the default requirements install mlx-audio, not qwen-asr; using `qwen_engine: transformers` there needs a separate environment. Below macOS 14 the installer stops with an error. If an older environment has WhisperX, rerunning `installer.py` first uninstalls the WhisperX stack (whisperx, torchcodec, faster-whisper, ctranslate2, pyannote-*), which conflicts with the MLX dependencies.
- **Model downloads**: models are downloaded from Hugging Face on first use (several GB for 1.7B plus the aligner). Set `HF_ENDPOINT` to use a mirror. If `_model_cache/<last part of the repo id>/config.json` exists (for example `_model_cache/Qwen3-ASR-1.7B`), that local copy is used.
- **Languages**: every recognition language in the sidebar is supported (Qwen3-ASR supports 30 languages). With `Auto`, each window of about 3 minutes is first probed with up to three 20 s clips, the clips vote on the language, and the window is then transcribed with that language forced (recognition takes about a third longer); for mixed speech the first language reported is treated as primary. If `Auto` detects a language outside the sidebar list (e.g. Korean, Vietnamese, Thai), sentence splitting picks a spaced or unspaced joiner automatically and uses the matching spaCy pipeline, or punctuation-only splitting when spaCy has none.
- **Degenerate output**: if a window's transcript is one phrase looping, or much shorter than what its probe clips heard, it is retried in 60 s windows; if it is still degenerate, recognition stops with an error asking you to set the language, instead of passing a broken transcript on. With a manually selected language, a window whose transcript is unusually sparse (under 2 letters/digits per second) is checked against a few auto-language probe clips; if they heard far more it fails, and if the probes heard a different language it fails right away with a hint that the selected language may not match the audio (use auto or switch the model size), instead of passing a retry in another language downstream. A transcript whose writing system clearly does not fit the selected language (e.g. Korean text with English or Chinese selected, Japanese with almost no kana) fails the same way; that check reads the text only, adds no recognition time and runs after each window, so it stops at the first mismatching window. Silence and music never trigger this.
- **Known limitation**: choosing the wrong language among languages written in Latin letters (e.g. Spanish selected for an English video) cannot be detected. The model may **translate** some windows into the selected language instead of transcribing what was said, leaving part original and part translation. Prefer `Auto`, or double-check the recognition language before processing.
- The transcription cache distinguishes backend, model size and engine; changing any of them re-runs recognition.
- WhisperX is not an installer option. To use it (including the Belle model for Chinese), install the packages yourself: [WhisperX (manual install)](whisperx-manual.en-US.md).
- The official `qwenllm/qwen3-asr` Docker image can host a standalone Qwen3-ASR service, but VideoLingo does not call it directly.

#### Optional MAI-Transcribe-2 (Azure Speech or OpenRouter)

Choose **ASR Runtime → MAI-Transcribe-2** in the sidebar, then select **Azure Speech** or **OpenRouter** as the MAI provider. Azure uses `whisper.mai_api_key` and `whisper.mai_region` (a region such as `eastus`, a resource endpoint, or blank for detection). OpenRouter uses its own key field, `whisper.mai_openrouter_api_key`; no Azure region is needed. Existing configurations without `whisper.mai_provider` continue to use Azure. The default ASR runtime remains local Qwen3-ASR.

Azure MAI uses the fast transcription API; OpenRouter uses its [dedicated audio transcription API](https://openrouter.ai/docs/guides/overview/multimodal/stt) with model `microsoft/mai-transcribe-2`. Both request clean text and word timestamps. OpenRouter uploads are split into roughly two-minute clips to fit its processing timeout. Audio is sent to the selected cloud provider and may incur charges. Azure MAI-Transcribe-2 is in public preview without an SLA; check that your resource region supports it. Provider and API version distinguish cached transcripts; credentials and resource region are excluded from cache identity. The [contributor's evaluation in #618](https://github.com/Huanshere/VideoLingo/pull/618) compared MAI with WhisperX, not the current Qwen3-ASR default.

<a id="gpu-runtime"></a>
### GPU runtime

- Install a driver compatible with your NVIDIA GPU. `nvidia-smi` reports the driver's CUDA capability, not an installed Toolkit version.
- On hosts, the installer selects PyTorch `cu128` for a reported capability >=12.8, otherwise `cu126` when NVIDIA is detected. An unreadable capability falls back to cu126, which is not a guarantee of compatibility with an old driver. Without NVIDIA it selects CPU packages. Compatible existing packages may be reused.
- The default Qwen3-ASR runs on PyTorch and uses the CUDA runtime bundled with the PyTorch wheels; no separate cuBLAS/cuDNN installation is needed. A manual WhisperX install needs CUDA 12 cuBLAS and cuDNN 9; see [WhisperX (manual install)](whisperx-manual.en-US.md#cuda-runtime).

The installer selects Python wheels; it does not install a system CUDA Toolkit. Newer CUDA 13-capable drivers do not require CUDA 13 Python packages for this project.

## HTTP API

The local HTTP API replaces Excel batch mode and shares the Streamlit pipeline. See the [API guide](https://github.com/Huanshere/VideoLingo/blob/main/docs/api.md).

## 🚨 Common Errors & Pitfalls

1. **'All array must be of the same length' or 'Key Error' during translation**: 
   - Reason 1: Weaker models have poor JSON format compliance causing response parsing errors.
   - Reason 2: LLM may refuse to translate sensitive content.
    Inspect `resp_content`, `resp` and `message` in `output/gpt_log/error.json`. Failed validation is logged separately from successful response caches. Diagnose the failing stage before clearing its cached results.

2. **'Retry Failed', 'SSL', 'Connection', 'Timeout'**: Usually network issues. Solution: Users in mainland China please switch network nodes and retry.

3. **`Qwen ASR engine '...' needs the 'qwen-asr' package`** (or `mlx-audio`): the recognition package is missing. Close VideoLingo, then start it again to repair: double-click `OneKeyStart.bat` on Windows, or run `uv run start.py` for a source installation. On Apple Silicon, if you set `qwen_engine: transformers` manually, change it back to `auto`.

4. **`Qwen3-ASR could not detect the language`** or **`... is still degenerate after retrying`**: `Auto` could not determine the language, or the transcript degenerated (a looping phrase, far too little text). Select the recognition language explicitly in the sidebar and retry, or try the other model size.

5. **CUDA out of memory**: set `whisper.qwen_model` to `0.6b` in `config.yaml`, or close other programs using the GPU.

6. **mlx cannot be resolved / no matching distribution on macOS**: mlx only ships wheels for Apple Silicon on macOS 14 or newer. Upgrade macOS first.

7. **WhisperX errors** (`cublas64_12.dll not found`, segfaults, `Weights only load failed`, TorchCodec, etc.): these only occur with the WhisperX backend selected; see [WhisperX (manual install)](whisperx-manual.en-US.md#common-errors).

8. **spaCy model missing**: VideoLingo normally downloads the model when it is first needed. Check your internet connection, then retry the step.

9. **Torch package versions disagree**: Close VideoLingo, then start it again to repair: double-click `OneKeyStart.bat` on Windows, or run `uv run start.py` for a source installation. Intel Macs use Torch/torchaudio 2.2.2 with torchvision 0.17.2; other platforms use 2.8.0 with 0.23.0. Keep the three packages matched. Demucs is not installed automatically on Intel Macs.

---

### Pages Docs Start.Zh CN (`docs/pages/docs/start.zh-CN.md`)

# 🚀 开始使用

## 📋 API 配置指南
VideoLingo 使用大模型进行翻译，TTS 仅在配音时需要。服务商和模型由用户自行选择。

### 1. **大模型的 API_KEY**：

在侧栏设置 API 地址、密钥和模型。客户端使用兼容 OpenAI 的 Chat Completions，
多个步骤需要结构化 JSON。选择接口实际支持的模型，参数量本身不能证明翻译质量
或 JSON 可靠性。仅在服务支持时开启 JSON 模式。

推荐使用 [OpenLux](https://www.openlux.ai/register?aff=wKYu) 中转，API 地址填 `https://api.openlux.ai/v1`。模型可按下面三档选。价格为 OpenLux 常见自动路由分组下的实测约价（美元 / 百万 token，随分组浮动）：

| 建议 | 模型 ID | 输入 | 输出 |
|:-----|:--------|-----:|-----:|
| 默认，性价比高 | `gpt-6-luna` | $0.006 | $0.029 |
| 质量更好 | `gpt-6-sol` | $0.074 | $0.37 |
| 质量最好 | `claude-opus-5-5` | $0.71 | $3.53 |

也可以使用其他兼容服务，例如 OpenRouter，地址填 `https://openrouter.ai/api/v1`，或使用本地兼容服务。即使服务不验证凭据，应用仍要求密钥字段非空，只有在服务明确忽略密钥时才使用占位值。Edge TTS 需要联网，不是离线语音合成。

### 2. **TTS 的 API**
VideoLingo提供了多种 tts 接入方式，以下是对比（如不使用配音可跳过）

| TTS 方案 | 提供商 | 优点 | 缺点 | 中文效果 | 非中文效果 |
|:---------|:---------|:-----|:-----|:---------|:-----------|
| 🎙️ OpenAI TTS | [OpenLux](https://www.openlux.ai/register?aff=wKYu) | 情感真实 | 中文听起来像外国人 | 😕 | 🤩 |
| 🎙️ Fish Audio | [Fish Audio](https://fish.audio) | 克隆视频里的声音，也可以用 fish.audio 上的任意音色 | API 需付费 | 🤩 | 😃 |
| 🎙️ SiliconFlow FishTTS | [硅基流动](https://cloud.siliconflow.cn/i/ttKDEsxE) | 语音克隆 | 克隆效果不稳定 | 😃 | 😃 |
| Edge TTS | 在线服务 | 此适配器无需单独 API 密钥 | 需要联网 | — | — |
| 🗣️ GPT-SoVITS | 本地 | 最强语音克隆 | 只支持中英文，需要本地训练推理，配置麻烦 | 🏆 | 🚫 |

- SiliconFlow FishTTS 请在 [硅基流动](https://cloud.siliconflow.cn/i/ttKDEsxE) 获取key，注意克隆功能需要付费充值积分；
- OpenAI TTS 使用 [OpenLux](https://www.openlux.ai/register?aff=wKYu)，和 LLM 用同一个 key 即可。其他兼容 OpenAI 语音接口的服务，可以在 `config.yaml` 里修改 `openai_tts.base_url` 和 `openai_tts.model`；
- Fish Audio 请在 [fish.audio](https://fish.audio/app/api-keys/) 获取 key。API 按用量计费，账户里需要有余额，克隆不另外收费；
- F5-TTS 仅支持 [302AI](https://gpt302.saaslink.net/C2oHR9)，克隆视频里的声音；
> 自定义 TTS 适配器位于 `core/tts_backend/custom_tts.py`。

<details>
<summary>SiliconFlow FishTTS 使用教程</summary>

目前支持 3 种模式：

1. `preset`: 使用固定音色，可以在 [官网Playground](https://cloud.siliconflow.cn/playground/text-to-speech/17885302608) 试听，默认 `anna`。
2. `clone(stable)`：API 模式 `custom`，按文本长度和时长限制，从任务列表选择合适的参考片段合并并上传为可复用音色，不一定是视频最初十秒。
3. `clone(dynamic)`：API 模式 `dynamic`，使用当前句子的参考片段。音色一致性和质量取决于参考音频及服务，不保证比 custom 模式更好。

</details>

<details>
<summary>OpenAI 声音怎么选？</summary>

声音列表可以在 [官网](https://platform.openai.com/docs/guides/text-to-speech/voice-options) 找到，例如 `alloy`, `echo`, `nova`等，在 `config.yaml` 中修改 `openai_tts.voice` 即可。默认模型是 `gpt-4o-mini-tts`。

</details>
<details>
<summary>Fish Audio 声音怎么选？</summary>

在页面里选择服务商 `Fish Audio`，有两种模式：

- **视频里的声音（克隆）**：默认模式。取视频开头 15-30 秒的人声作为所有句子的参考，不会在你的 Fish Audio 账户里保存任何东西。请只克隆你有权使用的声音。
- **固定音色**：列表里自带两个音色，`语彤` 和 `浩然`。想用其他音色时，选择 `fish.audio 上的其他音色`，在 [fish.audio](https://fish.audio) 找到喜欢的音色并打开，把页面网址或 ID 粘贴进来即可。

想把喜欢的音色放进列表，在 `config.yaml` 的 `fish_audio.voices` 里添加名字和 ID。模型是 `fish_audio.model`，默认 `s2.1-pro`。

</details>

<details>
<summary>Edge TTS 声音怎么选？</summary>

Edge TTS 是默认的配音方式，不需要 API 密钥。运行 `edge-tts --list-voices` 可以列出所有声音，例如 `zh-CN-XiaoxiaoNeural`、`en-US-JennyNeural`，请选择目标语言的声音，在 `config.yaml` 的 `edge_tts.voice` 或侧栏中修改。

</details>


<details>
<summary>GPT-SoVITS-v2 使用教程</summary>

1. 前往 [官方的语雀文档](https://www.yuque.com/baicaigongchang1145haoyuangong/ib3g1e/dkxgpiy9zb96hob4#KTvnO) 查看配置要求并下载整合包。

2. 将 `GPT-SoVITS-v2-xxx` 与 `VideoLingo` 放在同一个目录下。**注意是两文件夹并列。**

3. 选择以下任一方式配置模型：

   a. 自训练模型：
   - 训练好模型后， `GPT-SoVITS-v2-xxx\GPT_SoVITS\configs` 下的 `tts_infer.yaml` 已自动填写好你的模型地址，将其复制并重命名为 `你喜欢的英文角色名.yaml`
   - 在和 `yaml` 文件同个目录下，放入后续使用的参考音，命名为 `你喜欢的英文角色名_参考音频的文字内容.wav` 或 `.mp3`，例如 `Huanyuv2_你好，这是一条测试音频.wav`
   - 在 VideoLingo 网页的侧边栏中，将 `GPT-SoVITS 角色` 配置为 `你喜欢的英文角色名`。

   b. 使用预训练模型：
   - 从 [这里](https://vip.123pan.cn/1817874751/8137723) 下载我的模型，解压后覆盖到 `GPT-SoVITS-v2-xxx`。
   - 在 `GPT-SoVITS 角色` 配置为 `Huanyuv2`。

   c. 使用其他训练好的模型：
   - 将 `xxx.ckpt` 模型文件放在 `GPT_weights_v2` 文件夹下，将 `xxx.pth` 模型文件放在 `SoVITS_weights_v2` 文件夹下。
   - 参考方法 a，重命名 `tts_infer.yaml` 文件，并修改文件中的 `custom` 部分的 `t2s_weights_path` 和 `vits_weights_path` 指向你的模型，例如：
  
      ```yaml
      # 示例 法 b 的配置：
      t2s_weights_path: GPT_weights_v2/Huanyu_v2-e10.ckpt
      version: v2
      vits_weights_path: SoVITS_weights_v2/Huanyu_v2_e10_s150.pth
      ```
   - 参考方法 a，在和 `yaml` 文件同个目录下，放入后续使用的参考音频，命名为 `你喜欢的英文角色名_参考音频的文字内容.wav` 或 `.mp3`，例如 `Huanyuv2_你好，这是一条测试音频.wav`，程序会自动识别并使用。
   - ⚠️ 警告：**请使用英文命名 `角色名`** ，否则会出现错误。 `参考音频的文字内容` 可以使用中文。目前仍处于测试版，可能产生报错。


   ```
   # 期望的目录结构：
   .
   ├── VideoLingo
   │   └── ...
   └── GPT-SoVITS-v2-xxx
       ├── GPT_SoVITS
       │   └── configs
       │       ├── tts_infer.yaml
       │       ├── 你喜欢的英文角色名.yaml
       │       └── 你喜欢的英文角色名_参考音频的文字内容.wav
       ├── GPT_weights_v2
       │   └── [你的GPT模型文件]
       └── SoVITS_weights_v2
           └── [你的SoVITS模型文件]
   ```
        
配置完成后，注意在网页侧边栏选择 `参考音频模式`（具体原理可以参考语雀文档），VideoLingo 在配音步骤时会自动在弹出的命令行中打开 GPT-SoVITS 的推理 API 端口，配音完成后可手动关闭。注意，此方法的稳定性取决于选择的底模。</details>

## 🛠️ 快速上手

VideoLingo 支持 Windows、macOS（Apple Silicon / Intel）和 Linux。

### 让本地 AI Agent 帮你安装 🤖

如果你的 AI Agent 可以操作这台电脑，直接告诉它：

> `帮我安装 GitHub 上的 Huanshere/VideoLingo，并启动它。`

### Windows：双击安装 🎉

1. 从[最新版本页面](https://github.com/Huanshere/VideoLingo/releases/latest)下载 **Source code (zip)**，解压到桌面等方便找到的位置，并打开文件夹。
2. 双击 `OneKeyStart.bat`，保持窗口打开。首次运行会自动安装 uv、Python 3.12、应用依赖和 FFmpeg，需要联网。
3. 安装完成后，VideoLingo 会自动在浏览器中打开。在侧栏填写 API 地址、密钥和模型，就可以开始使用了。

如果使用 NVIDIA 显卡，先安装兼容的驱动，详见[显卡运行环境](#gpu-runtime)。

### 从源码安装（Windows、macOS、Linux）

```bash
git clone https://github.com/Huanshere/VideoLingo.git && cd VideoLingo
uv run start.py
```

以后在 VideoLingo 文件夹中运行 `uv run start.py` 即可启动。Apple Silicon 自动使用 MLX，Intel Mac 使用 CPU 识别。Intel Mac 默认配音只保留新生成的语音，不保留原视频的背景音。



<a id="asr-runtime"></a>
### 语音识别（Qwen3-ASR + ForcedAligner）

本地识别默认使用 **Qwen3-ASR** 转写，再用 **Qwen3-ForcedAligner-0.6B** 生成词级时间轴。开启人声分离时，转写用原始音频，对齐用分离出的人声。

- **模型大小**：`config.yaml` 的 `whisper.qwen_model`（侧栏不提供）。`1.7b`（默认）更准确；`0.6b` 更快、更省显存。对齐模型固定为 ForcedAligner-0.6B。
- **推理引擎**：`whisper.qwen_engine: auto` 时自动选择，一般不需要改。

| 平台 | 引擎 | 模型 | 说明 |
|:-----|:-----|:-----|:-----|
| Apple Silicon Mac | MLX（mlx-audio） | `mlx-community/Qwen3-ASR-{1.7B,0.6B}-8bit`、`mlx-community/Qwen3-ForcedAligner-0.6B-8bit` | 需要 **macOS 14 或更高**（mlx 只提供 macOS ≥14 的 arm64 包） |
| Windows / Linux + NVIDIA | 官方 qwen-asr（transformers） | `Qwen/Qwen3-ASR-{1.7B,0.6B}`、`Qwen/Qwen3-ForcedAligner-0.6B` | `cuda:0`，显卡支持 bf16 时用 bf16，否则 fp16 |
| Windows / Linux 无 NVIDIA | 同上 | 同上 | CPU fp32，可以运行但**很慢**，建议选 0.6B 或改用 ElevenLabs |
| Intel Mac | 官方 qwen-asr（transformers） | `Qwen/Qwen3-ASR-{1.7B,0.6B}`、`Qwen/Qwen3-ForcedAligner-0.6B` | CPU fp32，自动安装 PyTorch 2.2.2；识别较慢，建议选 0.6B。可选的人声分离暂不自动安装 |

- 在 Apple Silicon 上，默认依赖只安装 mlx-audio，不安装 qwen-asr；要用 `qwen_engine: transformers` 需要另建环境。macOS 低于 14 时安装器会直接报错退出。旧环境里如果装过 WhisperX，重跑 `installer.py` 时会先卸载WhisperX 整套依赖（whisperx、torchcodec、faster-whisper、ctranslate2、pyannote-*），它们与 MLX 依赖冲突。
- **模型下载**：首次识别时从 Hugging Face 下载，1.7B 加对齐模型共数 GB。可用 `HF_ENDPOINT` 指定镜像。如果 `_model_cache/<仓库名末段>/config.json` 存在（例如 `_model_cache/Qwen3-ASR-1.7B`），会直接使用这份本地模型。
- **语言**：侧栏「识别语言」中的语言都受支持（Qwen3-ASR 共支持 30 种语言）。选 `Auto` 时，每个约 3 分钟的窗口先取最多 3 段 20 秒的片段检测语言并投票，再以投票结果强制该语言转写整窗（识别时间约增加三分之一）；中英混说时以第一语言为准。Auto 检测出侧栏以外的语言（如韩语、越南语、泰语）时，后续断句会自动选用空格或无空格拼接，并使用对应的 spaCy 模型（没有时按标点断句）。
- **异常输出检测**：如果某个窗口的结果是同一短语循环重复，或明显少于检测片段听到的内容，会先切成 60 秒的小窗重试；仍然异常时直接报错，提示明确选择识别语言，而不会把残缺的结果交给后续步骤。手动指定语言时，如果某个窗口的文字异常稀少（每秒不到 2 个字母/数字），会额外用 auto 检测几段片段；片段听到的内容远多于转写结果时同样会报错；如果检测片段听出的是另一种语言，会直接报错提示「所选识别语言可能与音频不符，请改用 auto 或切换模型大小」，不会把换了语言的重试结果交给后续步骤。转写文字的书写系统与所选语言明显不符时（例如选了英文或中文却输出韩文，选了日文却几乎没有假名）也会报同样的错误，这个检查只看文字，不增加识别时间，每个窗口转写完就会检查，第一个窗口不符就会停止。静音和纯音乐不会触发报错。
- **已知限制**：同为拉丁字母的语言之间选错（例如英文视频选了西班牙语）无法自动检测。模型可能把部分窗口直接**翻译**成所选语言，而不是按原话转写，结果会半是原文半是译文。请优先使用 `Auto`，或在处理前确认识别语言设置正确。
- 识别结果缓存区分后端、模型大小和引擎，切换任一项都会重新识别。
- WhisperX 不是安装器选项。若要使用（包括中文 Belle 模型），请自行安装依赖，见 [WhisperX（手动安装）](whisperx-manual.zh-CN.md)。
- 也可以使用官方 Docker 镜像 `qwenllm/qwen3-asr` 单独部署 Qwen3-ASR 服务，但 VideoLingo 目前不直接调用它。

#### 可选的 MAI-Transcribe-2（Azure Speech 或 OpenRouter）

在侧栏选择 **ASR Runtime → MAI-Transcribe-2**，再选择 **Azure Speech** 或 **OpenRouter**。Azure 使用 `whisper.mai_api_key` 和 `whisper.mai_region`（例如 `eastus`、资源终结点，或留空自动检测）；OpenRouter 使用单独的密钥输入框 `whisper.mai_openrouter_api_key`，不需要 Azure 区域。旧配置未设置 `whisper.mai_provider` 时仍使用 Azure。默认识别方式仍是本地 Qwen3-ASR。

Azure MAI 使用快速转写 API；OpenRouter 使用[专用音频转写 API](https://openrouter.ai/docs/guides/overview/multimodal/stt) 和 `microsoft/mai-transcribe-2` 模型。两种方式都请求清理后的文本及词级时间戳。OpenRouter 会将音频拆为约两分钟的片段，以适应处理超时限制。音频会发送到选定的云服务商，可能产生费用。Azure MAI-Transcribe-2 目前为无 SLA 的公开预览；请确认资源区域支持该模型。缓存会区分服务商及 API 版本，但不包含密钥或 Azure 区域。[#618 的贡献者评测](https://github.com/Huanshere/VideoLingo/pull/618)比较的是 MAI 与 WhisperX，尚未比较当前默认的 Qwen3-ASR。

<a id="gpu-runtime"></a>
### GPU 运行库

- 安装与 NVIDIA 显卡兼容的驱动。`nvidia-smi` 显示的是驱动支持的 CUDA 版本，不是已安装的 Toolkit 版本。
- 主机安装器在驱动报告 >=12.8 时选择 PyTorch `cu128`，否则在检测到 NVIDIA 时选择 `cu126`。无法读取支持版本时也回退到 cu126，但这不保证旧驱动一定兼容。没有 NVIDIA 则选择 CPU 包，已有兼容包可能直接复用。
- 默认的 Qwen3-ASR 通过 PyTorch 运行，使用 PyTorch 安装包自带的 CUDA 运行库，不需要另外安装 cuBLAS/cuDNN。自行安装的 WhisperX 需要 CUDA 12 cuBLAS 和 cuDNN 9，见 [WhisperX（手动安装）](whisperx-manual.zh-CN.md#cuda-runtime)。

安装器选择的是 Python 包，不会安装系统 CUDA Toolkit。支持 CUDA 13 的新驱动不代表本项目需要 CUDA 13 的 Python 包。

## HTTP API

Excel 批处理已由本地 HTTP API 替代，与 Streamlit 共用处理流程。参见 [API 使用说明](https://github.com/Huanshere/VideoLingo/blob/main/docs/api.md)。

## 🚨 常见报错与踩坑

1. **翻译过程的 'All array must be of the same length' 或 'Key Error'**: 
   - 原因1：弱模型遵循JSON格式能力较弱导致响应解析错误。
   - 原因2：对于敏感内容，LLM可能拒绝翻译。
    检查 `output/gpt_log/error.json` 中的 `resp_content`、`resp`、`message`。校验失败记录与成功响应缓存分开保存，先确定出错步骤，再考虑清除该步骤的缓存。

2. **'Retry Failed', 'SSL', 'Connection', 'Timeout'**: 通常是网络问题。解决方案：中国大陆用户请切换网络节点重试。

3. **`Qwen ASR engine '...' needs the 'qwen-asr' package`（或 `mlx-audio`）**：当前缺少识别包。关闭 VideoLingo 后重新启动即可修复：Windows 双击 `OneKeyStart.bat`，源码安装运行 `uv run start.py`。Apple Silicon 上如果手动设置了 `qwen_engine: transformers`，请改回 `auto`。

4. **`Qwen3-ASR could not detect the language`** 或 **`... is still degenerate after retrying`**：`Auto` 模式下没能识别出语言，或识别结果退化（循环重复、内容过少）。在侧栏明确选择识别语言后重试，也可以换另一个模型大小。

5. **显存不足（CUDA out of memory）**：在 `config.yaml` 里把 `whisper.qwen_model` 改为 `0.6b`；也可以关闭其他占用显卡的程序。

6. **macOS 安装时报 mlx 无法解析 / 没有可用的安装包**：mlx 只提供 macOS 14 及以上的 Apple Silicon 安装包，请先升级系统。

7. **WhisperX 相关报错**（`cublas64_12.dll not found`、段错误、`Weights only load failed`、TorchCodec 等）：只在选择了 WhisperX 后端时出现，见 [WhisperX（手动安装）](whisperx-manual.zh-CN.md#common-errors)。

8. **spaCy 模型缺失**：VideoLingo 首次用到模型时通常会自动下载。检查网络连接，再重试这一步。

9. **Torch 组件版本不一致**：关闭 VideoLingo 后重新启动即可修复：Windows 双击 `OneKeyStart.bat`，源码安装运行 `uv run start.py`。Intel Mac 使用 Torch/torchaudio 2.2.2、torchvision 0.17.2；其他平台使用 2.8.0、0.23.0。三者需要匹配。Intel Mac 不会自动安装 Demucs。

---

### Pages Docs Tech.En US (`docs/pages/docs/tech.en-US.md`)

## Videolingo Video Translation System Technical Documentation

Videolingo is a highly integrated video translation system capable of automatically executing a series of complex operations, including video downloading, audio extraction, speech recognition, text processing, translation, subtitle generation, text-to-speech synthesis, and audio-video synthesis. The system leverages AI technologies (ASR, NLP, LLMs, TTS) and provides both a local HTTP API for serial automation and an interactive web interface using Streamlit for task management and system configuration.

The project has undergone significant refactoring, resulting in a more modular and robust architecture. Core functionalities are now organized into distinct packages and modules, primarily located within the `core` directory and its subdirectories (`asr_backend`, `spacy_utils`, `st_utils`, `tts_backend`, `utils`), alongside `api.py` for HTTP automation and a `translations` directory for internationalization.

For developers, many components within the `core` directory – especially files named with the `_X_*.py` numbering scheme – represent distinct steps in the processing pipeline and can be executed individually for debugging purposes. Intermediate and final outputs are typically stored in the `output` directory, with mechanisms for cleanup and archival to the `history` directory.

The following outlines the core technical modules and workflows:

**1. Installation and Setup:**

*   `install.py`: The primary installation script. Automates dependency installation (including PyTorch with GPU/CPU detection), environment configuration (languages, PyPI mirror), FFmpeg checks, Noto font installation (Linux), and launching the Streamlit application.
*   `setup.py`: A standard Python project setup file, utilizing `setuptools`. Defines project metadata (name, version) and dependencies (read from `requirements.txt`) for packaging and installation via pip.

**2. Video Acquisition Module:**

*   `core/_1_ytdlp.py`: Integrates the `yt-dlp` library to download videos from URLs. Handles `yt-dlp` updates, filename sanitization, resolution selection, and cookie usage for authenticated downloads. Also includes a function to locate a single video file within the output directory.

**3. Audio Processing and Speech Recognition (ASR) Module (`core/asr_backend`):**

*   `core/asr_backend/demucs_vl.py`: Employs the Demucs model (`htdemucs`) to separate audio into vocal and background tracks, improving the quality of subsequent ASR.
*   `core/asr_backend/audio_preprocess.py`: Contains fundamental functions for preparing audio: volume normalization (`pydub`), video-to-audio conversion (`ffmpeg`), silence detection (`ffmpeg`), audio duration calculation (`ffmpeg`), splitting long audio files into manageable segments, processing ASR results into DataFrames, saving results, and storing detected languages.
*   `core/asr_backend/qwen_asr_local.py`: Default local ASR. Transcribes the raw audio with Qwen3-ASR (1.7B/0.6B) and aligns the text against the vocal track with Qwen3-ForcedAligner-0.6B, in windows of at most 180 s cut at low-energy points. Picks MLX (mlx-audio, 8-bit) on Apple Silicon and the official qwen-asr transformers backend elsewhere, re-attaches punctuation that the aligner strips, and returns the same segment/word structure as WhisperX.
*   `core/asr_backend/whisperX_local.py`: Manual backend (`whisper.backend: whisperx`). The installer does not install it; see [WhisperX (manual install)](whisperx-manual.en-US.md). Implements local audio transcription using the WhisperX library. Optimizes performance based on available hardware (GPU/CPU), handles model downloads (with mirror checking), performs transcription and alignment, adjusts timestamps, and manages GPU memory.
*   `core/asr_backend/elevenlabs_asr.py`: Implements audio transcription using the ElevenLabs Speech to Text API, handling audio slicing, API interaction, format conversion (ElevenLabs to Whisper-like format), and temporary file management.
*   `core/_2_asr.py`: Orchestrates audio preparation, optional vocal separation, recognition with local Qwen3-ASR (default), a manually installed WhisperX backend, or ElevenLabs, content-addressed result caching, and result export.

**4. Text Processing and Translation Module (`core`, `core/spacy_utils`):**

*   **Sentence Splitting (`core/spacy_utils`):**
    *   `core/spacy_utils/load_nlp_model.py`: Loads and initializes the appropriate spaCy NLP model based on the detected language, handling model downloads if necessary.
    *   `core/spacy_utils/split_by_mark.py`: Performs initial sentence splitting using spaCy based on punctuation marks, with special handling for dashes and ellipses.
    *   `core/spacy_utils/split_by_comma.py`: Further refines sentence splitting based on commas, utilizing spaCy to analyze grammatical validity.
    *   `core/spacy_utils/split_by_connector.py`: Splits sentences based on linguistic connectors (conjunctions, relative pronouns) using spaCy, supporting multiple languages.
    *   `core/spacy_utils/split_long_by_root.py`: Splits overly long sentences using spaCy's dependency parsing (identifying sentence subjects) and fallback length-based splitting.
    *   `core/_3_1_split_nlp.py`: Orchestrates the spaCy-based splitting process, calling the various splitting functions (`split_by_mark`, `split_by_comma_main`, `split_sentences_main`, `split_long_by_root_main`).
*   **Meaning-Based Splitting and Translation:**
    *   `core/_3_2_split_meaning.py`: Intelligently splits long sentences based on semantics using a GPT model, ensuring shorter and more manageable units for translation and subtitling. Leverages prompts defined in `core/prompts.py`.
    *   `core/_4_1_summarize.py`: Uses an LLM (GPT) to generate summaries of video scripts and extract relevant terms (optionally augmented with custom terms from `custom_terms.xlsx`). Saves results to a JSON file. Leverages prompts defined in `core/prompts.py`.
    *   `core/translate_lines.py`: Implements the core line-by-line translation logic using a GPT model. Employs a two-step approach (fidelity and expressiveness) for high-quality translation, incorporating context prompting and retry mechanisms. Leverages prompts defined in `core/prompts.py`.
    *   `core/_4_2_translate.py`: Manages the overall translation process. Splits text into chunks, gathers context, calls `core/translate_lines.py` for parallel chunk translation, checks translation quality (similarity), aligns timestamps, trims text to fit audio durations, and saves results to Excel.

**5. Subtitle Processing and Synthesis Module (`core`):**

*   `core/_5_split_sub.py`: Splits long translated subtitles into shorter segments suitable for display, using weighted length calculations and GPT-based alignment with source subtitles. Leverages prompts defined in `core/prompts.py`.
*   `core/_6_gen_sub.py`: Generates the final SRT subtitle files. Aligns translated text with source timestamps, cleans text, formats timestamps, handles small gaps, and generates various SRT output formats (source, translated, combined) for display and audio dubbing.
*   `core/_7_sub_into_vid.py`: Merges ("burns") the generated SRT subtitles (source and translated) directly into the video file using `ffmpeg`, with customizable styling and GPU acceleration support. If burning is disabled, creates a placeholder video.

**6. Audio Dubbing Module (`core`, `core/tts_backend`):**

*   `core/_8_1_audio_task.py`: Parses the SRT file, merges short subtitles, cleans the text, trims text based on estimated duration using an LLM, and generates an Excel file (`_8_1_AUDIO_TASK.xlsx`) defining the tasks for the TTS engine. Leverages prompts defined in `core/prompts.py`.
*   `core/_8_2_dub_chunks.py`: Analyzes the audio task file, calculates time gaps and speaking rates, determines optimal cut points for dubbing chunks based on speed and pauses, merges lines where necessary, matches subtitles, and updates the task file.
*   `core/_9_refer_audio.py`: Extracts specific audio segments from the source vocal track based on timestamps defined in the audio task file, creating reference audio files used by certain TTS engines (e.g., GPT-SoVITS, F5-TTS, FishTTS).
*   **TTS Backends (`core/tts_backend`):**
    *   `core/tts_backend/custom_tts.py`: Placeholder/template for integrating custom TTS engines.
    *   `core/tts_backend/edge_tts.py`: Interface to Microsoft Edge TTS using the `edge-tts` command-line tool.
    *   `core/tts_backend/gpt_sovits_tts.py`: Interface to a local GPT-SoVITS server, including server startup logic.
    *   `core/tts_backend/openai_tts.py`: Interface to the OpenAI Text-to-Speech API.
    *   `core/tts_backend/sf_cosyvoice2.py`: Interface to the SiliconFlow CosyVoice2 TTS API, supporting reference audio.
    *   `core/tts_backend/sf_fishtts.py`: Interface to the SiliconFlow Fish TTS API, supporting preset, custom, and dynamic voice modes with reference audio.
    *   `core/tts_backend/fish_audio_tts.py`: Interface to the API of Fish Audio: the voice of the video is cloned from one reference, or a fixed voice is used by its ID.
    *   `core/tts_backend/_302_f5tts.py`: Interface to the 302.ai F5-TTS API, which uses reference audio for voice cloning.
    *   `core/tts_backend/estimate_duration.py`: Provides functions to estimate the speaking duration of text based on syllable counts and punctuation pauses for a given language. Used for audio task generation and subtitle trimming.
    *   `core/tts_backend/tts_main.py`: Central TTS dispatcher. Cleans input text, selects the appropriate TTS backend based on configuration (`load_key("tts_method")`), calls the corresponding TTS function, handles errors using retries and GPT-based text correction, validates audio duration, and saves the output WAV file.
*   `core/_10_gen_audio.py`: Generates individual audio segments using the selected TTS backend via `tts_main.py`. Adjusts the speed of the generated audio using a computed factor (`ffmpeg`) to match target durations specified in the task file, and concatenates segments into chunks. Uses `ThreadPoolExecutor` for parallel processing.
*   `core/_11_merge_audio.py`: Merges the generated and speed-adjusted audio segments (`.wav` files from `output/audio_segments/`) into a single, continuous dubbed audio track (`output/dub.wav`), adding silences according to subtitle timings. Also generates a corresponding SRT file (`output/dub.srt`).
*   `core/_12_dub_to_vid.py`: The final synthesis step for dubbing. Merges the original video, the generated dubbed audio track (`output/dub.wav`), and the separated background music (`output/background.mp3`, if Demucs was used) using `ffmpeg`. Optionally burns subtitles during this process. Includes audio normalization.

**7. Core Utilities and Configuration (`core/utils`):**

*   `core/prompts.py`: Defines standardized prompt templates used to guide the LLM (GPT) for tasks such as sentence splitting, summarization, translation (fidelity/expressiveness), subtitle alignment, and text optimization/correction for TTS.
*   `core/utils/ask_gpt.py`: Provides a robust interface (`ask_gpt` function) for interacting with the OpenAI GPT models. Includes caching (file-based), JSON response repair (`json_repair`), response validation, error handling with retries (`@except_handler`), and logging.
*   `core/utils/config_utils.py`: Utility functions (`load_key`, `update_key`) for loading and updating configuration settings from `config.yaml` using `ruamel.yaml` (preserves formatting) and `threading.Lock` for thread-safe access. Includes `get_joiner` for language-specific text concatenation.
*   `core/utils/decorator.py`: Defines reusable decorators: `except_handler` for adding retry logic and error reporting to functions, and `check_file_exists` for skipping function execution if the output file already exists. Uses `rich` for formatted output.
*   `core/utils/delete_retry_dubbing.py`: Provides a function (`delete_dubbing_files`) to clean up specific intermediate files and directories associated with the dubbing process (e.g., `dub.wav`, `output_dub.mp4`, `output/audio/segs`).
*   `core/utils/onekeycleanup.py`: Implements a `cleanup` function to organize and archive files from the `output` directory into a structured `history` directory based on the video name. Includes filename sanitization and robust file moving/deletion logic.
*   `core/utils/pypi_autochoose.py`: A utility for automatically testing and selecting the fastest PyPI mirror and configuring pip to use it. Uses `rich` for UI.
*   `core/utils/models.py`: Defines constants representing filepaths of various intermediate and output files used throughout the pipeline.
*   `core/__init__.py`, `core/asr_backend/__init__.py`, `core/spacy_utils/__init__.py`, `core/st_utils/__init__.py`, `core/tts_backend/__init__.py`: Package initialization files, defining the public interfaces (`__all__`) for their respective packages/subpackages.
*   `core/__init__.py`: Initializes the main `core` package, exporting key functions and modules from subpackages for easier access.

**8. Shared pipeline and HTTP API:**

* `core/pipeline.py`: One subtitle/dubbing plan shared by Streamlit and the API.
* `core/task_runner.py`: Sequential background execution, progress, pause and cooperative stop.
* `api.py`: Local HTTP input, run, status, stop, file and archive endpoints. Uses the existing `config.yaml`, `output/` and `history/`; replaces Excel batch processing.

**9. Streamlit Interface Module (`core/st_utils`, `st.py`):**

*   `core/st_utils/download_video_section.py`: Implements the Streamlit UI section for selecting the input video, allowing users to download from YouTube (using `core/_1_ytdlp.py`) or upload local files (video or audio, with audio-to-video conversion using `ffmpeg`).
*   `core/st_utils/sidebar_setting.py`: Creates the configuration sidebar in the Streamlit UI. Allows users to set the display language, LLM parameters (API keys, model, base URL), subtitle settings (source/target languages, Demucs toggle, burning toggle), and dubbing settings (TTS method and related parameters such as voice, API key). Loads/saves settings using `core/utils/config_utils.py` and triggers `st.rerun()` on changes. Includes API key validation.
*   `core/st_utils/imports_and_utils.py`: Contains common imports and utility functions for the Streamlit application, such as functions for creating download buttons for zipped subtitle files and CSS styling for buttons.
*   `st.py`: The main entry point for the Streamlit web application. Sets up the page configuration, displays the logo, creates the sidebar using `sidebar_setting.py`, manages the main UI sections (video downloading/uploading via `download_video_section.py`, text processing, audio processing), and triggers the core processing functions (`process_text`, `process_audio`) based on user interaction (button clicks). Uses `st.spinner` to indicate progress during long-running operations.

**10. Internationalization Module (`translations`):**

*   `translations/translations.py`: Implements UI translation functionality. Defines supported display languages, loads translated strings from JSON files based on the selected language (`load_key("display_language")`), and provides a `translate(key)` function to retrieve translated text, falling back to the original key if a translation is missing.

Videolingo automates the complete process from video acquisition to the final generation of videos with translated subtitles and dubbing.  The enhanced modular design allows each step to be more easily run and debugged, provides greater flexibility through multiple backend options (ASR, TTS), and offers improved configuration management and user interfaces for both interactive and API workflows.

---

### Pages Docs Tech.Zh CN (`docs/pages/docs/tech.zh-CN.md`)

## Videolingo 视频翻译系统技术文档

Videolingo 是一个高度集成的视频翻译系统，能够自动执行一系列复杂的操作，包括视频下载、音频提取、语音识别、文本处理、翻译、字幕生成、文本到语音合成以及音视频合成。该系统利用 AI 技术（ASR、NLP、LLMs、TTS），并提供用于串行自动化的本地 HTTP API和使用 Streamlit 的交互式 Web 界面，用于任务管理和系统配置。

该项目已经过重大重构，形成了一个更加模块化和健壮的结构。核心功能现在组织成不同的包和模块，主要位于 `core` 目录及其子目录（`asr_backend`、`spacy_utils`、`st_utils`、`tts_backend`、`utils`）中，以及用于 HTTP 自动化的 `api.py` 和用于国际化的 `translations` 目录。

对于开发人员来说，`core` 目录中的许多组件（尤其是编号为 `_X_*.py` 的文件）代表处理管道中的不同步骤，并且可以单独执行以进行调试。中间输出和最终输出通常存储在 `output` 目录中，并具有清理和归档到 `history` 目录的机制。

以下概述核心技术模块和工作流程：

**1. 安装和设置：**

*   `install.py`: 主要安装脚本。自动化依赖项安装（包括带有 GPU/CPU 检测的 PyTorch）、环境配置（语言、PyPI 镜像）、FFmpeg 检查、Noto 字体安装 (Linux) 以及启动 Streamlit 应用程序。
*   `setup.py`: 标准 Python 项目设置文件，使用 `setuptools`。定义项目元数据（名称、版本）和依赖项（从 `requirements.txt` 读取），用于通过 pip 进行打包和安装。

**2. 视频获取模块：**

*   `core/_1_ytdlp.py`: 集成 `yt-dlp` 库以从 URL 下载视频。处理 `yt-dlp` 更新、文件名清理、分辨率选择以及用于身份验证下载的 cookie 用法。还包括一个在输出目录中查找单个视频文件的函数。

**3. 音频处理和语音识别 (ASR) 模块 (`core/asr_backend`):**

*   `core/asr_backend/demucs_vl.py`: 使用 Demucs 模型 (`htdemucs`) 将音频分离为人声和背景音轨，从而提高后续 ASR 的质量。
*   `core/asr_backend/audio_preprocess.py`: 包含准备音频的基本功能：音量标准化 (`pydub`)、视频到音频的转换 (`ffmpeg`)、静音检测 (`ffmpeg`)、音频时长计算 (`ffmpeg`)、将长音频文件拆分为可管理的片段、将 ASR 结果处理为 DataFrames、保存结果以及存储检测到的语言。
*   `core/asr_backend/qwen_asr_local.py`: 默认的本地 ASR。用 Qwen3-ASR（1.7B/0.6B）转写原始音频，再用 Qwen3-ForcedAligner-0.6B 把文本对齐到人声音轨；音频按不超过 180 秒、在低能量处切分的窗口处理。Apple Silicon 上使用 MLX（mlx-audio，8bit），其他平台使用官方 qwen-asr 的 transformers 后端；把对齐时去掉的标点贴回词上，输出与 WhisperX 相同的 segment/word 结构。
*   `core/asr_backend/whisperX_local.py`: 手动安装的后端（`whisper.backend: whisperx`）。安装器不会安装它，见 [WhisperX（手动安装）](whisperx-manual.zh-CN.md)。使用 WhisperX 库实现本地音频转录。根据可用硬件（GPU/CPU）优化性能，处理模型下载（具有镜像检查），执行转录和对齐，调整时间戳，并管理 GPU 内存。
*   `core/asr_backend/elevenlabs_asr.py`: 使用 ElevenLabs 语音转文本 API 实现音频转录，处理音频切片、API 交互、格式转换（ElevenLabs 到类似 Whisper 的格式）和临时文件管理。
*   `core/_2_asr.py`: 统筹音频准备、可选的人声分离、识别（本地 Qwen3-ASR（默认）、自行安装的 WhisperX 或 ElevenLabs）、按内容寻址的识别结果缓存以及结果导出。

**4. 文本处理和翻译模块 (`core`, `core/spacy_utils`):**

*   **句子拆分 (`core/spacy_utils`):**
    *   `core/spacy_utils/load_nlp_model.py`: 根据检测到的语言加载和初始化适当的 spaCy NLP 模型，如果需要，处理模型下载。
    *   `core/spacy_utils/split_by_mark.py`: 使用 spaCy 基于标点符号执行初始句子拆分，并特殊处理破折号和省略号。
    *   `core/spacy_utils/split_by_comma.py`: 基于逗号进一步细化句子拆分，使用 spaCy 分析语法有效性。
    *   `core/spacy_utils/split_by_connector.py`: 使用 spaCy 基于语言连接词（连词、关系代词）拆分句子，支持多种语言。
    *   `core/spacy_utils/split_long_by_root.py`: 使用 spaCy 的依赖关系解析（识别句子主语）和基于回退长度的拆分来拆分过长的句子。
    *   `core/_3_1_split_nlp.py`: 编排基于 spaCy 的拆分过程，调用各种拆分函数（`split_by_mark`、`split_by_comma_main`、`split_sentences_main`、`split_long_by_root_main`）。
*   **基于含义的拆分和翻译：**
    *   `core/_3_2_split_meaning.py`: 使用 GPT 模型根据语义智能地拆分长句子，确保翻译和字幕的单元更短、更易于管理。利用 `core/prompts.py` 中定义的提示。
    *   `core/_4_1_summarize.py`: 使用 LLM (GPT) 生成视频脚本的摘要并提取相关术语（可以选择使用 `custom_terms.xlsx` 中的自定义术语进行增强）。将结果保存到 JSON 文件。利用 `core/prompts.py` 中定义的提示。
    *   `core/translate_lines.py`: 使用 GPT 模型实现核心的逐行翻译逻辑。采用两步法（忠实性和表达性）进行高质量翻译，结合上下文提示和重试机制。利用 `core/prompts.py` 中定义的提示。
    *   `core/_4_2_translate.py`: 管理整体翻译过程。将文本拆分为块，收集上下文，调用 `core/translate_lines.py` 进行并行块翻译，检查翻译质量（相似性），对齐时间戳，修剪文本以适应音频时长，并将结果保存到 Excel。

**5. 字幕处理和合成模块 (`core`):**

*   `core/_5_split_sub.py`: 将长的翻译字幕拆分为适合显示的较短片段，使用加权长度计算和基于 GPT 的与源字幕的对齐。利用 `core/prompts.py` 中定义的提示。
*   `core/_6_gen_sub.py`: 生成最终的 SRT 字幕文件。将翻译后的文本与源时间戳对齐，清理文本，格式化时间戳，处理小间隙，并为显示和音频配音生成各种 SRT 输出格式（源、翻译、组合）。
*   `core/_7_sub_into_vid.py`: 使用 `ffmpeg` 将生成的 SRT 字幕（源和翻译）直接合并（"烧录"）到视频文件中，具有可自定义的样式和 GPU 加速支持。如果禁用烧录，则创建一个占位符视频。

**6. 音频配音模块 (`core`, `core/tts_backend`):**

*   `core/_8_1_audio_task.py`: 解析 SRT 文件，合并短字幕，清理文本，使用 LLM 根据估计的时长修剪文本，并生成一个 Excel 文件 (`_8_1_AUDIO_TASK.xlsx`)，用于定义 TTS 引擎的任务。利用 `core/prompts.py` 中定义的提示。
*   `core/_8_2_dub_chunks.py`: 分析音频任务文件，计算时间间隙和语速，根据速度和停顿确定配音块的最佳切断点，必要时合并行，匹配字幕，并更新任务文件。
*   `core/_9_refer_audio.py`: 基于音频任务文件中定义的时间戳，从源人声音轨中提取特定的音频片段，创建某些 TTS 引擎（如 GPT-SoVITS、F5-TTS、FishTTS）使用的参考音频文件。
*   **TTS 后端 (`core/tts_backend`):**
    *   `core/tts_backend/custom_tts.py`: 用于集成自定义 TTS 引擎的占位符/模板。
    *   `core/tts_backend/edge_tts.py`: 使用 `edge-tts` 命令行工具的 Microsoft Edge TTS 的接口。
    *   `core/tts_backend/gpt_sovits_tts.py`: 本地 GPT-SoVITS 服务器的接口，包括服务器启动逻辑。
    *   `core/tts_backend/openai_tts.py`: OpenAI 文本转语音 API 的接口。
    *   `core/tts_backend/sf_cosyvoice2.py`: SiliconFlow CosyVoice2 TTS API 的接口，支持参考音频。
    *   `core/tts_backend/sf_fishtts.py`: SiliconFlow Fish TTS API 的接口，支持具有参考音频的预设、自定义和动态语音模式。
    *   `core/tts_backend/fish_audio_tts.py`: Fish Audio 官方 API 的接口，用一段参考克隆视频里的声音，或按 ID 使用固定音色。
    *   `core/tts_backend/_302_f5tts.py`: 302.ai F5-TTS API 的接口，使用参考音频进行语音克隆。
    *   `core/tts_backend/estimate_duration.py`: 提供根据特定语言的音节计数和标点符号停顿来估计文本的说话时长的函数。用于音频任务生成和字幕修剪。
    *   `core/tts_backend/tts_main.py`: 中央 TTS 调度器。清理输入文本，根据配置 (`load_key("tts_method")`) 选择适当的 TTS 后端，调用相应的 TTS 函数，使用重试和基于 GPT 的文本纠正来处理错误，验证音频时长，并保存输出 WAV 文件。
*   `core/_10_gen_audio.py`: 使用选定的 TTS 后端通过 `tts_main.py` 生成单独的音频片段。基于计算的因子调整生成的音频速度 (`ffmpeg`) 以适应任务文件中指定的目标时长，并将片段合并为块。使用 `ThreadPoolExecutor` 处理并行处理。
*   `core/_11_merge_audio.py`: 将生成的和速度调整的音频片段（来自 `output/audio_segments/` 的 `.wav` 文件）合并为单个连续的配音音轨 (`output/dub.wav`)，根据字幕时序添加静音。 还生成相应的 SRT 文件 (`output/dub.srt`)。
*   `core/_12_dub_to_vid.py`: 配音的最终合成步骤。使用 `ffmpeg` 合并原始视频、生成的配音音轨 (`output/dub.wav`) 和分离的背景音乐 (`output/background.mp3`，如果使用了 Demucs)。可选择在此过程中烧录字幕。包括音频标准化。

**7. 核心实用程序和配置 (`core/utils`):**

*   `core/prompts.py`: 定义标准化的提示模板，用于指导 LLM（GPT）完成诸如句子拆分、摘要、翻译（忠实性/表达性）、字幕对齐以及文本优化/校正以进行 TTS 等任务。
*   `core/utils/ask_gpt.py`: 提供一个强大的接口（`ask_gpt` 函数）用于与 OpenAI GPT 模型交互。 包括缓存（基于文件）、JSON 响应修复 (`json_repair`)、响应验证、带重试的错误处理 (`@except_handler`) 和日志记录。
*   `core/utils/config_utils.py`: 实用程序函数 (`load_key`, `update_key`)，用于使用 `ruamel.yaml`（保留格式）和 `threading.Lock` 以线程安全的方式从 `config.yaml` 加载和更新配置设置。包括 `get_joiner` 用于特定语言的文本连接。
*   `core/utils/decorator.py`: 定义可重用的装饰器：`except_handler` 用于向函数添加重试逻辑和错误报告，`check_file_exists` 用于如果输出文件已存在则跳过函数执行。 使用 `rich` 进行格式化的输出。
*   `core/utils/delete_retry_dubbing.py`: 提供一个函数 (`delete_dubbing_files`) 来清理与配音过程相关的特定中间文件和目录（例如，`dub.wav`、`output_dub.mp4`、`output/audio/segs`）。
*   `core/utils/onekeycleanup.py`: 实现 `cleanup` 函数，用于将文件从 `output` 目录组织和归档到基于视频名称的结构化的 `history` 目录中。包括文件名清理和强大的文件移动/删除逻辑。
*   `core/utils/pypi_autochoose.py`: 用于自动测试和选择最快的 PyPI 镜像并配置 pip 以使用它的实用程序。 使用 `rich` 进行 UI。
*   `core/utils/models.py`: 定义表示整个管道中使用的各种中间文件和输出文件的文件路径的常量。
*   `core/__init__.py`, `core/asr_backend/__init__.py`, `core/spacy_utils/__init__.py`, `core/st_utils/__init__.py`, `core/tts_backend/__init__.py`: 包初始化文件，定义其各自包/子包的公共接口 (`__all__`)。
*   `core/__init__.py`: 初始化主 `core` 包，从子包导出关键函数和模块，以便更轻松地访问。

**8. 共用流程和 HTTP API：**

* `core/pipeline.py`：Streamlit 与 API 共用的字幕和配音步骤。
* `core/task_runner.py`：串行后台执行、进度、暂停与协作式停止。
* `api.py`：提供输入、运行、状态、停止、文件和归档接口。保留 `config.yaml`、`output/` 和 `history/`，替代 Excel 批处理。

**9. Streamlit 界面模块 (`core/st_utils`, `st.py`):**

*   `core/st_utils/download_video_section.py`: 实现用于选择输入视频的 Streamlit UI 部分，允许用户从 YouTube 下载（使用 `core/_1_ytdlp.py`）或上传本地文件（视频或音频，使用 `ffmpeg` 进行音频到视频的转换）。
*   `core/st_utils/sidebar_setting.py`: 在 Streamlit UI 中创建配置侧边栏。 允许用户设置显示语言、LLM 参数（API 密钥、模型、基本 URL）、字幕设置（识别/目标语言、Demucs 开关、烧录开关）和配音设置（TTS 方法和相关参数，如语音、API 密钥）。使用 `core/utils/config_utils.py` 加载/保存设置，并在更改时触发 `st.rerun()`。包括 API 密钥验证。
*   `core/st_utils/imports_and_utils.py`: 包含 Streamlit 应用程序的通用导入和实用程序函数，例如创建压缩字幕文件的下载按钮的函数以及按钮的 CSS 样式。
*   `st.py`: Streamlit Web 应用程序的主要入口点。设置页面配置，显示徽标，使用 `sidebar_setting.py` 创建侧边栏，管理主 UI 部分（通过 `download_video_section.py` 进行视频下载/上传，文本处理，音频处理），并根据用户交互（按钮点击）触发核心处理函数（`process_text`，`process_audio`）。使用 `st.spinner` 指示长时间操作期间的进度。

**10. 国际化模块 (`translations`):**

*   `translations/translations.py`: 实现 UI 的翻译功能。 定义支持的显示语言，根据选定的语言 (`load_key("display_language")`) 从 JSON 文件加载翻译字符串，并提供 `translate(key)` 函数来检索翻译后的文本，如果缺少翻译，则回退到原始键。

Videolingo实现了从视频获取到最终生成具有翻译字幕和配音的视频的完整流程自动化。 增强的模块化设计使每个步骤都可以更轻松地运行和调试，通过多个后端选项（ASR、TTS）提供更大的灵活性，并为交互式和批量处理工作流程提供改进的配置管理和用户界面。

---

### Pages Docs Whisperx Manual.En US (`docs/pages/docs/whisperx-manual.en-US.md`)

# WhisperX (manual install)

VideoLingo's default local speech recognition is **Qwen3-ASR + Qwen3-ForcedAligner**; see the [Start guide](start.en-US.md#asr-runtime). WhisperX is still a local backend (`whisper.backend: whisperx`), including the forced Belle model for Chinese, but it is **not an installer option**. `setup_env.py` and `installer.py` do not install it, do not ask whether to install it, and have no flag for it. This page is the only install guide: you install the extra packages yourself.

Consider WhisperX when you want to compare against Qwen, want the punctuation-enhanced Belle Whisper model for Chinese, or already rely on a WhisperX-based workflow.

## Support

| Platform | Same environment as the default? | Notes |
|:---------|:---------------------------------|:------|
| Windows / Linux | Yes | Add it to the VideoLingo environment as shown below. Dependency resolution was verified with `uv pip compile` (Python 3.13); Windows CPU recognition, pyannote VAD and alignment were verified without system FFmpeg or working TorchCodec; Linux inference has not been run |
| Apple Silicon Mac | **No** | The default MLX backend needs `huggingface-hub>=1`; WhisperX 3.8.6 needs `huggingface-hub<1`. Create a **separate environment** for WhisperX instead of installing it into the default one |

## Install

These commands are for you to run. The installer will not run them and will not install WhisperX back later.

On Windows or Linux, run this from the VideoLingo folder after setup. It targets the project `.venv`; for a `--shared` environment, replace `.venv` with that environment's path. Do not run it in the default Apple Silicon environment; use a separate environment, as described below.

```bash
uv pip install --python .venv "whisperx>=3.8.6,<3.9" "ctranslate2>=4.5,<5" "pyannote-audio>=4.0.4,<5" "torchcodec>=0.7,<0.8"
```

These are the same constraints `requirements.txt` used in 3.0.4. WhisperX 3.8 needs Torch/torchaudio 2.8, torchvision 0.23 and Transformers 4, which match the default environment, so the installed PyTorch build does not change.

After enabling the local WhisperX backend below, run `uv run start.py --check`. It checks the environment VideoLingo uses and exercises FFmpeg decoding and pyannote's in-memory audio path; it does not require TorchCodec's file decoder. On Windows/Linux, rerunning `installer.py` or `--upgrade` does not uninstall WhisperX packages you added yourself. On Apple Silicon, rerunning the installer in the default environment removes the incompatible WhisperX stack (see below).

## Enable

In the sidebar choose "ASR Runtime: Local" → "Local ASR Backend: WhisperX (manual install)", or set in `config.yaml`:

```yaml
whisper:
  runtime: 'local'
  backend: 'whisperx'
  model: 'large-v3'   # WhisperX only. large-v3-turbo is rewritten to large-v3
  language: 'en'
```

- `whisper.model` accepts `large-v3`. A name containing `turbo` (including `large-v3-turbo`) is rewritten to `large-v3` before loading, because turbo loops on repeated words in this pipeline. The cache key uses `large-v3` as well.
- With recognition language `zh`, the WhisperX path forces `Huan69/Belle-whisper-large-v3-zh-punct-fasterwhisper` and ignores `whisper.model`.
- If Chinese is detected while the recognition language is not `zh`, the WhisperX path stops and asks you to select Chinese explicitly.
- The transcription cache key includes the backend, so WhisperX and Qwen results are never reused for each other.

<a id="cuda-runtime"></a>
## GPU runtime (CUDA 12 cuBLAS / cuDNN 9)

The default Qwen path only needs the CUDA runtime bundled with PyTorch. WhisperX uses CTranslate2 (faster-whisper), whose GPU execution also needs **CUDA 12 cuBLAS and cuDNN 9** accessible to the process. See [faster-whisper's GPU requirements](https://github.com/SYSTRAN/faster-whisper#gpu) and [CTranslate2 4.5's cuDNN 9 transition](https://github.com/OpenNMT/CTranslate2/releases/tag/v4.5.0).

- If the libraries are missing on Windows, obtain CUDA 12 libraries from NVIDIA's [CUDA 12.8 Update 1 archive](https://developer.nvidia.com/cuda-12-8-1-download-archive) and cuDNN 9 **for CUDA 12** from [NVIDIA](https://developer.nvidia.com/cudnn-downloads). Add the actual DLL directories to PATH and reopen the terminal. Do not invent a cuDNN directory by substituting the PyTorch build tag into an example path.
- On Linux, follow faster-whisper's linked instructions for exposing the installed cuBLAS/cuDNN libraries through `LD_LIBRARY_PATH` before starting Python. The Docker image is based on `cudnn-runtime` and includes these libraries.

<a id="ffmpeg-runtime"></a>
## FFmpeg: no manual installation

WhisperX in VideoLingo reuses the FFmpeg CLI automatically provided by the installer. It decodes files with `whisperx.audio.load_audio`, then passes NumPy audio to transcription/alignment. WhisperX passes an in-memory `waveform` and `sample_rate` to pyannote, which officially supports this even when TorchCodec cannot load. No extra shared FFmpeg build or PATH configuration is required for this pipeline.

This was verified on Windows with WhisperX 3.8.6, pyannote-audio 4.0.7 and TorchCodec 0.7.0: system PATH cleared, managed FFmpeg 8.0.1 enabled, and TorchCodec import confirmed to fail. The actual VideoLingo backend completed a 15.05-second English sample with tiny.en on CPU, pyannote VAD and word alignment (35 words). Large/Belle models and Linux/macOS inference were not exercised in this experiment.

TorchCodec remains installed to satisfy pyannote's Python package dependencies. Its shared-library decoder is only needed if custom code asks TorchCodec or pyannote to read filenames directly. That separate usage requires compatible shared libraries (TorchCodec 0.7 supports FFmpeg 4–7); the managed static executable does not provide them. On Windows, the optional `VIDEOLINGO_FFMPEG_DLL_DIR` override remains available for such custom usage. See [pyannote's audio input implementation](https://github.com/pyannote/pyannote-audio/blob/4.0.4/src/pyannote/audio/core/io.py).

## Apple Silicon: use a separate environment

On Apple Silicon the default requirements install mlx-audio (Transformers 5, `huggingface-hub>=1`), while WhisperX 3.8.6 requires `huggingface-hub<1`, so they cannot share one environment. `uv` only resolves the combination by picking the pre-release whisperx 3.8.7rc1, which is unverified and not recommended.

To use WhisperX on a Mac, create a separate virtual environment for it; do not run the install command above in the default environment. This repository does not ship an installer for that separate environment, and the combination has not been verified. If the default environment already has whisperx (for example after upgrading from an older version), rerunning `installer.py` first uninstalls the WhisperX stack (whisperx, torchcodec, faster-whisper, ctranslate2, pyannote-*) (packages that another installed package still requires are kept). The installer will not install WhisperX again. `installer.py --check` reports the combination as an error until that stack is gone.

<a id="common-errors"></a>
## Common errors

1. **`local_files_only=True`**: The selected local Whisper model or cache is incomplete. Check the model directory for `config.json`, `model.bin` and `tokenizer.json`. An offline lookup cannot download missing weights; a ping alone does not establish model availability.
2. **`cublas64_12.dll not found`**: The process cannot locate CUDA 12 cuBLAS. Check the selected environment, its Torch CUDA build and library search paths using the [GPU runtime](#cuda-runtime) section. A newer driver alone does not supply this DLL.
3. **Whisper model loading segfaults silently**: ctranslate2 version mismatches cuDNN version. Ensure `ctranslate2>=4.5.0` (supports cuDNN 9, which PyTorch 2.6+ ships with).
4. **`RuntimeError: Weights only load failed`**: PyTorch ≥2.6 changed `torch.load` default behavior. Already fixed via monkey-patch in `whisperX_local.py`. If you see this, your code is not up to date.
5. **WhisperX transcription hangs in Streamlit (CPU/GPU idle)**: `librosa.load()` deadlocks in Streamlit's non-main thread. Already fixed by replacing it with `whisperx.audio.load_audio()` (ffmpeg subprocess). If you see this, your code is not up to date.
6. **TorchCodec could not load warning**: This alone does not prevent VideoLingo's WhisperX pipeline from working, because it passes decoded waveforms to pyannote. Run `uv run start.py --check` to verify that audio path. Custom code passing filenames to pyannote needs the separate shared-library decoder described [above](#ffmpeg-runtime).
7. **`WhisperX is not installed`**: `whisper.backend` is `whisperx` but the package is not installed. Recognition stops before preparing audio. Install the packages yourself as above, or set `whisper.backend` back to `qwen`. The installer will not install WhisperX for you.

---

### Pages Docs Whisperx Manual.Zh CN (`docs/pages/docs/whisperx-manual.zh-CN.md`)

# WhisperX（手动安装）

VideoLingo 默认的本地语音识别是 **Qwen3-ASR + Qwen3-ForcedAligner**，见[使用文档](start.zh-CN.md#asr-runtime)。WhisperX 仍然是一个本地后端（`whisper.backend: whisperx`，中文仍强制使用 Belle 模型），但**不是安装器里的选项**。`setup_env.py` 和 `installer.py` 都不会安装它，也不会询问要不要安装，也没有对应开关。本页是唯一的安装说明：额外依赖需要你自己安装。

以下情况可以考虑改用 WhisperX：想和 Qwen 的结果做对比；希望中文使用带标点增强的 Belle Whisper 模型；已有基于 WhisperX 的使用习惯。

## 支持范围

| 平台 | 能否与默认环境共存 | 说明 |
|:-----|:-------------------|:-----|
| Windows / Linux | 可以 | 在 VideoLingo 环境中追加安装，见下文。依赖解析已用 `uv pip compile` 验证（Python 3.13），Windows CPU 已实跑识别、pyannote 语音检测和对齐，无需系统 FFmpeg 或可用的 TorchCodec；Linux 推理未实跑 |
| Apple Silicon Mac | **不可以** | 默认的 MLX 后端需要 `huggingface-hub>=1`，WhisperX 3.8.6 需要 `huggingface-hub<1`。请**另建一个独立环境**使用 WhisperX，不要装进默认环境 |

## 安装

下面的命令由你自己执行。安装器不会代为执行，之后也不会把 WhisperX 装回去。

在 Windows 或 Linux 上完成安装后，从 VideoLingo 文件夹运行下面的命令。它使用项目的 `.venv`；如果使用 `--shared`，请把 `.venv` 换成共享环境的路径。不要在 Apple Silicon 的默认环境里执行；Mac 的独立环境见下文。

```bash
uv pip install --python .venv "whisperx>=3.8.6,<3.9" "ctranslate2>=4.5,<5" "pyannote-audio>=4.0.4,<5" "torchcodec>=0.7,<0.8"
```

这 4 个包的版本约束与 3.0.4 版本中 `requirements.txt` 的约束相同。WhisperX 3.8 需要 Torch/torchaudio 2.8、torchvision 0.23 和 Transformers 4，与默认环境一致，因此不需要改动已装的 PyTorch。

启用本地 WhisperX 后，运行 `uv run start.py --check`，检查 VideoLingo 实际使用的环境，并验证 FFmpeg 解码和 pyannote 内存波形路径，不要求 TorchCodec 文件解码器可用。Windows/Linux 上重跑安装器不会卸载自行添加的 WhisperX；Apple Silicon 默认环境会移除与 MLX 不兼容的 WhisperX 依赖，详见下文。

## 启用

在侧栏选择「语音识别运行环境：本地」→「本地识别后端：WhisperX（手动安装）」，或在 `config.yaml` 中设置：

```yaml
whisper:
  runtime: 'local'
  backend: 'whisperx'
  model: 'large-v3'   # 仅 WhisperX 使用。large-v3-turbo 会被改写成 large-v3
  language: 'en'
```

- `whisper.model` 只接受 `large-v3`。名字里含 `turbo`（包括 `large-v3-turbo`）会在加载前改成 `large-v3`，因为这条管线里 turbo 会循环复读。缓存键同样按 `large-v3` 计算。
- 识别语言设为 `zh` 时，WhisperX 路径会强制使用 `Huan69/Belle-whisper-large-v3-zh-punct-fasterwhisper`，忽略 `whisper.model`。
- 自动检测到中文但识别语言不是 `zh` 时，WhisperX 路径会报错，要求明确选择中文。
- 识别结果缓存的键包含后端，WhisperX 和 Qwen 的结果不会互相复用。

<a id="cuda-runtime"></a>
## GPU 运行库（CUDA 12 cuBLAS / cuDNN 9）

默认的 Qwen 路径只需要 PyTorch 自带的 CUDA 运行库。WhisperX 使用 CTranslate2（faster-whisper），GPU 运行时还需要进程能找到 **CUDA 12 cuBLAS 和 cuDNN 9**。依据为 [faster-whisper GPU 要求](https://github.com/SYSTRAN/faster-whisper#gpu)和 [CTranslate2 4.5 的 cuDNN 9 变更](https://github.com/OpenNMT/CTranslate2/releases/tag/v4.5.0)。

- Windows 缺少这些库时，可从 NVIDIA 的 [CUDA 12.8 Update 1 归档](https://developer.nvidia.com/cuda-12-8-1-download-archive)取得 CUDA 12 库，从 [NVIDIA](https://developer.nvidia.com/cudnn-downloads)选择 **用于 CUDA 12 的 cuDNN 9**。将实际 DLL 所在目录加入 PATH 后重开终端，不能把 PyTorch 包版本直接替换进示例 cuDNN 路径。
- Linux 按上述 faster-whisper 文档，在启动 Python 前通过 `LD_LIBRARY_PATH` 暴露已安装的 cuBLAS/cuDNN 库。Docker 镜像基于 `cudnn-runtime`，已包含这些运行库。

<a id="ffmpeg-runtime"></a>
## FFmpeg：无需手动安装

VideoLingo 的 WhisperX 复用安装器自动提供的 FFmpeg 命令行程序。它先用 `whisperx.audio.load_audio` 解码，再把 NumPy 音频传给识别和对齐；WhisperX 传给 pyannote 的也是内存中的 `waveform` 和 `sample_rate`。pyannote 官方支持这条路径，即使 TorchCodec 加载失败也可以使用。因此本项目的 WhisperX 流程不需要额外下载共享库版 FFmpeg，也不需要配置 PATH。

Windows 本机已验证：WhisperX 3.8.6、pyannote-audio 4.0.7、TorchCodec 0.7.0，清空系统 PATH，只启用自动提供的 FFmpeg 8.0.1，并确认 TorchCodec 导入失败。实际调用本项目 WhisperX 后端，使用 tiny.en、CPU、pyannote 语音检测和逐词对齐，成功处理 15.05 秒英文音频并输出 35 个词。本次没有验证 large/Belle 模型及 Linux/macOS 推理。

TorchCodec 仍随可选依赖安装，以满足 pyannote 的 Python 包依赖。只有自行扩展代码、让 TorchCodec 或 pyannote 直接读取文件名时，才需要兼容的 FFmpeg 共享库（TorchCodec 0.7 支持 FFmpeg 4–7）；自动提供的静态程序不包含这些库。Windows 的 `VIDEOLINGO_FFMPEG_DLL_DIR` 可选覆盖仍保留给这类自定义用法。参见 [pyannote 音频输入实现](https://github.com/pyannote/pyannote-audio/blob/4.0.4/src/pyannote/audio/core/io.py)。

## Apple Silicon：另建环境

在 Apple Silicon 上，默认依赖会安装 mlx-audio（要求 Transformers 5、`huggingface-hub>=1`），而 WhisperX 3.8.6 要求 `huggingface-hub<1`，两者不能装在同一个环境里。`uv` 只有选中预发布版 whisperx 3.8.7rc1 才能解出依赖，该组合未经验证，不建议使用。

如需在 Mac 上使用 WhisperX，请为它另建一个独立的虚拟环境，不要在默认环境里执行上面的安装命令。本仓库不提供这个独立环境的安装脚本，相关组合也没有经过验证。如果默认环境里已经装了 whisperx（例如从旧版本升级），重跑 `installer.py` 时会先卸载 WhisperX 整套依赖（whisperx、torchcodec、faster-whisper、ctranslate2、pyannote-*）（仍被其他已装包依赖的会保留），并且不会再装回去。在这套依赖卸掉之前，`installer.py --check` 会把两者共存判为错误。

<a id="common-errors"></a>
## 常见报错

1. **`local_files_only=True`**：所选 Whisper 本地模型或缓存不完整。检查模型目录及 `config.json`、`model.bin`、`tokenizer.json`，离线查找不能下载缺失权重，ping 成功也不能证明模型可用。
2. **`cublas64_12.dll not found`**：进程找不到 CUDA 12 cuBLAS。按上面的 [GPU 运行库](#cuda-runtime) 核对实际启动环境、Torch CUDA 包和库搜索路径，仅更新驱动不提供这个 DLL。
3. **Whisper 模型加载时无报错直接段错误 (Segfault)**：ctranslate2 版本与 cuDNN 版本不匹配。确保 `ctranslate2>=4.5.0`（支持 cuDNN 9，PyTorch 2.6+ 自带 cuDNN 9）。
4. **`RuntimeError: Weights only load failed`**：PyTorch ≥2.6 更改了 `torch.load` 的默认行为。已在 `whisperX_local.py` 中通过猴补丁修复，如果遇到此问题说明代码未正确更新。
5. **Streamlit 中 WhisperX 转录卡住不动（CPU/GPU 均空闲）**：`librosa.load()` 在 Streamlit 的非主线程中死锁。已用 `whisperx.audio.load_audio()`（基于 ffmpeg 子进程）替换。如果遇到此问题说明代码未正确更新。
6. **TorchCodec could not load 警告**：单独出现这条警告不影响本项目 WhisperX 流程，因为传给 pyannote 的是已解码的波形。运行 `uv run start.py --check` 验证这条音频路径。自行扩展代码让 pyannote 直接读取文件名时，才需要[上述共享库解码器](#ffmpeg-runtime)。
7. **`WhisperX is not installed`**：`whisper.backend` 已设为 `whisperx`，但没有安装该包。识别会在准备音频之前停止。请按上面的步骤自行安装，或把 `whisper.backend` 改回 `qwen`。安装器不会替你安装 WhisperX。

---

--- METRICS ---
- Files Extracted: 34
- Estimated Token Budget: ~39487 tokens
- Canonical Reference: https://codewiki.google/github.com/Huanshere/VideoLingo