File: README.md
MLX-Audio
The best audio processing library built on Apple's MLX framework, providing fast and efficient text-to-speech (TTS), speech-to-text (STT), speech-to-speech (STS), music generation, and more on Apple Silicon.
Table of Contents
- Features
- Installation
- Quick Start
- Supported Models
- Model Examples
- Web Interface & API Server
- Quantization
- Swift
- Requirements
- License
- Citation
- Acknowledgements
Features
- Fast inference optimized for Apple Silicon (M series chips)
- Multiple model architectures for TTS, STT, STS, and music generation
- Multilingual support across models
- Voice customization and cloning capabilities
- Adjustable speech speed control
- Interactive web interface with 3D audio visualization
- OpenAI-compatible REST API
- Quantization support (3-bit, 4-bit, 6-bit, 8-bit, and more) for optimized performance
- Swift package for iOS/macOS integration
Installation
Using pip
pip install mlx-audioUsing uv to install only the command line tools
Latest release from pypi:
uv tool install --force mlx-audio --prerelease=allowLatest code from github:
uv tool install --force git+https://github.com/Blaizzy/mlx-audio.git --prerelease=allowFor development or web interface:
git clone https://github.com/Blaizzy/mlx-audio.git
cd mlx-audio
pip install -e ".[dev, server]"Quick Start
Command Line
# Basic TTS generation
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Hello, world!' --voice Chelsie
# With a different voice and language hint
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Welcome to MLX-Audio!' --voice Ethan --lang_code English
# Play audio immediately
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Hello!' --voice Chelsie --play
# Save to a specific directory
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Hello!' --voice Chelsie --output_path ./my_audio
# Stream audio during generation
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Hello!' --voice Chelsie --stream
# Stream audio during generation and save it to disk
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Hello!' --voice Chelsie --stream --save
# Join multiple generated segments into one file
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text $'Hello!\nHow are you?' --voice Chelsie --join_audioBy default, when generation yields multiple segments, mlx-audio saves numbered files such as audio_000.wav and audio_001.wav. Use --join_audio to save one combined file instead. When using --stream, add --save to write the streamed audio to disk.
Python API
from mlx_audio.tts.utils import load_model
# Load model
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit")
# Generate speech
for result in model.generate(
"Hello from MLX-Audio!",
voice="Chelsie",
lang_code="English",
):
print(f"Generated {result.audio.shape[0]} samples")
# result.audio contains the waveform as mx.arraySupported Models
Text-to-Speech (TTS)
| Model | Description | Languages | Repo |
|---|---|---|---|
| Kokoro | Fast, high-quality multilingual TTS | EN, JA, ZH, FR, ES, IT, PT, HI | bf16, 8bit, 6bit, 4bit |
| KittenTTS | Compact KittenTTS 0.8 models for edge-friendly TTS | EN | nano, micro, mini, collection |
| Qwen3-TTS | Alibaba's multilingual TTS with voice design | ZH, EN, JA, KO, + more | mlx-community/Qwen3-TTS-12Hz-1.7B-VoiceDesign-bf16 |
| Higgs Audio v3 | 4B conversational TTS with voice cloning and inline control tokens | 100 languages | bosonai/higgs-audio-v3-tts-4b |
| OmniVoice | Zero-shot multilingual TTS with voice cloning, batch generation, and nonverbal tags | 646+ languages | mlx-community/OmniVoice-bf16 |
| CSM / MisoTTS | Sesame-style conversational speech models with voice cloning | EN | mlx-community/csm-1b, MisoTTS bf16, MisoTTS 8bit |
| Dia | Dialogue-focused TTS | EN | mlx-community/Dia-1.6B-fp16 |
| OuteTTS | Efficient TTS model | EN | mlx-community/OuteTTS-1.0-0.6B-fp16 |
| Spark | SparkTTS model | EN, ZH | mlx-community/Spark-TTS-0.5B-bf16 |
| Chatterbox | Expressive multilingual TTS (v2/v3) | 23 languages | v3, v2 |
| Soprano | High-quality TTS | EN | mlx-community/Soprano-1.1-80M-bf16 |
| Ming Omni TTS (BailingMM) | Multimodal generation with voice cloning, style control, and speech/music/event generation | EN, ZH | mlx-community/Ming-omni-tts-16.8B-A3B-bf16 |
| Ming Omni TTS (Dense) | Lightweight dense Ming Omni variant for voice cloning and style control | EN, ZH | mlx-community/Ming-omni-tts-0.5B-bf16 |
| KugelAudio | SOTA 7B AR+Diffusion TTS for European languages | EN, DE, FR, ES, IT, PT, NL, PL, RU, UK, + 14 more | kugelaudio/kugelaudio-0-open |
| Voxtral TTS | Mistral's 4B multilingual TTS (20 voices, 9 languages) | EN, FR, ES, DE, IT, PT, NL, AR, HI | mlx-community/Voxtral-4B-TTS-2603-mlx-bf16 |
| LongCat-AudioDiT | SOTA diffusion TTS in waveform latent space with voice cloning | ZH, EN | mlx-community/LongCat-AudioDiT-1B-bf16 |
| MeloTTS | Lightweight VITS2-based TTS with streaming | EN (more coming) | mlx-community/MeloTTS-English-MLX |
| MOSS-TTS | 8B delay-pattern and local-transformer multilingual TTS with voice cloning | 31 languages | OpenMOSS-Team/MOSS-TTS-v1.5, OpenMOSS-Team/MOSS-TTS, OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5, OpenMOSS-Team/MOSS-TTS-Local-Transformer |
| MOSS-TTS-Nano | Tiny multilingual voice-cloning TTS | 20 languages | mlx-community/MOSS-TTS-Nano-100M |
| Higgs Audio v2 | 3B Llama-backed TTS with real-time voice cloning | EN, ZH, KO, DE, ES | bf16 (upstream), q8, q6 |
Speech-to-Text (STT)
| Model | Description | Languages | Repo |
|---|---|---|---|
| Whisper | OpenAI's robust STT model | 99+ languages | mlx-community/whisper-large-v3-turbo-asr-fp16 |
| Distil-Whisper | Distilled fast Whisper variants | EN | distil-whisper/distil-large-v3 |
| Qwen3-ASR | Alibaba's multilingual ASR | ZH, EN, JA, KO, + more | mlx-community/Qwen3-ASR-1.7B-8bit |
| Mega-ASR | Routed Qwen3-ASR with automatic clean/base vs degraded/LoRA switching | EN (fixtures), multilingual Qwen3-ASR backbone | README |
| Qwen3-ForcedAligner | Word-level audio alignment | ZH, EN, JA, KO, + more | mlx-community/Qwen3-ForcedAligner-0.6B-8bit |
| MOSS-Transcribe-Diarize | Timestamped transcription with speaker labels | Multiple major languages | https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize |
| Parakeet | NVIDIA's accurate STT | EN (v2), 25 EU languages (v3) | mlx-community/parakeet-tdt-0.6b-v3 |
| Nemotron 3.5 ASR (streaming) | NVIDIA's cache-aware streaming FastConformer-RNNT with language-ID prompting | 40 language-locales | mlx-community/nemotron-3.5-asr-streaming-0.6b ยท README |
| Voxtral | Mistral's speech model | Multiple | mlx-community/Voxtral-Mini-3B-2507-bf16 |
| Voxtral Realtime | Mistral's 4B streaming STT | Multiple | 4bit, fp16 |
| VibeVoice-ASR | Microsoft's 9B ASR with diarization & timestamps | Multiple | mlx-community/VibeVoice-ASR-bf16 |
| Canary | NVIDIA's multilingual ASR with translation | 25 EU + RU, UK | README |
| Moonshine | Useful Sensors' lightweight ASR | EN | README |
| MMS | Meta's massively multilingual ASR with adapters | 1000+ | README |
| Granite Speech | IBM's ASR + speech translation | EN, FR, DE, ES, PT, JA | README |
| Qwen2-Audio | Alibaba's multimodal audio understanding (ASR, captioning, emotion, translation) | Multiple | mlx-community/Qwen2-Audio-7B-Instruct-4bit |
| MOSS-Music | OpenMOSS music understanding and lyrics ASR | EN, ZH | README |
Voice Activity Detection / Speaker Diarization (VAD)
| Model | Description | Languages | Repo |
|---|---|---|---|
| Silero VAD | Lightweight speech/non-speech detection with streaming state | Language-agnostic | mlx-community/silero-vad |
| Sortformer v1 | NVIDIA's end-to-end speaker diarization (up to 4 speakers) | Language-agnostic | mlx-community/diar_sortformer_4spk-v1-fp32 |
| Sortformer v2.1 | NVIDIA's streaming speaker diarization with AOSC compression | Language-agnostic | mlx-community/diar_streaming_sortformer_4spk-v2.1-fp32 |
See the model READMEs for API details, streaming examples, and conversion steps.
Speech-to-Speech (STS)
| Model | Description | Use Case | Repo |
|---|---|---|---|
| SAM-Audio | Text-guided source separation | Extract specific sounds | mlx-community/sam-audio-large |
| Liquid2.5-Audio* | Speech-to-Speech, Text-to-Speech and Speech-to-Text | Speech interactions | mlx-community/LFM2.5-Audio-1.5B-8bit |
| MossFormer2 SE | Speech enhancement | Noise removal | starkdmi/MossFormer2_SE_48K_MLX |
| DeepFilterNet (1/2/3) | Speech enhancement | Noise suppression | mlx-community/DeepFilterNet-mlx |
Music Generation
| Model | Description | Languages | Repo |
|---|---|---|---|
| MiniMax Music 3 | Hierarchical AR + flow-matching song generation with lyrics and 44.1 kHz stereo output | Multilingual lyrics | BF16, 8-bit, 6-bit, 4-bit, MXFP8, MXFP4, NVFP4, guide |
Model Examples
Qwen3-TTS
Alibaba's state-of-the-art multilingual TTS with voice cloning, emotion control, and voice design capabilities.
from mlx_audio.tts.utils import load_model
model = load_model("mlx-community/Qwen3-TTS-12Hz-0.6B-Base-bf16")
results = list(model.generate(
text="Hello, welcome to MLX-Audio!",
voice="Chelsie",
language="English",
))
audio = results[0].audio # mx.arraySee the Qwen3-TTS README for voice cloning, CustomVoice, VoiceDesign, and all available models.
OmniVoice
OmniVoice is a zero-shot multilingual TTS model for 646+ languages with voice cloning, batch generation, pronunciation controls, and nonverbal tags such as [laughter] and [sigh]. It uses a bidirectional Qwen3 backbone with iterative masked generation and a HiggsAudioV2 acoustic tokenizer.
from mlx_audio.tts.utils import load_model
model = load_model("mlx-community/OmniVoice-bf16")
# Basic multilingual TTS
for result in model.generate(
text="Hello from OmniVoice running on Apple Silicon.",
language="english",
duration_s=5.0,
num_steps=32,
):
audio = result.audio
# Zero-shot voice cloning
for result in model.generate(
text="This sentence uses the reference speaker.",
language="english",
ref_audio="reference.wav",
ref_text="Transcript of the reference audio.",
duration_s=5.0,
):
audio = result.audioFor stable voice cloning, provide ref_text that matches the reference clip. OmniVoice also supports generate_batch() for batched TTS and inline pronunciation controls.
Ming Omni TTS (BailingMM)
mlx_audio.tts.generate \
--model mlx-community/Ming-omni-tts-16.8B-A3B-bf16 \
--prompt "Please generate speech based on the following description.\n" \
--text "This is a quick Ming Omni test." \
--lang_code en \
--output_path audio_io \
--file_prefix ming_basic \
--verboseSee the Ming Omni TTS README for CLI and Python cookbook examples, and the Ming Omni Dense README for the mlx-community/Ming-omni-tts-0.5B-bf16 workflow.
Kokoro TTS
Kokoro is a fast, multilingual TTS model with 54 voice presets.
from mlx_audio.tts.utils import load_model
model = load_model("mlx-community/Kokoro-82M-bf16")
# Or use a quantized variant for lower memory usage:
# model = load_model("mlx-community/Kokoro-82M-8bit")
# model = load_model("mlx-community/Kokoro-82M-4bit")
# Generate with different voices
for result in model.generate(
text="Welcome to MLX-Audio!",
voice="af_heart", # American female
speed=1.0,
lang_code="a" # American English
):
audio = result.audioAvailable Voices:
- American English:
af_heart,af_bella,af_nova,af_sky,am_adam,am_echo, etc. - British English:
bf_alice,bf_emma,bm_daniel,bm_george, etc. - Japanese:
jf_alpha,jm_kumo, etc. - Chinese:
zf_xiaobei,zm_yunxi, etc.
Kokoro requires pip install misaki for text processing. Japanese and Mandarin may additionally require pip install misaki[ja] or pip install misaki[zh].
Language Codes:
| Code | Language | Note |
|---|---|---|
a |
American English | Default; requires pip install misaki |
b |
British English | Requires pip install misaki |
j |
Japanese | Requires pip install misaki[ja] |
z |
Mandarin Chinese | Requires pip install misaki[zh] |
e |
Spanish | Requires pip install misaki |
f |
French | Requires pip install misaki |
CSM (Voice Cloning)
Clone any voice using a reference audio sample:
mlx_audio.tts.generate \
--model mlx-community/csm-1b \
--text "Hello from Sesame." \
--ref_audio ./reference_voice.wav \
--playWhisper STT
from mlx_audio.stt.generate import generate_transcription
result = generate_transcription(
model="mlx-community/whisper-large-v3-turbo-asr-fp16",
audio="audio.wav",
)
print(result.text)Qwen3-ASR & ForcedAligner
Alibaba's multilingual speech models for transcription and word-level alignment.
from mlx_audio.stt import load
# Speech recognition
model = load("mlx-community/Qwen3-ASR-0.6B-8bit")
result = model.generate("audio.wav", language="English")
print(result.text)
# Word-level forced alignment
aligner = load("mlx-community/Qwen3-ForcedAligner-0.6B-8bit")
result = aligner.generate("audio.wav", text="I have a dream", language="English")
for item in result:
print(f"[{item.start_time:.2f}s - {item.end_time:.2f}s] {item.text}")See the Qwen3-ASR README for CLI usage, all models, and more examples.
VibeVoice-ASR
Microsoft's 9B parameter speech-to-text model with speaker diarization and timestamps. Supports long-form audio (up to 60 minutes) and outputs structured JSON.
from mlx_audio.stt.utils import load
model = load("mlx-community/VibeVoice-ASR-bf16")
# Basic transcription
result = model.generate(audio="meeting.wav", max_tokens=8192, temperature=0.0)
print(result.text)
# [{"Start":0,"End":5.2,"Speaker":0,"Content":"Hello everyone, let's begin."},
# {"Start":5.5,"End":9.8,"Speaker":1,"Content":"Thanks for joining today."}]
# Access parsed segments
for seg in result.segments:
print(f"[{seg['start_time']:.1f}-{seg['end_time']:.1f}] Speaker {seg['speaker_id']}: {seg['text']}")Streaming transcription:
# Stream tokens as they are generated
for text in model.stream_transcribe(audio="speech.wav", max_tokens=4096):
print(text, end="", flush=True)With context (hotwords/metadata):
result = model.generate(
audio="technical_talk.wav",
context="MLX, Apple Silicon, PyTorch, Transformer",
max_tokens=8192,
temperature=0.0,
)CLI usage:
# Basic transcription
python -m mlx_audio.stt.generate \
--model mlx-community/VibeVoice-ASR-bf16 \
--audio meeting.wav \
--output-path output \
--format json \
--max-tokens 8192 \
--verbose
# With context/hotwords
python -m mlx_audio.stt.generate \
--model mlx-community/VibeVoice-ASR-bf16 \
--audio technical_talk.wav \
--output-path output \
--format json \
--max-tokens 8192 \
--context "MLX, Apple Silicon, PyTorch, Transformer" \
--verboseParakeet (Multilingual STT)
NVIDIA's high-accuracy speech-to-text model. Parakeet v3 supports 25 European languages.
from mlx_audio.stt.utils import load
# Load the multilingual v3 model
model = load("mlx-community/parakeet-tdt-0.6b-v3")
# Transcribe audio
result = model.generate("audio.wav")
print(f"Text: {result.text}")
# Access word-level timestamps
for sentence in result.sentences:
print(f"[{sentence.start:.2f}s - {sentence.end:.2f}s] {sentence.text}")Streaming transcription:
for chunk in model.generate("long_audio.wav", stream=True):
print(chunk.text, end="", flush=True)Supported languages (v3):
Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish, Russian, Ukrainian
CLI usage:
python -m mlx_audio.stt.generate \
--model mlx-community/parakeet-tdt-0.6b-v3 \
--audio speech.wav \
--output-path output \
--format json \
--verboseKugelAudio
SOTA open-source 7B TTS model for 24 European languages, based on Microsoft VibeVoice.
Uses a hybrid AR + Diffusion architecture (Qwen2.5 LM + SDE-DPM-Solver++ diffusion head + VAE decoder).
from mlx_audio.tts.utils import load_model
model = load_model("kugelaudio/kugelaudio-0-open")
for result in model.generate(
text="Hello, welcome to MLX-Audio!",
cfg_scale=3.0, # Classifier-free guidance (1.0=fast, 3.0=quality)
ddpm_steps=10, # Diffusion steps (5=fast, 10=balanced, 20=max quality)
):
audio = result.audio # mx.array, 24kHzThe model loads directly from HuggingFace (weights are remapped automatically via sanitize()).
To quantize or save in a pre-converted format:
python -m mlx_audio.convert \
--hf-path kugelaudio/kugelaudio-0-open \
--mlx-path ./kugelaudio-0-open-bf16 \
--dtype bfloat16Supported languages (24): English, German, French, Spanish, Italian, Portuguese, Dutch, Polish, Russian, Ukrainian, Czech, Romanian, Hungarian, Swedish, Danish, Finnish, Norwegian, Greek, Bulgarian, Slovak, Croatian, Serbian, Turkish
Note: Requires ~17GB memory (7B params in bfloat16).
Pre-encoded voice presets (voice cloning) are not yet available in the upstream model โ the model generates speech with a default voice.
LongCat-AudioDiT
SOTA diffusion-based TTS operating in the waveform latent space. Uses Conditional Flow Matching with a DiT backbone and WAV-VAE codec at 24kHz. Supports zero-shot voice cloning.
from mlx_audio.tts.utils import load
model = load("mlx-community/LongCat-AudioDiT-1B-bf16")
# Zero-shot TTS
result = next(model.generate("Hello, this is a test of AudioDiT."))
audio = result.audio # mx.array, 24kHz
# Voice cloning (use "apg" guidance for best similarity)
result = next(model.generate(
text="Today is warm turning to rain.",
ref_audio="reference.wav",
ref_text="Transcript of the reference audio.",
guidance_method="apg",
cfg_strength=4.0,
steps=16,
))See the LongCat-AudioDiT README for all parameters and CLI usage.
Voxtral TTS
Mistral's 4B multilingual text-to-speech with 20 voice presets across 9 languages.
from mlx_audio.tts.utils import load
model = load("mlx-community/Voxtral-4B-TTS-2603-mlx-bf16")
for result in model.generate(text="Hello, how are you today?", voice="casual_male"):
print(result.audio_duration)Voices: casual_male, casual_female, cheerful_female, neutral_male, neutral_female, fr_male, fr_female, es_male, es_female, de_male, de_female, it_male, it_female, pt_male, pt_female, nl_male, nl_female, ar_male, hi_male, hi_female
Voxtral Realtime
Mistral's 4B parameter streaming speech-to-text model, optimized for low-latency transcription.
Available variants: 4bit (smaller/faster) | fp16 (full precision)
from mlx_audio.stt.utils import load
# Use 4bit for faster inference, fp16 for full precision
model = load("mlx-community/Voxtral-Mini-4B-Realtime-2602-4bit")
# Transcribe audio
result = model.generate("audio.wav")
print(result.text)
# Streaming transcription
for chunk in model.generate("audio.wav", stream=True):
print(chunk, end="", flush=True)
# Adjust transcription delay (lower = faster but less accurate)
result = model.generate("audio.wav", transcription_delay_ms=240)MedASR (Medical Transcription)
Specialized model for medical terms and dictation.
from mlx_audio.stt.utils import load, transcribe
model = load("mlx-community/medasr")
result = transcribe("medical_dictation.wav", model=model)
print(result["text"])Live Transcription Example:
# Continuous live transcription with VAD
python examples/medasr_live.pySAM-Audio (Source Separation)
Separate specific sounds from audio using text prompts:
from mlx_audio.sts import SAMAudio, SAMAudioProcessor, save_audio
model = SAMAudio.from_pretrained("mlx-community/sam-audio-large")
processor = SAMAudioProcessor.from_pretrained("mlx-community/sam-audio-large")
batch = processor(
descriptions=["A person speaking"],
audios=["mixed_audio.wav"],
)
result = model.separate_long(
batch.audios,
descriptions=batch.descriptions,
anchors=batch.anchor_ids,
chunk_seconds=10.0,
overlap_seconds=3.0,
ode_opt={"method": "midpoint", "step_size": 2/32},
)
save_audio(result.target[0], "voice.wav")
save_audio(result.residual[0], "background.wav")MossFormer2 (Speech Enhancement)
Remove noise from speech recordings:
from mlx_audio.sts import MossFormer2SEModel, save_audio
model = MossFormer2SEModel.from_pretrained("starkdmi/MossFormer2_SE_48K_MLX")
enhanced = model.enhance("noisy_speech.wav")
save_audio(enhanced, "clean.wav", 48000)Web Interface & API Server
MLX-Audio includes a modern web interface and OpenAI-compatible API.
Starting the Server
# Start API server
mlx_audio.server --host 0.0.0.0 --port 8000
# Start web UI (in another terminal)
cd mlx_audio/ui
npm install && npm run devAPI Endpoints
Text-to-Speech (OpenAI-compatible):
curl -X POST http://localhost:8000/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{"model": "mlx-community/Kokoro-82M-bf16", "input": "Hello!", "voice": "af_heart"}' \
--output speech.wavSpeech-to-Text:
curl -X POST http://localhost:8000/v1/audio/transcriptions \
-F "[email protected]" \
-F "model=mlx-community/whisper-large-v3-turbo-asr-fp16"Quantization
Reduce model size and improve performance with quantization using the convert script:
# Convert and quantize to 4-bit
python -m mlx_audio.convert \
--hf-path prince-canuma/Kokoro-82M \
--mlx-path ./Kokoro-82M-4bit \
--quantize \
--q-bits 4 \
--upload-repo username/Kokoro-82M-4bit (optional: if you want to upload the model to Hugging Face)
# Convert with MXFP4 quantization
python -m mlx_audio.convert \
--hf-path prince-canuma/Kokoro-82M \
--mlx-path ./Kokoro-82M-mxfp4 \
--quantize \
--q-mode mxfp4
# Convert with specific dtype (bfloat16)
python -m mlx_audio.convert \
--hf-path prince-canuma/Kokoro-82M \
--mlx-path ./Kokoro-82M-bf16 \
--dtype bfloat16 \
--upload-repo username/Kokoro-82M-bf16 (optional: if you want to upload the model to Hugging Face)Options:
| Flag | Description |
|---|---|
--hf-path |
Source Hugging Face model or local path |
--mlx-path |
Output directory for converted model |
-q, --quantize |
Enable quantization |
--q-bits |
Bits per weight (optional, defaults depend on --q-mode) |
--q-group-size |
Group size for quantization (optional, defaults depend on --q-mode) |
--q-mode |
Quantization mode: affine, mxfp4, mxfp8, nvfp4 |
--dtype |
Weight dtype: float16, bfloat16, float32 |
--upload-repo |
Upload converted model to HF Hub |
Swift
Looking for Swift/iOS support? Check out mlx-audio-swift for on-device TTS using MLX on macOS and iOS.
Requirements
- Python 3.10+
- Apple Silicon Mac (M1/M2/M3/M4)
- MLX framework
- ffmpeg (required for MP3/FLAC/OGG/Opus/Vorbis audio encoding)
Installing ffmpeg
ffmpeg is required for saving audio in MP3, FLAC, OGG, Opus, or Vorbis format. Install it using:
# macOS (using Homebrew)
brew install ffmpeg
# Ubuntu/Debian
sudo apt install ffmpegWAV format works without ffmpeg.
License
Citation
@misc{mlx-audio,
author = {Canuma, Prince},
title = {MLX Audio},
year = {2025},
howpublished = {\url{https://github.com/Blaizzy/mlx-audio}},
note = {Audio processing library for Apple Silicon with TTS, STT, and STS capabilities.}
}Acknowledgements
- Apple MLX Team for the MLX framework
File: docs/guides/index.md
Guides
Practical how-to guides for common mlx-audio workflows.
:material-play-speed:{ .lg .middle } Streaming Audio
Stream TTS output and STT transcription for low-latency applications.
:material-account-voice:{ .lg .middle } Voice Cloning
Clone voices using reference audio with CSM, Qwen3-TTS, and other models.
:material-compress:{ .lg .middle } Quantization
Reduce model size and speed up inference with 3-bit to 8-bit quantization.
:material-web:{ .lg .middle } Web UI & API Server
Run the OpenAI-compatible REST API and interactive web interface.
File: docs/guides/quantization.md
Quantization
Quantization reduces model size and can improve inference speed by storing weights at lower precision. MLX Audio supports several quantization formats through the mlx_audio.convert module.
Why Quantize?
| Benefit | Details |
|---|---|
| Smaller models | A 4-bit model is roughly 4x smaller than float16 |
| Faster inference | Lower-precision weights move through memory faster on Apple Silicon |
| Lower memory usage | Fit larger models into unified memory |
The trade-off is a potential reduction in audio quality, especially at very low bit widths.
Available Bit Widths
| Bits | Typical Use Case | Quality | Size Reduction |
|---|---|---|---|
| 3-bit | Maximum compression, acceptable for quick prototyping | Lower | ~5x vs fp16 |
| 4-bit | Good balance of quality and size | Good | ~4x vs fp16 |
| 6-bit | Near-lossless for most models | Very good | ~2.5x vs fp16 |
| 8-bit | Minimal quality loss | Excellent | ~2x vs fp16 |
Quantization Modes
MLX Audio supports several quantization modes:
| Mode | Description |
|---|---|
affine |
Standard affine quantization (default) |
mxfp4 |
Microscaling FP4 format |
mxfp8 |
Microscaling FP8 format |
nvfp4 |
NVIDIA FP4 format |
Converting and Quantizing Models
Use the mlx_audio.convert CLI to convert a HuggingFace model to MLX format with quantization:
Basic 4-bit Quantization
python -m mlx_audio.convert \
--hf-path prince-canuma/Kokoro-82M \
--mlx-path ./Kokoro-82M-4bit \
--quantize \
--q-bits 4MXFP4 Quantization
python -m mlx_audio.convert \
--hf-path prince-canuma/Kokoro-82M \
--mlx-path ./Kokoro-82M-mxfp4 \
--quantize \
--q-mode mxfp4Convert to bfloat16 (No Quantization)
python -m mlx_audio.convert \
--hf-path prince-canuma/Kokoro-82M \
--mlx-path ./Kokoro-82M-bf16 \
--dtype bfloat16Upload to Hugging Face Hub
Add --upload-repo to push the converted model directly:
python -m mlx_audio.convert \
--hf-path prince-canuma/Kokoro-82M \
--mlx-path ./Kokoro-82M-4bit \
--quantize \
--q-bits 4 \
--upload-repo username/Kokoro-82M-4bitConversion Options Reference
| Flag | Description |
|---|---|
--hf-path |
Source HuggingFace model or local path |
--mlx-path |
Output directory for the converted model |
-q, --quantize |
Enable quantization |
--q-bits |
Bits per weight (e.g., 3, 4, 6, 8) |
--q-group-size |
Group size for quantization (defaults depend on mode) |
--q-mode |
Quantization mode: affine, mxfp4, mxfp8, nvfp4 |
--dtype |
Weight dtype when not quantizing: float16, bfloat16, float32 |
--upload-repo |
Upload converted model to HuggingFace Hub |
Using Pre-Quantized Models
The mlx-community organization on Hugging Face hosts many pre-quantized models ready to use. For example:
from mlx_audio.tts.utils import load_model
# Load a pre-quantized model -- no conversion needed
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-CustomVoice-6bit")
results = list(model.generate(
text="This is running from a 6-bit quantized model!",
voice="serena",
))from mlx_audio.stt.utils import load
# 4-bit Voxtral for faster STT
model = load("mlx-community/Voxtral-Mini-4B-Realtime-2602-4bit")
result = model.generate("audio.wav")
print(result.text)Quality vs. Performance Trade-offs
!!! info "General guidance"
Start with 4-bit or 6-bit quantization. Move to 8-bit only if you hear artifacts, or to 3-bit if you need the smallest possible model.
- TTS models tend to be sensitive to quantization at 3-bit. 4-bit is usually the sweet spot.
- STT models (e.g., Whisper, Voxtral Realtime) often tolerate 4-bit quantization with minimal accuracy loss.
- Larger models (1B+ parameters) generally tolerate lower bit widths better than smaller ones.
- Always listen to output when evaluating TTS quantization -- word error rate alone does not capture prosody degradation.
Mixed Quantization Recipes
MLX Audio also supports mixed-precision quantization recipes that apply different bit widths to different layers:
| Recipe | Description |
|---|---|
mixed_2_6 |
2-bit for some layers, 6-bit for others |
mixed_3_4 |
3-bit / 4-bit mix |
mixed_3_6 |
3-bit / 6-bit mix |
mixed_4_6 |
4-bit / 6-bit mix |
These can provide better quality-to-size ratios by keeping critical layers at higher precision.
File: docs/guides/streaming.md
Streaming Audio
MLX Audio supports streaming audio generation for low-latency playback. Instead of waiting for the entire utterance to be synthesized, you can start playing audio chunks as they are produced.
CLI Streaming
Add the --stream flag to any TTS generation command. When streaming is enabled, audio is played back in real time as chunks become available:
mlx_audio.tts.generate \
--model mlx-community/Kokoro-82M-bf16 \
--text "Hello, this is a streaming example!" \
--lang_code a \
--stream!!! tip "Streaming implies playback"
The --stream flag automatically enables --play, so there is no need to pass both.
Controlling Chunk Size
The --streaming_interval argument controls how frequently audio chunks are emitted (in seconds). Smaller values reduce latency but add per-chunk overhead:
mlx_audio.tts.generate \
--model mlx-community/Kokoro-82M-bf16 \
--text "Adjusting the streaming interval changes latency." \
--lang_code a \
--stream \
--streaming_interval 1.5The default interval is 2.0 seconds.
Python Streaming
TTS Streaming
Every model's generate() method accepts stream=True. When enabled, it yields GenerationResult objects as chunks rather than waiting for full synthesis:
from mlx_audio.tts.utils import load_model
model = load_model("mlx-community/Kokoro-82M-bf16")
for result in model.generate(
text="This audio will stream chunk by chunk.",
voice="af_heart",
lang_code="a",
stream=True,
streaming_interval=2.0,
):
# result.audio is an mx.array with one chunk of audio
print(f"Chunk: {result.audio.shape[0]} samples")
# Feed result.audio to an audio player or bufferEach GenerationResult chunk contains:
| Attribute | Description |
|---|---|
audio |
mx.array waveform for this chunk |
sample_rate |
Sample rate in Hz |
is_streaming_chunk |
True for intermediate chunks |
is_final_chunk |
True for the last chunk |
Qwen3-TTS Streaming
Qwen3-TTS models support streaming across all generation methods -- generate(), generate_custom_voice(), and generate_voice_design():
from mlx_audio.tts.utils import load_model
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-CustomVoice-6bit")
audio_chunks = []
for result in model.generate(
text="Hello, how are you today?",
voice="serena",
stream=True,
streaming_interval=0.32, # ~4 tokens at 12.5 Hz
):
audio_chunks.append(result.audio)
# Play or process each chunk for low-latency output!!! info "Streaming interval for Qwen3-TTS"
At 12.5 Hz token rate, a streaming_interval of 0.32 seconds corresponds to roughly 4 tokens per chunk. Lower values reduce latency but increase overhead.
STT Streaming
Several speech-to-text models support streaming transcription. Pass stream=True to generate():
=== "Voxtral Realtime"
```python
from mlx_audio.stt.utils import load
model = load("mlx-community/Voxtral-Mini-4B-Realtime-2602-4bit")
for chunk in model.generate("audio.wav", stream=True):
print(chunk, end="", flush=True)
```=== "Parakeet"
```python
from mlx_audio.stt.utils import load
model = load("mlx-community/parakeet-tdt-0.6b-v3")
for chunk in model.generate("long_audio.wav", stream=True):
print(chunk.text, end="", flush=True)
```=== "VibeVoice-ASR"
```python
from mlx_audio.stt.utils import load
model = load("mlx-community/VibeVoice-ASR-bf16")
for text in model.stream_transcribe(audio="speech.wav", max_tokens=4096):
print(text, end="", flush=True)
```STT Streaming via CLI
python -m mlx_audio.stt.generate \
--model mlx-community/whisper-large-v3-turbo-asr-fp16 \
--audio speech.wav \
--output-path output \
--format json \
--streamAPI Server Streaming
The API server supports streaming for both TTS and STT. Set "stream": true in your request:
Streaming TTS
curl -X POST http://localhost:8000/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-community/Kokoro-82M-bf16",
"input": "Streaming over HTTP!",
"voice": "af_heart",
"stream": true,
"streaming_interval": 2.0,
"response_format": "wav"
}' \
--output streamed_speech.wavStreaming STT
curl -X POST http://localhost:8000/v1/audio/transcriptions \
-F "[email protected]" \
-F "model=mlx-community/whisper-large-v3-turbo-asr-fp16" \
-F "stream=true"Streaming STT returns newline-delimited JSON (application/x-ndjson), with each line containing a text field and optional timing information.
Real-Time WebSocket Transcription
The API server also exposes a WebSocket endpoint for real-time audio transcription:
ws://localhost:8000/v1/audio/transcriptions/realtimeThis is useful for live microphone input or continuous audio streams.
Tips
- Latency vs. quality -- Shorter
streaming_intervalvalues give lower latency but may produce more chunk boundaries. Start with the default and decrease as needed. - Memory -- Streaming does not significantly change peak memory usage since model weights remain loaded; it only changes when decoded audio is returned.
- Joining chunks -- If you need a single contiguous file, concatenate the
result.audioarrays after the loop and write once withmlx_audio.audio_io.write().
File: docs/guides/voice-cloning.md
Voice Cloning
Several MLX Audio models can clone a speaker's voice from a short reference audio sample. This guide covers the supported models and how to use them.
Overview
| Model | Method | Reference Audio Required | Notes |
|---|---|---|---|
| CSM / MisoTTS | --ref_audio CLI / ref_audio kwarg |
Yes (WAV) | Sesame-style conversational speech models |
| Qwen3-TTS Base | ref_audio + ref_text kwargs |
Yes (WAV) + transcript | Alibaba multilingual TTS |
| OmniVoice | ref_audio + ref_text kwargs |
Yes (WAV) + transcript recommended | 646+ language zero-shot cloning, best with prompt preprocessing |
| Spark | ref_audio kwarg |
Yes | SparkTTS voice cloning |
| Chatterbox | ref_audio kwarg |
Yes | Expressive multilingual TTS |
| OuteTTS | ref_audio kwarg |
Yes | Efficient TTS with cloning |
| Ming Omni TTS | ref_audio kwarg |
Yes | Multimodal with voice cloning |
CSM (Conversational Speech Model)
CSM is the simplest path to voice cloning. Provide a WAV file of the target voice and CSM will match it:
CLI
mlx_audio.tts.generate \
--model mlx-community/csm-1b \
--text "Hello from Sesame." \
--ref_audio ./reference_voice.wav \
--playPython
from mlx_audio.tts.utils import load_model
from mlx_audio.utils import load_audio
model = load_model("mlx-community/csm-1b")
for result in model.generate(
text="Hello from Sesame.",
ref_audio="./reference_voice.wav",
):
audio = result.audio!!! tip "Automatic transcription"
If you do not provide --ref_text, MLX Audio will automatically transcribe the reference audio using a Whisper model. You can specify which STT model to use with --stt_model.
Qwen3-TTS
Qwen3-TTS Base models support voice cloning by providing both a reference audio file and its transcript:
from mlx_audio.tts.utils import load_model
model = load_model("mlx-community/Qwen3-TTS-12Hz-0.6B-Base-bf16")
results = list(model.generate(
text="Hello, welcome to MLX-Audio!",
ref_audio="sample_audio.wav",
ref_text="This is what my voice sounds like.",
))
audio = results[0].audio # mx.arrayCustomVoice (Emotion Control)
The CustomVoice variant lets you combine a predefined voice with emotion and style instructions:
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-CustomVoice-bf16")
results = list(model.generate_custom_voice(
text="I'm so excited to meet you!",
speaker="Vivian",
language="English",
instruct="Very happy and excited.",
))VoiceDesign (Create Any Voice)
The VoiceDesign variant creates a voice from a text description -- no reference audio needed:
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-VoiceDesign-bf16")
results = list(model.generate_voice_design(
text="Big brother, you're back!",
language="English",
instruct="A cheerful young female voice with high pitch and energetic tone.",
))Available Qwen3-TTS Models
| Model | Method | Description |
|---|---|---|
Qwen3-TTS-12Hz-0.6B-Base-bf16 |
generate() |
Fast, predefined voices + cloning |
Qwen3-TTS-12Hz-1.7B-Base-bf16 |
generate() |
Higher quality |
Qwen3-TTS-12Hz-0.6B-CustomVoice-bf16 |
generate_custom_voice() |
Voices + emotion |
Qwen3-TTS-12Hz-1.7B-CustomVoice-bf16 |
generate_custom_voice() |
Better emotion control |
Qwen3-TTS-12Hz-1.7B-VoiceDesign-bf16 |
generate_voice_design() |
Create any voice from a description |
Available Speakers (Base / CustomVoice)
- Chinese:
Vivian,Serena,Uncle_Fu,Dylan(Beijing Dialect),Eric(Sichuan Dialect) - English:
Ryan,Aiden
Ming Omni TTS
Ming Omni TTS supports voice cloning and style control:
mlx_audio.tts.generate \
--model mlx-community/Ming-omni-tts-16.8B-A3B-bf16 \
--text "This is a Ming Omni voice cloning test." \
--ref_audio ./reference_voice.wav \
--lang_code en \
--verboseSee the Ming Omni TTS model page for detailed cookbook examples.
Best Practices
Preparing Reference Audio
!!! warning "Quality matters"
The quality of your cloned voice depends heavily on the reference audio.
- Duration -- 5 to 15 seconds of clean speech works best. Very short clips lack enough speaker information; very long clips may confuse the model.
- Format -- Use WAV at 16 kHz or higher. The library will resample automatically, but starting with a good sample rate avoids artifacts.
- Noise -- Record in a quiet environment. Background noise will be cloned along with the voice. Consider using MossFormer2 or DeepFilterNet to enhance noisy recordings first.
- Content -- Natural, conversational speech produces the best results. Avoid whispering or shouting unless that is the target style.
Providing Reference Text
Some models (Qwen3-TTS) require a transcript of the reference audio (ref_text). If you omit it, MLX Audio will transcribe the clip automatically using Whisper:
mlx_audio.tts.generate \
--model mlx-community/Qwen3-TTS-12Hz-0.6B-Base-bf16 \
--text "Cloned speech output." \
--ref_audio reference.wav \
--stt_model mlx-community/whisper-large-v3-turbo-asr-fp16Providing the transcript yourself avoids loading the STT model and speeds up generation.
OmniVoice
OmniVoice supports multilingual zero-shot voice cloning with a HiggsAudioV2 acoustic tokenizer and iterative masked generation.
!!! warning "ref_text must match preprocessed audio"
OmniVoice preprocessing removes silence and trims the reference clip. If you transcribe the original file, the ASR transcript may be longer than the preprocessed audio, causing the extra text to leak into generation. Always transcribe the preprocessed audio, not the raw recording. See examples/omnivoice_clone_demo.py for the correct workflow.
from mlx_audio.tts.utils import load_model as load_tts
from mlx_audio.tts.models.omnivoice.utils import create_voice_clone_prompt
from mlx_audio.stt.utils import load_model as load_stt
from mlx_audio.audio_io import write as audio_write
import mlx.core as mx, numpy as np, tempfile, os
tts = load_tts("mlx-community/OmniVoice-bf16")
# Preprocess โ encode โ decode โ transcribe (matches original pipeline)
ref_tokens = create_voice_clone_prompt("reference.wav", tokenizer=tts.audio_tokenizer)
mx.eval(ref_tokens)
preprocessed = np.array(tts.audio_tokenizer.decode(ref_tokens).astype(mx.float32))
tmp = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
audio_write(tmp.name, preprocessed, 24000); tmp.close()
ref_text = load_stt("mlx-community/Qwen3-ASR-0.6B-8bit").generate(tmp.name).text
os.unlink(tmp.name)
results = list(tts.generate(
text="Hello from OmniVoice.",
language="english",
ref_tokens=ref_tokens,
ref_text=ref_text,
))
audio_write("output.wav", np.array(results[0].audio), results[0].sample_rate)OmniVoice-specific notes
- Reference text is required for stable cloning. Without it, output quality degrades significantly โ garbled speech, wrong language, or missing words.
- Transcribe after preprocessing, not before. The original k2-fsa/OmniVoice runs Whisper on audio after silence removal. This demo replicates that approach.
- Prompt preprocessing matters. MLX Audio mirrors the original Python pipeline with RMS normalization, silence removal, trimming at silence gaps, and torchaudio-compatible resampling before reference encoding.
- Best reference length: roughly 5โ15 seconds of actual speech after silence trimming.
- Supported inline controls: nonverbal tags such as
[laughter],[sigh], and pronunciation overrides for English CMU dictionary forms and Chinese pinyin forms.
Example: English CMU pronunciation control
results = list(model.generate(
text="He plays the [B EY1 S] guitar while catching a [B AE1 S] fish.",
language="english",
))Example: Nonverbal tags
results = list(model.generate(
text="I just heard the funniest joke [laughter] that was incredible.",
language="english",
))Combining Cloning with Streaming
Voice cloning and streaming work together. Add --stream to any cloning command:
mlx_audio.tts.generate \
--model mlx-community/csm-1b \
--text "Streaming with a cloned voice." \
--ref_audio reference.wav \
--streamFile: docs/guides/web-ui-api-server.md
Web UI & API Server
MLX Audio ships with a FastAPI-based API server and a Next.js web interface (Studio UI) for interactive audio generation and transcription.
Starting the Server
API Server Only
mlx_audio.server --host 0.0.0.0 --port 8000API Server with Studio UI
Pass --start-ui to launch the Next.js web interface alongside the API server:
mlx_audio.server --host 0.0.0.0 --port 8000 --start-uiThe API will be available at http://localhost:8000 and the Studio UI at http://localhost:3000.
Server Options
| Flag | Default | Description |
|---|---|---|
--host |
localhost |
Host to bind the server to |
--port |
8000 |
Port for the API server |
--reload |
false |
Auto-reload on code changes (development) |
--start-ui |
false |
Start the Studio UI alongside the API |
--allowed-origins |
* |
CORS allowed origins (space-separated) |
--log-dir |
logs |
Directory for server logs |
--realtime-model |
null |
Default model for /v1/realtime when the client omits ?model= |
--realtime-transcription-delay-ms |
null |
Transcription latency/quality knob for models that support it (e.g. voxtral_realtime) |
--vad-model |
mlx-community/silero-vad |
Streaming VAD model used for server-side turn detection (server_vad) on /v1/realtime |
--tts-max-batch-size |
8 |
Maximum compatible TTS speech requests per continuous batch session |
The two realtime flags also read from MLX_AUDIO_REALTIME_MODEL and MLX_AUDIO_REALTIME_TRANSCRIPTION_DELAY_MS if present; the CLI flags take precedence. --vad-model likewise reads from MLX_AUDIO_VAD_MODEL.
The TTS batching flag also reads from MLX_AUDIO_TTS_MAX_BATCH_SIZE; the CLI flag takes precedence.
CORS Configuration
By default, all origins are allowed. To restrict origins:
mlx_audio.server --allowed-origins http://localhost:3000 https://myapp.example.comOr set the MLX_AUDIO_ALLOWED_ORIGINS environment variable with a comma-separated list.
TTS Continuous Batching
Compatible non-streaming TTS /v1/audio/speech requests are routed through the server's continuous batching path when the model exposes a create_tts_batch_session(...) hook.
Requests that are not supported by the model's continuous batching hook continue through the existing serial or fixed-window batch paths.
mlx_audio.server --host 127.0.0.1 --port 8000 \
--tts-max-batch-size 8OpenAI-Compatible API
The server implements OpenAI-compatible endpoints, so existing code that targets the OpenAI audio API can point to MLX Audio with minimal changes.
Text-to-Speech
POST /v1/audio/speech
curl -X POST http://localhost:8000/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-community/Kokoro-82M-bf16",
"input": "Hello, world!",
"voice": "af_heart",
"response_format": "mp3"
}' \
--output speech.mp3Request body:
| Field | Type | Default | Description |
|---|---|---|---|
model |
string | required | Model ID (HuggingFace repo or local path) |
input |
string | required | Text to synthesize |
voice |
string | null |
Voice preset name |
speed |
float | 1.0 |
Playback speed multiplier |
lang_code |
string | "a" |
Language code |
response_format |
string | "mp3" |
Output format: mp3, wav, flac, ogg, opus |
stream |
bool | false |
Stream audio chunks |
streaming_interval |
float | 2.0 |
Seconds between stream chunks |
temperature |
float | 0.7 |
Sampling temperature |
max_tokens |
int | 1200 |
Maximum generation tokens |
ref_audio |
string | null |
Path to reference audio for voice cloning |
ref_text |
string | null |
Transcript of reference audio |
instruct |
string | null |
Style/emotion instruction |
Streaming TTS
curl -X POST http://localhost:8000/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"model": "mlx-community/Kokoro-82M-bf16",
"input": "Streaming audio over HTTP.",
"voice": "af_heart",
"stream": true,
"response_format": "wav"
}' \
--output streamed.wavSpeech-to-Text
POST /v1/audio/transcriptions
curl -X POST http://localhost:8000/v1/audio/transcriptions \
-F "[email protected]" \
-F "model=mlx-community/whisper-large-v3-turbo-asr-fp16"Form fields:
| Field | Type | Default | Description |
|---|---|---|---|
file |
file | required | Audio file to transcribe |
model |
string | required | STT model ID |
language |
string | null |
Language code |
max_tokens |
int | 1024 |
Maximum output tokens |
stream |
bool | false |
Stream results as NDJSON |
context |
string | null |
Hotwords or metadata to guide transcription |
verbose |
bool | false |
Include extra details |
Audio Source Separation
POST /v1/audio/separations
curl -X POST http://localhost:8000/v1/audio/separations \
-F "[email protected]" \
-F "model=mlx-community/sam-audio-large-fp16" \
-F "description=speech"Returns JSON with base64-encoded target and residual WAV buffers.
Model Management
# List loaded models
curl http://localhost:8000/v1/models
# Load a model
curl -X POST "http://localhost:8000/v1/models?model_name=mlx-community/Kokoro-82M-bf16"
# Unload a model
curl -X DELETE "http://localhost:8000/v1/models?model_name=mlx-community/Kokoro-82M-bf16"Real-Time WebSocket Transcription
The server exposes two WebSocket endpoints for live transcription. Both accept 16-bit signed little-endian PCM audio; they differ in wire protocol and intended consumers.
`/v1/audio/transcriptions/realtime` (VAD-based streaming)
ws://localhost:8000/v1/audio/transcriptions/realtimeSend raw PCM frames as binary WebSocket messages; the server performs VAD, chunks on silence, and emits transcription JSON messages back. Uses the preloaded Whisper-style STT model โ connect after calling POST /v1/models to select it.
`/v1/realtime` (OpenAI Realtime-compatible)
ws://localhost:8000/v1/realtime?model=<model-id>Implements a subset of the OpenAI Realtime API wire protocol, so existing Realtime clients can target MLX Audio by swapping the base URL. Any STT model that exposes create_streaming_session works โ including voxtral_realtime, which is streaming-first and recommended.
Model selection order (first match wins):
?model=query parameter on connect.session.update.model(orsession.audio.input.transcription.model) event after connect.--realtime-modelCLI flag /MLX_AUDIO_REALTIME_MODELenv var.
If none is set, the server replies with an error event and closes the socket.
Client โ server events (subset of OpenAI Realtime):
| Type | Purpose |
|---|---|
session.update |
Change the model, input sample rate, transcription config, or turn detection mode |
input_audio_buffer.append |
Append a chunk of base64-encoded PCM16 to the current item |
input_audio_buffer.commit |
Signal end-of-utterance; the server drains deltas and emits completed. Manual-commit mode only โ unused under server_vad |
Declaring the input sample rate. The server assumes incoming PCM is 24 kHz by default (the OpenAI Realtime client convention). If you are sending audio at a different rate โ e.g. a 16 kHz microphone capture or a 48 kHz file โ tell the server via session.update so it resamples correctly. Without this, 16 kHz audio interpreted as 24 kHz would sound sped-up to the model and transcribe as garbage.
{
"type": "session.update",
"session": {
"audio": {"input": {"format": {"type": "audio/pcm", "rate": 16000}}}
}
}This only declares your input rate. The server resamples from that to whatever rate the model expects internally โ you never need to match the model's native rate.
Turn detection (server_vad). By default (turn_detection: null) the client owns turn boundaries and signals end-of-utterance with input_audio_buffer.commit. Set turn_detection to {"type": "server_vad"} via session.update and the server instead runs a streaming VAD (Silero by default โ see --vad-model) over the incoming audio: it detects when the speaker starts and stops, emits input_audio_buffer.speech_started / input_audio_buffer.speech_stopped, and auto-commits each turn. In this mode the client never sends input_audio_buffer.commit.
{
"type": "session.update",
"session": {
"audio": {
"input": {
"turn_detection": {
"type": "server_vad",
"threshold": 0.5,
"prefix_padding_ms": 300,
"silence_duration_ms": 500
}
}
}
}
}| Field | Default | Purpose |
|---|---|---|
threshold |
0.5 |
Speech-probability threshold above which a VAD frame counts as speech |
prefix_padding_ms |
300 |
Shifts the reported audio_start_ms earlier; does not gate transcription |
silence_duration_ms |
500 |
Sub-threshold audio after speech before the turn is considered finished |
To use a different VAD model, launch the server with --vad-model (or set MLX_AUDIO_VAD_MODEL). semantic_vad is not implemented yet โ requesting it returns an error event.
Server โ client events:
| Type | Purpose |
|---|---|
session.created / session.updated |
Session snapshot (id, model, input format, turn detection) |
conversation.item.added |
New user item opened for the next audio chunk |
input_audio_buffer.speech_started |
(server_vad) Speaker started; carries audio_start_ms |
input_audio_buffer.speech_stopped |
(server_vad) Speaker stopped; the server auto-commits the turn |
input_audio_buffer.committed |
Acknowledges a commit โ client commit or server_vad auto-commit |
conversation.item.input_audio_transcription.delta |
Incremental transcript token(s) |
conversation.item.input_audio_transcription.completed |
Final transcript for the item |
error |
Error message; the socket may close afterwards |
Minimal Python client:
import asyncio, base64, json, websockets, numpy as np, soundfile as sf
async def transcribe(path: str):
audio, sr = sf.read(path, dtype="int16", always_2d=False)
uri = "ws://localhost:8000/v1/realtime?model=iris-sfg/Voxtral-Mini-4B-Realtime-2602-4bit"
async with websockets.connect(uri) as ws:
# Wait for session.created
await ws.recv()
# Declare the input rate (the server will resample to the model's rate)
await ws.send(json.dumps({
"type": "session.update",
"session": {"audio": {"input": {"format": {"type": "audio/pcm", "rate": sr}}}},
}))
# Stream audio in 100 ms chunks
step = sr // 10
for i in range(0, len(audio), step):
chunk = audio[i : i + step].tobytes()
await ws.send(json.dumps({
"type": "input_audio_buffer.append",
"audio": base64.b64encode(chunk).decode(),
}))
await ws.send(json.dumps({"type": "input_audio_buffer.commit"}))
async for raw in ws:
evt = json.loads(raw)
if evt["type"].endswith(".delta"):
print(evt["delta"], end="", flush=True)
elif evt["type"].endswith(".completed"):
print()
break
asyncio.run(transcribe("audio.wav"))Tuning transcription delay (for models like voxtral_realtime that expose the knob):
mlx_audio.server \
--realtime-model iris-sfg/Voxtral-Mini-4B-Realtime-2602-4bit \
--realtime-transcription-delay-ms 960Lower values reduce latency at the cost of accuracy. Models that don't declare a transcription_delay_ms parameter silently ignore the flag.
!!! note "Single-process MLX serialization"
MLX inference on a given device is serialized inside the server. /v1/realtime schedules session.step() calls cooperatively โ two concurrent websocket clients share GPU time rather than running in parallel. This is fine as long as each stream transcribes faster than real-time; otherwise run multiple server instances behind a load balancer.
Web Interface (Studio UI)
The Studio UI is a Next.js application located in mlx_audio/ui/. It provides:
- A text input for TTS generation with voice and model selectors
- Audio upload and recording for STT
- 3D audio visualization
- Model loading and management
Running the UI Separately
If you prefer to run the UI outside the server process:
cd mlx_audio/ui
npm install
npm run devThe UI runs on http://localhost:3000 and expects the API server at http://localhost:8000.
Using with the OpenAI Python Client
Because the API is OpenAI-compatible, you can use the official OpenAI Python client:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-needed", # MLX Audio does not require an API key
)
# Text-to-Speech
response = client.audio.speech.create(
model="mlx-community/Kokoro-82M-bf16",
voice="af_heart",
input="Hello from the OpenAI client!",
)
response.stream_to_file("output.mp3")
# Speech-to-Text
with open("audio.wav", "rb") as f:
transcript = client.audio.transcriptions.create(
model="mlx-community/whisper-large-v3-turbo-asr-fp16",
file=f,
)
print(transcript.text)Installation
The server requires the server optional dependency group:
pip install "mlx-audio[server]"This installs FastAPI, Uvicorn, and python-multipart.
File: docs/getting-started/index.md
Getting Started
Welcome to MLX Audio! This section will help you install the library and run your first text-to-speech generation or speech transcription in minutes.
Requirements
- Python 3.10+
- Apple Silicon Mac (M1 / M2 / M3 / M4)
- ffmpeg (required for MP3, FLAC, OGG, Opus, or Vorbis encoding; WAV works without it)
Where to Go Next
:octicons-download-24:{ .lg .middle } Installation
Install mlx-audio with pip or uv, including optional extras for TTS, STT, and development.
:octicons-terminal-24:{ .lg .middle } Quick Start: CLI
Generate speech and transcribe audio from the command line.
:octicons-code-24:{ .lg .middle } Quick Start: Python
Use the Python API to generate, transcribe, and process audio programmatically.
File: docs/getting-started/installation.md
Installation
Quick Install
=== "pip"
```bash
pip install mlx-audio
```=== "uv (tool)"
Install the CLI tools globally with uv:
```bash
# Latest release from PyPI
uv tool install --force mlx-audio --prerelease=allow
```
Or install from the latest source:
```bash
# Latest code from GitHub
uv tool install --force git+https://github.com/Blaizzy/mlx-audio.git --prerelease=allow
```Optional Extras
mlx-audio ships optional dependency groups so you only install what you need:
| Extra | What it includes |
|---|---|
tts |
Text-to-speech dependencies |
stt |
Speech-to-text dependencies |
sts |
Speech-to-speech dependencies |
docs |
MkDocs and API reference tooling |
all |
Everything (TTS + STT + STS) |
dev |
Development and testing tools |
Install extras with bracket syntax:
# TTS only
pip install "mlx-audio[tts]"
# STT only
pip install "mlx-audio[stt]"
# Everything
pip install "mlx-audio[all]"
# Documentation tooling
pip install "mlx-audio[docs]"Development Setup
Clone the repository and install in editable mode with dev dependencies:
git clone https://github.com/Blaizzy/mlx-audio.git
cd mlx-audio
pip install -e ".[dev]"If you are working on the docs site, use:
pip install -e ".[dev,docs]"ffmpeg
ffmpeg is required for saving audio in MP3, FLAC, OGG, Opus, or Vorbis format. WAV works without it.
=== "macOS (Homebrew)"
```bash
brew install ffmpeg
```=== "Ubuntu / Debian"
```bash
sudo apt install ffmpeg
```!!! tip
If you only need WAV output, you can skip installing ffmpeg entirely.
File: docs/getting-started/quickstart-cli.md
Quick Start: CLI
mlx-audio provides command-line tools for both text-to-speech generation and speech-to-text transcription.
Text-to-Speech
!!! note
These TTS quickstart examples use mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit.
Basic Generation
Generate speech from text with a single command:
mlx_audio.tts.generate \
--model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit \
--text "Hello, world!" \
--voice Chelsie \
--lang_code EnglishPlay Audio Immediately
Add --play to hear the result without saving:
mlx_audio.tts.generate \
--model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit \
--text "Hello, world!" \
--voice Chelsie \
--lang_code English \
--playVoice and Language Selection
Choose a voice preset and provide a language hint:
mlx_audio.tts.generate \
--model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit \
--text "Welcome to MLX-Audio!" \
--voice Ethan \
--lang_code EnglishSave to a Directory
mlx_audio.tts.generate \
--model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit \
--text "Hello!" \
--voice Chelsie \
--lang_code English \
--output_path ./my_audioVoice Cloning (CSM)
Clone a voice from a reference audio file:
mlx_audio.tts.generate \
--model mlx-community/csm-1b \
--text "Hello from Sesame." \
--ref_audio ./reference_voice.wav \
--playSpeech-to-Text
Transcribe with Whisper
python -m mlx_audio.stt.generate \
--model mlx-community/whisper-large-v3-turbo-asr-fp16 \
--audio audio.wav \
--output-path output \
--format json \
--verboseTranscribe with Parakeet
python -m mlx_audio.stt.generate \
--model mlx-community/parakeet-tdt-0.6b-v3 \
--audio speech.wav \
--output-path output \
--format json \
--verboseTranscribe with VibeVoice-ASR
python -m mlx_audio.stt.generate \
--model mlx-community/VibeVoice-ASR-bf16 \
--audio meeting.wav \
--output-path output \
--format json \
--max-tokens 8192 \
--verboseAdd context/hotwords for better accuracy on domain-specific terms:
python -m mlx_audio.stt.generate \
--model mlx-community/VibeVoice-ASR-bf16 \
--audio technical_talk.wav \
--output-path output \
--format json \
--max-tokens 8192 \
--context "MLX, Apple Silicon, PyTorch, Transformer" \
--verboseCommon Flags
| Flag | Description |
|---|---|
--model |
Hugging Face model ID or local path |
--text |
Input text for TTS generation |
--audio |
Input audio file for STT transcription |
--voice |
Voice preset name (e.g., Chelsie, Ethan, casual_male) |
--speed |
Speech speed multiplier (default: 1.0) |
--lang_code |
Language hint (e.g., English, Chinese, or auto) |
--play |
Play audio immediately after generation |
--output_path |
Directory to save output files |
--verbose |
Show detailed generation info |
--format |
Output format for STT (json, etc.) |
!!! note
Different models support different flags. Check the Models section for model-specific options.
File: docs/getting-started/quickstart-python.md
Quick Start: Python
Use the mlx-audio Python API to generate speech, transcribe audio, and process audio programmatically.
Text-to-Speech
!!! note
These TTS quickstart examples use mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit.
Basic Generation
from mlx_audio.tts.utils import load_model
# Load a TTS model
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit")
# Generate speech
for result in model.generate(
"Hello from MLX-Audio!",
voice="Chelsie",
lang_code="English",
):
print(f"Generated {result.audio.shape[0]} samples")
# result.audio contains the waveform as mx.arrayVoice and Language Options
from mlx_audio.tts.utils import load_model
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit")
for result in model.generate(
text="Welcome to MLX-Audio!",
voice="Ethan",
lang_code="English",
):
audio = result.audio!!! tip "Qwen3-TTS Base Model"
Pass voice to pick a preset speaker and lang_code to hint the language, for example voice="Chelsie" and lang_code="English".
Voxtral TTS
from mlx_audio.tts.utils import load
model = load("mlx-community/Voxtral-4B-TTS-2603-mlx-bf16")
for result in model.generate(text="Hello, how are you today?", voice="casual_male"):
print(result.audio_duration)Qwen3-TTS
from mlx_audio.tts.utils import load_model
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit")
results = list(model.generate(
text="Hello, welcome to MLX-Audio!",
voice="Chelsie",
lang_code="English",
))
audio = results[0].audio # mx.arraySpeech-to-Text
Whisper
from mlx_audio.stt.generate import generate_transcription
result = generate_transcription(
model="mlx-community/whisper-large-v3-turbo-asr-fp16",
audio="audio.wav",
)
print(result.text)Qwen3-ASR
from mlx_audio.stt import load
# Speech recognition
model = load("mlx-community/Qwen3-ASR-0.6B-8bit")
result = model.generate("audio.wav", language="English")
print(result.text)Word-Level Alignment (Qwen3-ForcedAligner)
from mlx_audio.stt import load
aligner = load("mlx-community/Qwen3-ForcedAligner-0.6B-8bit")
result = aligner.generate("audio.wav", text="I have a dream", language="English")
for item in result:
print(f"[{item.start_time:.2f}s - {item.end_time:.2f}s] {item.text}")Parakeet (with Timestamps)
from mlx_audio.stt.utils import load
model = load("mlx-community/parakeet-tdt-0.6b-v3")
result = model.generate("audio.wav")
print(f"Text: {result.text}")
# Word-level timestamps
for sentence in result.sentences:
print(f"[{sentence.start:.2f}s - {sentence.end:.2f}s] {sentence.text}")Streaming Transcription
Several STT models support streaming for low-latency output:
=== "Voxtral Realtime"
```python
from mlx_audio.stt.utils import load
model = load("mlx-community/Voxtral-Mini-4B-Realtime-2602-4bit")
for chunk in model.generate("audio.wav", stream=True):
print(chunk, end="", flush=True)
```=== "Parakeet"
```python
from mlx_audio.stt.utils import load
model = load("mlx-community/parakeet-tdt-0.6b-v3")
for chunk in model.generate("long_audio.wav", stream=True):
print(chunk.text, end="", flush=True)
```=== "VibeVoice"
```python
from mlx_audio.stt.utils import load
model = load("mlx-community/VibeVoice-ASR-bf16")
for text in model.stream_transcribe(audio="speech.wav", max_tokens=4096):
print(text, end="", flush=True)
```Saving Audio
Save with STS Utilities
from mlx_audio.sts import save_audio
# Save an audio array to a WAV file
save_audio(audio_array, "output.wav", sample_rate=24000)!!! note
Saving to MP3, FLAC, OGG, Opus, or Vorbis requires ffmpeg. WAV works without it.