mlx-audio Documentation

A text-to-speech (TTS), speech-to-text (STT) and speech-to-speech (STS) library built on Apple's MLX framework, providing efficient speech analysis on Apple...

RAW Doc

File: README.md

MLX-Audio




The best audio processing library built on Apple's MLX framework, providing fast and efficient text-to-speech (TTS), speech-to-text (STT), speech-to-speech (STS), music generation, and more on Apple Silicon.

Table of Contents

Features

  • Fast inference optimized for Apple Silicon (M series chips)
  • Multiple model architectures for TTS, STT, STS, and music generation
  • Multilingual support across models
  • Voice customization and cloning capabilities
  • Adjustable speech speed control
  • Interactive web interface with 3D audio visualization
  • OpenAI-compatible REST API
  • Quantization support (3-bit, 4-bit, 6-bit, 8-bit, and more) for optimized performance
  • Swift package for iOS/macOS integration

Installation

Using pip

bash
pip install mlx-audio

Using uv to install only the command line tools

Latest release from pypi:

bash
uv tool install --force mlx-audio --prerelease=allow

Latest code from github:

bash
uv tool install --force git+https://github.com/Blaizzy/mlx-audio.git --prerelease=allow

For development or web interface:

bash
git clone https://github.com/Blaizzy/mlx-audio.git
cd mlx-audio
pip install -e ".[dev, server]"

Quick Start

Command Line

bash
# Basic TTS generation
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Hello, world!' --voice Chelsie

# With a different voice and language hint
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Welcome to MLX-Audio!' --voice Ethan --lang_code English

# Play audio immediately
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Hello!' --voice Chelsie --play

# Save to a specific directory
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Hello!' --voice Chelsie --output_path ./my_audio

# Stream audio during generation
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Hello!' --voice Chelsie --stream

# Stream audio during generation and save it to disk
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text 'Hello!' --voice Chelsie --stream --save

# Join multiple generated segments into one file
mlx_audio.tts.generate --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit --text $'Hello!\nHow are you?' --voice Chelsie --join_audio

By default, when generation yields multiple segments, mlx-audio saves numbered files such as audio_000.wav and audio_001.wav. Use --join_audio to save one combined file instead. When using --stream, add --save to write the streamed audio to disk.

Python API

python
from mlx_audio.tts.utils import load_model

# Load model
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit")

# Generate speech
for result in model.generate(
    "Hello from MLX-Audio!",
    voice="Chelsie",
    lang_code="English",
):
    print(f"Generated {result.audio.shape[0]} samples")
    # result.audio contains the waveform as mx.array

Supported Models

Text-to-Speech (TTS)

Model Description Languages Repo
Kokoro Fast, high-quality multilingual TTS EN, JA, ZH, FR, ES, IT, PT, HI bf16, 8bit, 6bit, 4bit
KittenTTS Compact KittenTTS 0.8 models for edge-friendly TTS EN nano, micro, mini, collection
Qwen3-TTS Alibaba's multilingual TTS with voice design ZH, EN, JA, KO, + more mlx-community/Qwen3-TTS-12Hz-1.7B-VoiceDesign-bf16
Higgs Audio v3 4B conversational TTS with voice cloning and inline control tokens 100 languages bosonai/higgs-audio-v3-tts-4b
OmniVoice Zero-shot multilingual TTS with voice cloning, batch generation, and nonverbal tags 646+ languages mlx-community/OmniVoice-bf16
CSM / MisoTTS Sesame-style conversational speech models with voice cloning EN mlx-community/csm-1b, MisoTTS bf16, MisoTTS 8bit
Dia Dialogue-focused TTS EN mlx-community/Dia-1.6B-fp16
OuteTTS Efficient TTS model EN mlx-community/OuteTTS-1.0-0.6B-fp16
Spark SparkTTS model EN, ZH mlx-community/Spark-TTS-0.5B-bf16
Chatterbox Expressive multilingual TTS (v2/v3) 23 languages v3, v2
Soprano High-quality TTS EN mlx-community/Soprano-1.1-80M-bf16
Ming Omni TTS (BailingMM) Multimodal generation with voice cloning, style control, and speech/music/event generation EN, ZH mlx-community/Ming-omni-tts-16.8B-A3B-bf16
Ming Omni TTS (Dense) Lightweight dense Ming Omni variant for voice cloning and style control EN, ZH mlx-community/Ming-omni-tts-0.5B-bf16
KugelAudio SOTA 7B AR+Diffusion TTS for European languages EN, DE, FR, ES, IT, PT, NL, PL, RU, UK, + 14 more kugelaudio/kugelaudio-0-open
Voxtral TTS Mistral's 4B multilingual TTS (20 voices, 9 languages) EN, FR, ES, DE, IT, PT, NL, AR, HI mlx-community/Voxtral-4B-TTS-2603-mlx-bf16
LongCat-AudioDiT SOTA diffusion TTS in waveform latent space with voice cloning ZH, EN mlx-community/LongCat-AudioDiT-1B-bf16
MeloTTS Lightweight VITS2-based TTS with streaming EN (more coming) mlx-community/MeloTTS-English-MLX
MOSS-TTS 8B delay-pattern and local-transformer multilingual TTS with voice cloning 31 languages OpenMOSS-Team/MOSS-TTS-v1.5, OpenMOSS-Team/MOSS-TTS, OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5, OpenMOSS-Team/MOSS-TTS-Local-Transformer
MOSS-TTS-Nano Tiny multilingual voice-cloning TTS 20 languages mlx-community/MOSS-TTS-Nano-100M
Higgs Audio v2 3B Llama-backed TTS with real-time voice cloning EN, ZH, KO, DE, ES bf16 (upstream), q8, q6

Speech-to-Text (STT)

Model Description Languages Repo
Whisper OpenAI's robust STT model 99+ languages mlx-community/whisper-large-v3-turbo-asr-fp16
Distil-Whisper Distilled fast Whisper variants EN distil-whisper/distil-large-v3
Qwen3-ASR Alibaba's multilingual ASR ZH, EN, JA, KO, + more mlx-community/Qwen3-ASR-1.7B-8bit
Mega-ASR Routed Qwen3-ASR with automatic clean/base vs degraded/LoRA switching EN (fixtures), multilingual Qwen3-ASR backbone README
Qwen3-ForcedAligner Word-level audio alignment ZH, EN, JA, KO, + more mlx-community/Qwen3-ForcedAligner-0.6B-8bit
MOSS-Transcribe-Diarize Timestamped transcription with speaker labels Multiple major languages https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize
Parakeet NVIDIA's accurate STT EN (v2), 25 EU languages (v3) mlx-community/parakeet-tdt-0.6b-v3
Nemotron 3.5 ASR (streaming) NVIDIA's cache-aware streaming FastConformer-RNNT with language-ID prompting 40 language-locales mlx-community/nemotron-3.5-asr-streaming-0.6b ยท README
Voxtral Mistral's speech model Multiple mlx-community/Voxtral-Mini-3B-2507-bf16
Voxtral Realtime Mistral's 4B streaming STT Multiple 4bit, fp16
VibeVoice-ASR Microsoft's 9B ASR with diarization & timestamps Multiple mlx-community/VibeVoice-ASR-bf16
Canary NVIDIA's multilingual ASR with translation 25 EU + RU, UK README
Moonshine Useful Sensors' lightweight ASR EN README
MMS Meta's massively multilingual ASR with adapters 1000+ README
Granite Speech IBM's ASR + speech translation EN, FR, DE, ES, PT, JA README
Qwen2-Audio Alibaba's multimodal audio understanding (ASR, captioning, emotion, translation) Multiple mlx-community/Qwen2-Audio-7B-Instruct-4bit
MOSS-Music OpenMOSS music understanding and lyrics ASR EN, ZH README

Voice Activity Detection / Speaker Diarization (VAD)

Model Description Languages Repo
Silero VAD Lightweight speech/non-speech detection with streaming state Language-agnostic mlx-community/silero-vad
Sortformer v1 NVIDIA's end-to-end speaker diarization (up to 4 speakers) Language-agnostic mlx-community/diar_sortformer_4spk-v1-fp32
Sortformer v2.1 NVIDIA's streaming speaker diarization with AOSC compression Language-agnostic mlx-community/diar_streaming_sortformer_4spk-v2.1-fp32

See the model READMEs for API details, streaming examples, and conversion steps.

Speech-to-Speech (STS)

Model Description Use Case Repo
SAM-Audio Text-guided source separation Extract specific sounds mlx-community/sam-audio-large
Liquid2.5-Audio* Speech-to-Speech, Text-to-Speech and Speech-to-Text Speech interactions mlx-community/LFM2.5-Audio-1.5B-8bit
MossFormer2 SE Speech enhancement Noise removal starkdmi/MossFormer2_SE_48K_MLX
DeepFilterNet (1/2/3) Speech enhancement Noise suppression mlx-community/DeepFilterNet-mlx

Music Generation

Model Description Languages Repo
MiniMax Music 3 Hierarchical AR + flow-matching song generation with lyrics and 44.1 kHz stereo output Multilingual lyrics BF16, 8-bit, 6-bit, 4-bit, MXFP8, MXFP4, NVFP4, guide

Model Examples

Qwen3-TTS

Alibaba's state-of-the-art multilingual TTS with voice cloning, emotion control, and voice design capabilities.

python
from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/Qwen3-TTS-12Hz-0.6B-Base-bf16")
results = list(model.generate(
    text="Hello, welcome to MLX-Audio!",
    voice="Chelsie",
    language="English",
))

audio = results[0].audio  # mx.array

See the Qwen3-TTS README for voice cloning, CustomVoice, VoiceDesign, and all available models.

OmniVoice

OmniVoice is a zero-shot multilingual TTS model for 646+ languages with voice cloning, batch generation, pronunciation controls, and nonverbal tags such as [laughter] and [sigh]. It uses a bidirectional Qwen3 backbone with iterative masked generation and a HiggsAudioV2 acoustic tokenizer.

python
from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/OmniVoice-bf16")

# Basic multilingual TTS
for result in model.generate(
    text="Hello from OmniVoice running on Apple Silicon.",
    language="english",
    duration_s=5.0,
    num_steps=32,
):
    audio = result.audio

# Zero-shot voice cloning
for result in model.generate(
    text="This sentence uses the reference speaker.",
    language="english",
    ref_audio="reference.wav",
    ref_text="Transcript of the reference audio.",
    duration_s=5.0,
):
    audio = result.audio

For stable voice cloning, provide ref_text that matches the reference clip. OmniVoice also supports generate_batch() for batched TTS and inline pronunciation controls.

Ming Omni TTS (BailingMM)

bash
mlx_audio.tts.generate \
    --model mlx-community/Ming-omni-tts-16.8B-A3B-bf16 \
    --prompt "Please generate speech based on the following description.\n" \
    --text "This is a quick Ming Omni test." \
    --lang_code en \
    --output_path audio_io \
    --file_prefix ming_basic \
    --verbose

See the Ming Omni TTS README for CLI and Python cookbook examples, and the Ming Omni Dense README for the mlx-community/Ming-omni-tts-0.5B-bf16 workflow.

Kokoro TTS

Kokoro is a fast, multilingual TTS model with 54 voice presets.

python
from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/Kokoro-82M-bf16")
# Or use a quantized variant for lower memory usage:
# model = load_model("mlx-community/Kokoro-82M-8bit")
# model = load_model("mlx-community/Kokoro-82M-4bit")

# Generate with different voices
for result in model.generate(
    text="Welcome to MLX-Audio!",
    voice="af_heart",  # American female
    speed=1.0,
    lang_code="a"  # American English
):
    audio = result.audio

Available Voices:

  • American English: af_heart, af_bella, af_nova, af_sky, am_adam, am_echo, etc.
  • British English: bf_alice, bf_emma, bm_daniel, bm_george, etc.
  • Japanese: jf_alpha, jm_kumo, etc.
  • Chinese: zf_xiaobei, zm_yunxi, etc.

Kokoro requires pip install misaki for text processing. Japanese and Mandarin may additionally require pip install misaki[ja] or pip install misaki[zh].

Language Codes:

Code Language Note
a American English Default; requires pip install misaki
b British English Requires pip install misaki
j Japanese Requires pip install misaki[ja]
z Mandarin Chinese Requires pip install misaki[zh]
e Spanish Requires pip install misaki
f French Requires pip install misaki

CSM (Voice Cloning)

Clone any voice using a reference audio sample:

bash
mlx_audio.tts.generate \
    --model mlx-community/csm-1b \
    --text "Hello from Sesame." \
    --ref_audio ./reference_voice.wav \
    --play

Whisper STT

python
from mlx_audio.stt.generate import generate_transcription

result = generate_transcription(
    model="mlx-community/whisper-large-v3-turbo-asr-fp16",
    audio="audio.wav",
)
print(result.text)

Qwen3-ASR & ForcedAligner

Alibaba's multilingual speech models for transcription and word-level alignment.

python
from mlx_audio.stt import load

# Speech recognition
model = load("mlx-community/Qwen3-ASR-0.6B-8bit")
result = model.generate("audio.wav", language="English")
print(result.text)

# Word-level forced alignment
aligner = load("mlx-community/Qwen3-ForcedAligner-0.6B-8bit")
result = aligner.generate("audio.wav", text="I have a dream", language="English")
for item in result:
    print(f"[{item.start_time:.2f}s - {item.end_time:.2f}s] {item.text}")

See the Qwen3-ASR README for CLI usage, all models, and more examples.

VibeVoice-ASR

Microsoft's 9B parameter speech-to-text model with speaker diarization and timestamps. Supports long-form audio (up to 60 minutes) and outputs structured JSON.

python
from mlx_audio.stt.utils import load

model = load("mlx-community/VibeVoice-ASR-bf16")

# Basic transcription
result = model.generate(audio="meeting.wav", max_tokens=8192, temperature=0.0)
print(result.text)
# [{"Start":0,"End":5.2,"Speaker":0,"Content":"Hello everyone, let's begin."},
#  {"Start":5.5,"End":9.8,"Speaker":1,"Content":"Thanks for joining today."}]

# Access parsed segments
for seg in result.segments:
    print(f"[{seg['start_time']:.1f}-{seg['end_time']:.1f}] Speaker {seg['speaker_id']}: {seg['text']}")

Streaming transcription:

python
# Stream tokens as they are generated
for text in model.stream_transcribe(audio="speech.wav", max_tokens=4096):
    print(text, end="", flush=True)

With context (hotwords/metadata):

python
result = model.generate(
    audio="technical_talk.wav",
    context="MLX, Apple Silicon, PyTorch, Transformer",
    max_tokens=8192,
    temperature=0.0,
)

CLI usage:

bash
# Basic transcription
python -m mlx_audio.stt.generate \
    --model mlx-community/VibeVoice-ASR-bf16 \
    --audio meeting.wav \
    --output-path output \
    --format json \
    --max-tokens 8192 \
    --verbose

# With context/hotwords
python -m mlx_audio.stt.generate \
    --model mlx-community/VibeVoice-ASR-bf16 \
    --audio technical_talk.wav \
    --output-path output \
    --format json \
    --max-tokens 8192 \
    --context "MLX, Apple Silicon, PyTorch, Transformer" \
    --verbose

Parakeet (Multilingual STT)

NVIDIA's high-accuracy speech-to-text model. Parakeet v3 supports 25 European languages.

python
from mlx_audio.stt.utils import load

# Load the multilingual v3 model
model = load("mlx-community/parakeet-tdt-0.6b-v3")

# Transcribe audio
result = model.generate("audio.wav")
print(f"Text: {result.text}")

# Access word-level timestamps
for sentence in result.sentences:
    print(f"[{sentence.start:.2f}s - {sentence.end:.2f}s] {sentence.text}")

Streaming transcription:

python
for chunk in model.generate("long_audio.wav", stream=True):
    print(chunk.text, end="", flush=True)

Supported languages (v3):
Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish, Russian, Ukrainian

CLI usage:

bash
python -m mlx_audio.stt.generate \
    --model mlx-community/parakeet-tdt-0.6b-v3 \
    --audio speech.wav \
    --output-path output \
    --format json \
    --verbose

KugelAudio

SOTA open-source 7B TTS model for 24 European languages, based on Microsoft VibeVoice.
Uses a hybrid AR + Diffusion architecture (Qwen2.5 LM + SDE-DPM-Solver++ diffusion head + VAE decoder).

python
from mlx_audio.tts.utils import load_model

model = load_model("kugelaudio/kugelaudio-0-open")

for result in model.generate(
    text="Hello, welcome to MLX-Audio!",
    cfg_scale=3.0,       # Classifier-free guidance (1.0=fast, 3.0=quality)
    ddpm_steps=10,       # Diffusion steps (5=fast, 10=balanced, 20=max quality)
):
    audio = result.audio  # mx.array, 24kHz

The model loads directly from HuggingFace (weights are remapped automatically via sanitize()).
To quantize or save in a pre-converted format:

bash
python -m mlx_audio.convert \
    --hf-path kugelaudio/kugelaudio-0-open \
    --mlx-path ./kugelaudio-0-open-bf16 \
    --dtype bfloat16

Supported languages (24): English, German, French, Spanish, Italian, Portuguese, Dutch, Polish, Russian, Ukrainian, Czech, Romanian, Hungarian, Swedish, Danish, Finnish, Norwegian, Greek, Bulgarian, Slovak, Croatian, Serbian, Turkish

Note: Requires ~17GB memory (7B params in bfloat16).
Pre-encoded voice presets (voice cloning) are not yet available in the upstream model โ€” the model generates speech with a default voice.

LongCat-AudioDiT

SOTA diffusion-based TTS operating in the waveform latent space. Uses Conditional Flow Matching with a DiT backbone and WAV-VAE codec at 24kHz. Supports zero-shot voice cloning.

python
from mlx_audio.tts.utils import load

model = load("mlx-community/LongCat-AudioDiT-1B-bf16")

# Zero-shot TTS
result = next(model.generate("Hello, this is a test of AudioDiT."))
audio = result.audio  # mx.array, 24kHz

# Voice cloning (use "apg" guidance for best similarity)
result = next(model.generate(
    text="Today is warm turning to rain.",
    ref_audio="reference.wav",
    ref_text="Transcript of the reference audio.",
    guidance_method="apg",
    cfg_strength=4.0,
    steps=16,
))

See the LongCat-AudioDiT README for all parameters and CLI usage.

Voxtral TTS

Mistral's 4B multilingual text-to-speech with 20 voice presets across 9 languages.

python
from mlx_audio.tts.utils import load

model = load("mlx-community/Voxtral-4B-TTS-2603-mlx-bf16")

for result in model.generate(text="Hello, how are you today?", voice="casual_male"):
    print(result.audio_duration)

Voices: casual_male, casual_female, cheerful_female, neutral_male, neutral_female, fr_male, fr_female, es_male, es_female, de_male, de_female, it_male, it_female, pt_male, pt_female, nl_male, nl_female, ar_male, hi_male, hi_female

Voxtral Realtime

Mistral's 4B parameter streaming speech-to-text model, optimized for low-latency transcription.

Available variants: 4bit (smaller/faster) | fp16 (full precision)

python
from mlx_audio.stt.utils import load

# Use 4bit for faster inference, fp16 for full precision
model = load("mlx-community/Voxtral-Mini-4B-Realtime-2602-4bit")

# Transcribe audio
result = model.generate("audio.wav")
print(result.text)

# Streaming transcription
for chunk in model.generate("audio.wav", stream=True):
    print(chunk, end="", flush=True)

# Adjust transcription delay (lower = faster but less accurate)
result = model.generate("audio.wav", transcription_delay_ms=240)

MedASR (Medical Transcription)

Specialized model for medical terms and dictation.

python
from mlx_audio.stt.utils import load, transcribe

model = load("mlx-community/medasr")
result = transcribe("medical_dictation.wav", model=model)
print(result["text"])

Live Transcription Example:

bash
# Continuous live transcription with VAD
python examples/medasr_live.py

SAM-Audio (Source Separation)

Separate specific sounds from audio using text prompts:

python
from mlx_audio.sts import SAMAudio, SAMAudioProcessor, save_audio

model = SAMAudio.from_pretrained("mlx-community/sam-audio-large")
processor = SAMAudioProcessor.from_pretrained("mlx-community/sam-audio-large")

batch = processor(
    descriptions=["A person speaking"],
    audios=["mixed_audio.wav"],
)

result = model.separate_long(
    batch.audios,
    descriptions=batch.descriptions,
    anchors=batch.anchor_ids,
    chunk_seconds=10.0,
    overlap_seconds=3.0,
    ode_opt={"method": "midpoint", "step_size": 2/32},
)

save_audio(result.target[0], "voice.wav")
save_audio(result.residual[0], "background.wav")

MossFormer2 (Speech Enhancement)

Remove noise from speech recordings:

python
from mlx_audio.sts import MossFormer2SEModel, save_audio

model = MossFormer2SEModel.from_pretrained("starkdmi/MossFormer2_SE_48K_MLX")
enhanced = model.enhance("noisy_speech.wav")
save_audio(enhanced, "clean.wav", 48000)

Web Interface & API Server

MLX-Audio includes a modern web interface and OpenAI-compatible API.

Starting the Server

bash
# Start API server
mlx_audio.server --host 0.0.0.0 --port 8000

# Start web UI (in another terminal)
cd mlx_audio/ui
npm install && npm run dev

API Endpoints

Text-to-Speech (OpenAI-compatible):

bash
curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model": "mlx-community/Kokoro-82M-bf16", "input": "Hello!", "voice": "af_heart"}' \
  --output speech.wav

Speech-to-Text:

bash
curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F "[email protected]" \
  -F "model=mlx-community/whisper-large-v3-turbo-asr-fp16"

Quantization

Reduce model size and improve performance with quantization using the convert script:

bash
# Convert and quantize to 4-bit
python -m mlx_audio.convert \
    --hf-path prince-canuma/Kokoro-82M \
    --mlx-path ./Kokoro-82M-4bit \
    --quantize \
    --q-bits 4 \
    --upload-repo username/Kokoro-82M-4bit (optional: if you want to upload the model to Hugging Face)

# Convert with MXFP4 quantization
python -m mlx_audio.convert \
    --hf-path prince-canuma/Kokoro-82M \
    --mlx-path ./Kokoro-82M-mxfp4 \
    --quantize \
    --q-mode mxfp4

# Convert with specific dtype (bfloat16)
python -m mlx_audio.convert \
    --hf-path prince-canuma/Kokoro-82M \
    --mlx-path ./Kokoro-82M-bf16 \
    --dtype bfloat16 \
    --upload-repo username/Kokoro-82M-bf16 (optional: if you want to upload the model to Hugging Face)

Options:

Flag Description
--hf-path Source Hugging Face model or local path
--mlx-path Output directory for converted model
-q, --quantize Enable quantization
--q-bits Bits per weight (optional, defaults depend on --q-mode)
--q-group-size Group size for quantization (optional, defaults depend on --q-mode)
--q-mode Quantization mode: affine, mxfp4, mxfp8, nvfp4
--dtype Weight dtype: float16, bfloat16, float32
--upload-repo Upload converted model to HF Hub

Swift

Looking for Swift/iOS support? Check out mlx-audio-swift for on-device TTS using MLX on macOS and iOS.

Requirements

  • Python 3.10+
  • Apple Silicon Mac (M1/M2/M3/M4)
  • MLX framework
  • ffmpeg (required for MP3/FLAC/OGG/Opus/Vorbis audio encoding)

Installing ffmpeg

ffmpeg is required for saving audio in MP3, FLAC, OGG, Opus, or Vorbis format. Install it using:

bash
# macOS (using Homebrew)
brew install ffmpeg

# Ubuntu/Debian
sudo apt install ffmpeg

WAV format works without ffmpeg.

License

MIT License

Citation

bibtex
@misc{mlx-audio,
  author = {Canuma, Prince},
  title = {MLX Audio},
  year = {2025},
  howpublished = {\url{https://github.com/Blaizzy/mlx-audio}},
  note = {Audio processing library for Apple Silicon with TTS, STT, and STS capabilities.}
}

Acknowledgements


File: docs/guides/index.md

Guides

Practical how-to guides for common mlx-audio workflows.


File: docs/guides/quantization.md

Quantization

Quantization reduces model size and can improve inference speed by storing weights at lower precision. MLX Audio supports several quantization formats through the mlx_audio.convert module.

Why Quantize?

Benefit Details
Smaller models A 4-bit model is roughly 4x smaller than float16
Faster inference Lower-precision weights move through memory faster on Apple Silicon
Lower memory usage Fit larger models into unified memory

The trade-off is a potential reduction in audio quality, especially at very low bit widths.

Available Bit Widths

Bits Typical Use Case Quality Size Reduction
3-bit Maximum compression, acceptable for quick prototyping Lower ~5x vs fp16
4-bit Good balance of quality and size Good ~4x vs fp16
6-bit Near-lossless for most models Very good ~2.5x vs fp16
8-bit Minimal quality loss Excellent ~2x vs fp16

Quantization Modes

MLX Audio supports several quantization modes:

Mode Description
affine Standard affine quantization (default)
mxfp4 Microscaling FP4 format
mxfp8 Microscaling FP8 format
nvfp4 NVIDIA FP4 format

Converting and Quantizing Models

Use the mlx_audio.convert CLI to convert a HuggingFace model to MLX format with quantization:

Basic 4-bit Quantization

bash
python -m mlx_audio.convert \
    --hf-path prince-canuma/Kokoro-82M \
    --mlx-path ./Kokoro-82M-4bit \
    --quantize \
    --q-bits 4

MXFP4 Quantization

bash
python -m mlx_audio.convert \
    --hf-path prince-canuma/Kokoro-82M \
    --mlx-path ./Kokoro-82M-mxfp4 \
    --quantize \
    --q-mode mxfp4

Convert to bfloat16 (No Quantization)

bash
python -m mlx_audio.convert \
    --hf-path prince-canuma/Kokoro-82M \
    --mlx-path ./Kokoro-82M-bf16 \
    --dtype bfloat16

Upload to Hugging Face Hub

Add --upload-repo to push the converted model directly:

bash
python -m mlx_audio.convert \
    --hf-path prince-canuma/Kokoro-82M \
    --mlx-path ./Kokoro-82M-4bit \
    --quantize \
    --q-bits 4 \
    --upload-repo username/Kokoro-82M-4bit

Conversion Options Reference

Flag Description
--hf-path Source HuggingFace model or local path
--mlx-path Output directory for the converted model
-q, --quantize Enable quantization
--q-bits Bits per weight (e.g., 3, 4, 6, 8)
--q-group-size Group size for quantization (defaults depend on mode)
--q-mode Quantization mode: affine, mxfp4, mxfp8, nvfp4
--dtype Weight dtype when not quantizing: float16, bfloat16, float32
--upload-repo Upload converted model to HuggingFace Hub

Using Pre-Quantized Models

The mlx-community organization on Hugging Face hosts many pre-quantized models ready to use. For example:

python
from mlx_audio.tts.utils import load_model

# Load a pre-quantized model -- no conversion needed
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-CustomVoice-6bit")

results = list(model.generate(
    text="This is running from a 6-bit quantized model!",
    voice="serena",
))
python
from mlx_audio.stt.utils import load

# 4-bit Voxtral for faster STT
model = load("mlx-community/Voxtral-Mini-4B-Realtime-2602-4bit")
result = model.generate("audio.wav")
print(result.text)

Quality vs. Performance Trade-offs

!!! info "General guidance"
Start with 4-bit or 6-bit quantization. Move to 8-bit only if you hear artifacts, or to 3-bit if you need the smallest possible model.

  • TTS models tend to be sensitive to quantization at 3-bit. 4-bit is usually the sweet spot.
  • STT models (e.g., Whisper, Voxtral Realtime) often tolerate 4-bit quantization with minimal accuracy loss.
  • Larger models (1B+ parameters) generally tolerate lower bit widths better than smaller ones.
  • Always listen to output when evaluating TTS quantization -- word error rate alone does not capture prosody degradation.

Mixed Quantization Recipes

MLX Audio also supports mixed-precision quantization recipes that apply different bit widths to different layers:

Recipe Description
mixed_2_6 2-bit for some layers, 6-bit for others
mixed_3_4 3-bit / 4-bit mix
mixed_3_6 3-bit / 6-bit mix
mixed_4_6 4-bit / 6-bit mix

These can provide better quality-to-size ratios by keeping critical layers at higher precision.


File: docs/guides/streaming.md

Streaming Audio

MLX Audio supports streaming audio generation for low-latency playback. Instead of waiting for the entire utterance to be synthesized, you can start playing audio chunks as they are produced.

CLI Streaming

Add the --stream flag to any TTS generation command. When streaming is enabled, audio is played back in real time as chunks become available:

bash
mlx_audio.tts.generate \
    --model mlx-community/Kokoro-82M-bf16 \
    --text "Hello, this is a streaming example!" \
    --lang_code a \
    --stream

!!! tip "Streaming implies playback"
The --stream flag automatically enables --play, so there is no need to pass both.

Controlling Chunk Size

The --streaming_interval argument controls how frequently audio chunks are emitted (in seconds). Smaller values reduce latency but add per-chunk overhead:

bash
mlx_audio.tts.generate \
    --model mlx-community/Kokoro-82M-bf16 \
    --text "Adjusting the streaming interval changes latency." \
    --lang_code a \
    --stream \
    --streaming_interval 1.5

The default interval is 2.0 seconds.

Python Streaming

TTS Streaming

Every model's generate() method accepts stream=True. When enabled, it yields GenerationResult objects as chunks rather than waiting for full synthesis:

python
from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/Kokoro-82M-bf16")

for result in model.generate(
    text="This audio will stream chunk by chunk.",
    voice="af_heart",
    lang_code="a",
    stream=True,
    streaming_interval=2.0,
):
    # result.audio is an mx.array with one chunk of audio
    print(f"Chunk: {result.audio.shape[0]} samples")
    # Feed result.audio to an audio player or buffer

Each GenerationResult chunk contains:

Attribute Description
audio mx.array waveform for this chunk
sample_rate Sample rate in Hz
is_streaming_chunk True for intermediate chunks
is_final_chunk True for the last chunk

Qwen3-TTS Streaming

Qwen3-TTS models support streaming across all generation methods -- generate(), generate_custom_voice(), and generate_voice_design():

python
from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-CustomVoice-6bit")

audio_chunks = []
for result in model.generate(
    text="Hello, how are you today?",
    voice="serena",
    stream=True,
    streaming_interval=0.32,  # ~4 tokens at 12.5 Hz
):
    audio_chunks.append(result.audio)
    # Play or process each chunk for low-latency output

!!! info "Streaming interval for Qwen3-TTS"
At 12.5 Hz token rate, a streaming_interval of 0.32 seconds corresponds to roughly 4 tokens per chunk. Lower values reduce latency but increase overhead.

STT Streaming

Several speech-to-text models support streaming transcription. Pass stream=True to generate():

=== "Voxtral Realtime"

text
```python
from mlx_audio.stt.utils import load

model = load("mlx-community/Voxtral-Mini-4B-Realtime-2602-4bit")

for chunk in model.generate("audio.wav", stream=True):
    print(chunk, end="", flush=True)
```

=== "Parakeet"

text
```python
from mlx_audio.stt.utils import load

model = load("mlx-community/parakeet-tdt-0.6b-v3")

for chunk in model.generate("long_audio.wav", stream=True):
    print(chunk.text, end="", flush=True)
```

=== "VibeVoice-ASR"

text
```python
from mlx_audio.stt.utils import load

model = load("mlx-community/VibeVoice-ASR-bf16")

for text in model.stream_transcribe(audio="speech.wav", max_tokens=4096):
    print(text, end="", flush=True)
```

STT Streaming via CLI

bash
python -m mlx_audio.stt.generate \
    --model mlx-community/whisper-large-v3-turbo-asr-fp16 \
    --audio speech.wav \
    --output-path output \
    --format json \
    --stream

API Server Streaming

The API server supports streaming for both TTS and STT. Set "stream": true in your request:

Streaming TTS

bash
curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Kokoro-82M-bf16",
    "input": "Streaming over HTTP!",
    "voice": "af_heart",
    "stream": true,
    "streaming_interval": 2.0,
    "response_format": "wav"
  }' \
  --output streamed_speech.wav

Streaming STT

bash
curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F "[email protected]" \
  -F "model=mlx-community/whisper-large-v3-turbo-asr-fp16" \
  -F "stream=true"

Streaming STT returns newline-delimited JSON (application/x-ndjson), with each line containing a text field and optional timing information.

Real-Time WebSocket Transcription

The API server also exposes a WebSocket endpoint for real-time audio transcription:

text
ws://localhost:8000/v1/audio/transcriptions/realtime

This is useful for live microphone input or continuous audio streams.

Tips

  • Latency vs. quality -- Shorter streaming_interval values give lower latency but may produce more chunk boundaries. Start with the default and decrease as needed.
  • Memory -- Streaming does not significantly change peak memory usage since model weights remain loaded; it only changes when decoded audio is returned.
  • Joining chunks -- If you need a single contiguous file, concatenate the result.audio arrays after the loop and write once with mlx_audio.audio_io.write().

File: docs/guides/voice-cloning.md

Voice Cloning

Several MLX Audio models can clone a speaker's voice from a short reference audio sample. This guide covers the supported models and how to use them.

Overview

Model Method Reference Audio Required Notes
CSM / MisoTTS --ref_audio CLI / ref_audio kwarg Yes (WAV) Sesame-style conversational speech models
Qwen3-TTS Base ref_audio + ref_text kwargs Yes (WAV) + transcript Alibaba multilingual TTS
OmniVoice ref_audio + ref_text kwargs Yes (WAV) + transcript recommended 646+ language zero-shot cloning, best with prompt preprocessing
Spark ref_audio kwarg Yes SparkTTS voice cloning
Chatterbox ref_audio kwarg Yes Expressive multilingual TTS
OuteTTS ref_audio kwarg Yes Efficient TTS with cloning
Ming Omni TTS ref_audio kwarg Yes Multimodal with voice cloning

CSM (Conversational Speech Model)

CSM is the simplest path to voice cloning. Provide a WAV file of the target voice and CSM will match it:

CLI

bash
mlx_audio.tts.generate \
    --model mlx-community/csm-1b \
    --text "Hello from Sesame." \
    --ref_audio ./reference_voice.wav \
    --play

Python

python
from mlx_audio.tts.utils import load_model
from mlx_audio.utils import load_audio

model = load_model("mlx-community/csm-1b")

for result in model.generate(
    text="Hello from Sesame.",
    ref_audio="./reference_voice.wav",
):
    audio = result.audio

!!! tip "Automatic transcription"
If you do not provide --ref_text, MLX Audio will automatically transcribe the reference audio using a Whisper model. You can specify which STT model to use with --stt_model.

Qwen3-TTS

Qwen3-TTS Base models support voice cloning by providing both a reference audio file and its transcript:

python
from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/Qwen3-TTS-12Hz-0.6B-Base-bf16")

results = list(model.generate(
    text="Hello, welcome to MLX-Audio!",
    ref_audio="sample_audio.wav",
    ref_text="This is what my voice sounds like.",
))

audio = results[0].audio  # mx.array

CustomVoice (Emotion Control)

The CustomVoice variant lets you combine a predefined voice with emotion and style instructions:

python
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-CustomVoice-bf16")

results = list(model.generate_custom_voice(
    text="I'm so excited to meet you!",
    speaker="Vivian",
    language="English",
    instruct="Very happy and excited.",
))

VoiceDesign (Create Any Voice)

The VoiceDesign variant creates a voice from a text description -- no reference audio needed:

python
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-VoiceDesign-bf16")

results = list(model.generate_voice_design(
    text="Big brother, you're back!",
    language="English",
    instruct="A cheerful young female voice with high pitch and energetic tone.",
))

Available Qwen3-TTS Models

Model Method Description
Qwen3-TTS-12Hz-0.6B-Base-bf16 generate() Fast, predefined voices + cloning
Qwen3-TTS-12Hz-1.7B-Base-bf16 generate() Higher quality
Qwen3-TTS-12Hz-0.6B-CustomVoice-bf16 generate_custom_voice() Voices + emotion
Qwen3-TTS-12Hz-1.7B-CustomVoice-bf16 generate_custom_voice() Better emotion control
Qwen3-TTS-12Hz-1.7B-VoiceDesign-bf16 generate_voice_design() Create any voice from a description

Available Speakers (Base / CustomVoice)

  • Chinese: Vivian, Serena, Uncle_Fu, Dylan (Beijing Dialect), Eric (Sichuan Dialect)
  • English: Ryan, Aiden

Ming Omni TTS

Ming Omni TTS supports voice cloning and style control:

bash
mlx_audio.tts.generate \
    --model mlx-community/Ming-omni-tts-16.8B-A3B-bf16 \
    --text "This is a Ming Omni voice cloning test." \
    --ref_audio ./reference_voice.wav \
    --lang_code en \
    --verbose

See the Ming Omni TTS model page for detailed cookbook examples.

Best Practices

Preparing Reference Audio

!!! warning "Quality matters"
The quality of your cloned voice depends heavily on the reference audio.

  • Duration -- 5 to 15 seconds of clean speech works best. Very short clips lack enough speaker information; very long clips may confuse the model.
  • Format -- Use WAV at 16 kHz or higher. The library will resample automatically, but starting with a good sample rate avoids artifacts.
  • Noise -- Record in a quiet environment. Background noise will be cloned along with the voice. Consider using MossFormer2 or DeepFilterNet to enhance noisy recordings first.
  • Content -- Natural, conversational speech produces the best results. Avoid whispering or shouting unless that is the target style.

Providing Reference Text

Some models (Qwen3-TTS) require a transcript of the reference audio (ref_text). If you omit it, MLX Audio will transcribe the clip automatically using Whisper:

bash
mlx_audio.tts.generate \
    --model mlx-community/Qwen3-TTS-12Hz-0.6B-Base-bf16 \
    --text "Cloned speech output." \
    --ref_audio reference.wav \
    --stt_model mlx-community/whisper-large-v3-turbo-asr-fp16

Providing the transcript yourself avoids loading the STT model and speeds up generation.

OmniVoice

OmniVoice supports multilingual zero-shot voice cloning with a HiggsAudioV2 acoustic tokenizer and iterative masked generation.

!!! warning "ref_text must match preprocessed audio"
OmniVoice preprocessing removes silence and trims the reference clip. If you transcribe the original file, the ASR transcript may be longer than the preprocessed audio, causing the extra text to leak into generation. Always transcribe the preprocessed audio, not the raw recording. See examples/omnivoice_clone_demo.py for the correct workflow.

python
from mlx_audio.tts.utils import load_model as load_tts
from mlx_audio.tts.models.omnivoice.utils import create_voice_clone_prompt
from mlx_audio.stt.utils import load_model as load_stt
from mlx_audio.audio_io import write as audio_write
import mlx.core as mx, numpy as np, tempfile, os

tts = load_tts("mlx-community/OmniVoice-bf16")

# Preprocess โ†’ encode โ†’ decode โ†’ transcribe (matches original pipeline)
ref_tokens = create_voice_clone_prompt("reference.wav", tokenizer=tts.audio_tokenizer)
mx.eval(ref_tokens)

preprocessed = np.array(tts.audio_tokenizer.decode(ref_tokens).astype(mx.float32))
tmp = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
audio_write(tmp.name, preprocessed, 24000); tmp.close()

ref_text = load_stt("mlx-community/Qwen3-ASR-0.6B-8bit").generate(tmp.name).text
os.unlink(tmp.name)

results = list(tts.generate(
    text="Hello from OmniVoice.",
    language="english",
    ref_tokens=ref_tokens,
    ref_text=ref_text,
))

audio_write("output.wav", np.array(results[0].audio), results[0].sample_rate)

OmniVoice-specific notes

  • Reference text is required for stable cloning. Without it, output quality degrades significantly โ€” garbled speech, wrong language, or missing words.
  • Transcribe after preprocessing, not before. The original k2-fsa/OmniVoice runs Whisper on audio after silence removal. This demo replicates that approach.
  • Prompt preprocessing matters. MLX Audio mirrors the original Python pipeline with RMS normalization, silence removal, trimming at silence gaps, and torchaudio-compatible resampling before reference encoding.
  • Best reference length: roughly 5โ€“15 seconds of actual speech after silence trimming.
  • Supported inline controls: nonverbal tags such as [laughter], [sigh], and pronunciation overrides for English CMU dictionary forms and Chinese pinyin forms.

Example: English CMU pronunciation control

python
results = list(model.generate(
    text="He plays the [B EY1 S] guitar while catching a [B AE1 S] fish.",
    language="english",
))

Example: Nonverbal tags

python
results = list(model.generate(
    text="I just heard the funniest joke [laughter] that was incredible.",
    language="english",
))

Combining Cloning with Streaming

Voice cloning and streaming work together. Add --stream to any cloning command:

bash
mlx_audio.tts.generate \
    --model mlx-community/csm-1b \
    --text "Streaming with a cloned voice." \
    --ref_audio reference.wav \
    --stream

File: docs/guides/web-ui-api-server.md

Web UI & API Server

MLX Audio ships with a FastAPI-based API server and a Next.js web interface (Studio UI) for interactive audio generation and transcription.

Starting the Server

API Server Only

bash
mlx_audio.server --host 0.0.0.0 --port 8000

API Server with Studio UI

Pass --start-ui to launch the Next.js web interface alongside the API server:

bash
mlx_audio.server --host 0.0.0.0 --port 8000 --start-ui

The API will be available at http://localhost:8000 and the Studio UI at http://localhost:3000.

Server Options

Flag Default Description
--host localhost Host to bind the server to
--port 8000 Port for the API server
--reload false Auto-reload on code changes (development)
--start-ui false Start the Studio UI alongside the API
--allowed-origins * CORS allowed origins (space-separated)
--log-dir logs Directory for server logs
--realtime-model null Default model for /v1/realtime when the client omits ?model=
--realtime-transcription-delay-ms null Transcription latency/quality knob for models that support it (e.g. voxtral_realtime)
--vad-model mlx-community/silero-vad Streaming VAD model used for server-side turn detection (server_vad) on /v1/realtime
--tts-max-batch-size 8 Maximum compatible TTS speech requests per continuous batch session

The two realtime flags also read from MLX_AUDIO_REALTIME_MODEL and MLX_AUDIO_REALTIME_TRANSCRIPTION_DELAY_MS if present; the CLI flags take precedence. --vad-model likewise reads from MLX_AUDIO_VAD_MODEL.
The TTS batching flag also reads from MLX_AUDIO_TTS_MAX_BATCH_SIZE; the CLI flag takes precedence.

CORS Configuration

By default, all origins are allowed. To restrict origins:

bash
mlx_audio.server --allowed-origins http://localhost:3000 https://myapp.example.com

Or set the MLX_AUDIO_ALLOWED_ORIGINS environment variable with a comma-separated list.

TTS Continuous Batching

Compatible non-streaming TTS /v1/audio/speech requests are routed through the server's continuous batching path when the model exposes a create_tts_batch_session(...) hook.

Requests that are not supported by the model's continuous batching hook continue through the existing serial or fixed-window batch paths.

bash
mlx_audio.server --host 127.0.0.1 --port 8000 \
  --tts-max-batch-size 8

OpenAI-Compatible API

The server implements OpenAI-compatible endpoints, so existing code that targets the OpenAI audio API can point to MLX Audio with minimal changes.

Text-to-Speech

POST /v1/audio/speech

bash
curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Kokoro-82M-bf16",
    "input": "Hello, world!",
    "voice": "af_heart",
    "response_format": "mp3"
  }' \
  --output speech.mp3

Request body:

Field Type Default Description
model string required Model ID (HuggingFace repo or local path)
input string required Text to synthesize
voice string null Voice preset name
speed float 1.0 Playback speed multiplier
lang_code string "a" Language code
response_format string "mp3" Output format: mp3, wav, flac, ogg, opus
stream bool false Stream audio chunks
streaming_interval float 2.0 Seconds between stream chunks
temperature float 0.7 Sampling temperature
max_tokens int 1200 Maximum generation tokens
ref_audio string null Path to reference audio for voice cloning
ref_text string null Transcript of reference audio
instruct string null Style/emotion instruction

Streaming TTS

bash
curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Kokoro-82M-bf16",
    "input": "Streaming audio over HTTP.",
    "voice": "af_heart",
    "stream": true,
    "response_format": "wav"
  }' \
  --output streamed.wav

Speech-to-Text

POST /v1/audio/transcriptions

bash
curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F "[email protected]" \
  -F "model=mlx-community/whisper-large-v3-turbo-asr-fp16"

Form fields:

Field Type Default Description
file file required Audio file to transcribe
model string required STT model ID
language string null Language code
max_tokens int 1024 Maximum output tokens
stream bool false Stream results as NDJSON
context string null Hotwords or metadata to guide transcription
verbose bool false Include extra details

Audio Source Separation

POST /v1/audio/separations

bash
curl -X POST http://localhost:8000/v1/audio/separations \
  -F "[email protected]" \
  -F "model=mlx-community/sam-audio-large-fp16" \
  -F "description=speech"

Returns JSON with base64-encoded target and residual WAV buffers.

Model Management

bash
# List loaded models
curl http://localhost:8000/v1/models

# Load a model
curl -X POST "http://localhost:8000/v1/models?model_name=mlx-community/Kokoro-82M-bf16"

# Unload a model
curl -X DELETE "http://localhost:8000/v1/models?model_name=mlx-community/Kokoro-82M-bf16"

Real-Time WebSocket Transcription

The server exposes two WebSocket endpoints for live transcription. Both accept 16-bit signed little-endian PCM audio; they differ in wire protocol and intended consumers.

`/v1/audio/transcriptions/realtime` (VAD-based streaming)

text
ws://localhost:8000/v1/audio/transcriptions/realtime

Send raw PCM frames as binary WebSocket messages; the server performs VAD, chunks on silence, and emits transcription JSON messages back. Uses the preloaded Whisper-style STT model โ€” connect after calling POST /v1/models to select it.

`/v1/realtime` (OpenAI Realtime-compatible)

text
ws://localhost:8000/v1/realtime?model=<model-id>

Implements a subset of the OpenAI Realtime API wire protocol, so existing Realtime clients can target MLX Audio by swapping the base URL. Any STT model that exposes create_streaming_session works โ€” including voxtral_realtime, which is streaming-first and recommended.

Model selection order (first match wins):

  1. ?model= query parameter on connect.
  2. session.update.model (or session.audio.input.transcription.model) event after connect.
  3. --realtime-model CLI flag / MLX_AUDIO_REALTIME_MODEL env var.

If none is set, the server replies with an error event and closes the socket.

Client โ†’ server events (subset of OpenAI Realtime):

Type Purpose
session.update Change the model, input sample rate, transcription config, or turn detection mode
input_audio_buffer.append Append a chunk of base64-encoded PCM16 to the current item
input_audio_buffer.commit Signal end-of-utterance; the server drains deltas and emits completed. Manual-commit mode only โ€” unused under server_vad

Declaring the input sample rate. The server assumes incoming PCM is 24 kHz by default (the OpenAI Realtime client convention). If you are sending audio at a different rate โ€” e.g. a 16 kHz microphone capture or a 48 kHz file โ€” tell the server via session.update so it resamples correctly. Without this, 16 kHz audio interpreted as 24 kHz would sound sped-up to the model and transcribe as garbage.

json
{
  "type": "session.update",
  "session": {
    "audio": {"input": {"format": {"type": "audio/pcm", "rate": 16000}}}
  }
}

This only declares your input rate. The server resamples from that to whatever rate the model expects internally โ€” you never need to match the model's native rate.

Turn detection (server_vad). By default (turn_detection: null) the client owns turn boundaries and signals end-of-utterance with input_audio_buffer.commit. Set turn_detection to {"type": "server_vad"} via session.update and the server instead runs a streaming VAD (Silero by default โ€” see --vad-model) over the incoming audio: it detects when the speaker starts and stops, emits input_audio_buffer.speech_started / input_audio_buffer.speech_stopped, and auto-commits each turn. In this mode the client never sends input_audio_buffer.commit.

json
{
  "type": "session.update",
  "session": {
    "audio": {
      "input": {
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.5,
          "prefix_padding_ms": 300,
          "silence_duration_ms": 500
        }
      }
    }
  }
}
Field Default Purpose
threshold 0.5 Speech-probability threshold above which a VAD frame counts as speech
prefix_padding_ms 300 Shifts the reported audio_start_ms earlier; does not gate transcription
silence_duration_ms 500 Sub-threshold audio after speech before the turn is considered finished

To use a different VAD model, launch the server with --vad-model (or set MLX_AUDIO_VAD_MODEL). semantic_vad is not implemented yet โ€” requesting it returns an error event.

Server โ†’ client events:

Type Purpose
session.created / session.updated Session snapshot (id, model, input format, turn detection)
conversation.item.added New user item opened for the next audio chunk
input_audio_buffer.speech_started (server_vad) Speaker started; carries audio_start_ms
input_audio_buffer.speech_stopped (server_vad) Speaker stopped; the server auto-commits the turn
input_audio_buffer.committed Acknowledges a commit โ€” client commit or server_vad auto-commit
conversation.item.input_audio_transcription.delta Incremental transcript token(s)
conversation.item.input_audio_transcription.completed Final transcript for the item
error Error message; the socket may close afterwards

Minimal Python client:

python
import asyncio, base64, json, websockets, numpy as np, soundfile as sf

async def transcribe(path: str):
    audio, sr = sf.read(path, dtype="int16", always_2d=False)
    uri = "ws://localhost:8000/v1/realtime?model=iris-sfg/Voxtral-Mini-4B-Realtime-2602-4bit"
    async with websockets.connect(uri) as ws:
        # Wait for session.created
        await ws.recv()
        # Declare the input rate (the server will resample to the model's rate)
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {"audio": {"input": {"format": {"type": "audio/pcm", "rate": sr}}}},
        }))
        # Stream audio in 100 ms chunks
        step = sr // 10
        for i in range(0, len(audio), step):
            chunk = audio[i : i + step].tobytes()
            await ws.send(json.dumps({
                "type": "input_audio_buffer.append",
                "audio": base64.b64encode(chunk).decode(),
            }))
        await ws.send(json.dumps({"type": "input_audio_buffer.commit"}))

        async for raw in ws:
            evt = json.loads(raw)
            if evt["type"].endswith(".delta"):
                print(evt["delta"], end="", flush=True)
            elif evt["type"].endswith(".completed"):
                print()
                break

asyncio.run(transcribe("audio.wav"))

Tuning transcription delay (for models like voxtral_realtime that expose the knob):

bash
mlx_audio.server \
  --realtime-model iris-sfg/Voxtral-Mini-4B-Realtime-2602-4bit \
  --realtime-transcription-delay-ms 960

Lower values reduce latency at the cost of accuracy. Models that don't declare a transcription_delay_ms parameter silently ignore the flag.

!!! note "Single-process MLX serialization"
MLX inference on a given device is serialized inside the server. /v1/realtime schedules session.step() calls cooperatively โ€” two concurrent websocket clients share GPU time rather than running in parallel. This is fine as long as each stream transcribes faster than real-time; otherwise run multiple server instances behind a load balancer.

Web Interface (Studio UI)

The Studio UI is a Next.js application located in mlx_audio/ui/. It provides:

  • A text input for TTS generation with voice and model selectors
  • Audio upload and recording for STT
  • 3D audio visualization
  • Model loading and management

Running the UI Separately

If you prefer to run the UI outside the server process:

bash
cd mlx_audio/ui
npm install
npm run dev

The UI runs on http://localhost:3000 and expects the API server at http://localhost:8000.

Using with the OpenAI Python Client

Because the API is OpenAI-compatible, you can use the official OpenAI Python client:

python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",  # MLX Audio does not require an API key
)

# Text-to-Speech
response = client.audio.speech.create(
    model="mlx-community/Kokoro-82M-bf16",
    voice="af_heart",
    input="Hello from the OpenAI client!",
)
response.stream_to_file("output.mp3")

# Speech-to-Text
with open("audio.wav", "rb") as f:
    transcript = client.audio.transcriptions.create(
        model="mlx-community/whisper-large-v3-turbo-asr-fp16",
        file=f,
    )
print(transcript.text)

Installation

The server requires the server optional dependency group:

bash
pip install "mlx-audio[server]"

This installs FastAPI, Uvicorn, and python-multipart.


File: docs/getting-started/index.md

Getting Started

Welcome to MLX Audio! This section will help you install the library and run your first text-to-speech generation or speech transcription in minutes.

Requirements

  • Python 3.10+
  • Apple Silicon Mac (M1 / M2 / M3 / M4)
  • ffmpeg (required for MP3, FLAC, OGG, Opus, or Vorbis encoding; WAV works without it)

Where to Go Next


File: docs/getting-started/installation.md

Installation

Quick Install

=== "pip"

text
```bash
pip install mlx-audio
```

=== "uv (tool)"

text
Install the CLI tools globally with uv:

```bash
# Latest release from PyPI
uv tool install --force mlx-audio --prerelease=allow
```

Or install from the latest source:

```bash
# Latest code from GitHub
uv tool install --force git+https://github.com/Blaizzy/mlx-audio.git --prerelease=allow
```

Optional Extras

mlx-audio ships optional dependency groups so you only install what you need:

Extra What it includes
tts Text-to-speech dependencies
stt Speech-to-text dependencies
sts Speech-to-speech dependencies
docs MkDocs and API reference tooling
all Everything (TTS + STT + STS)
dev Development and testing tools

Install extras with bracket syntax:

bash
# TTS only
pip install "mlx-audio[tts]"

# STT only
pip install "mlx-audio[stt]"

# Everything
pip install "mlx-audio[all]"

# Documentation tooling
pip install "mlx-audio[docs]"

Development Setup

Clone the repository and install in editable mode with dev dependencies:

bash
git clone https://github.com/Blaizzy/mlx-audio.git
cd mlx-audio
pip install -e ".[dev]"

If you are working on the docs site, use:

bash
pip install -e ".[dev,docs]"

ffmpeg

ffmpeg is required for saving audio in MP3, FLAC, OGG, Opus, or Vorbis format. WAV works without it.

=== "macOS (Homebrew)"

text
```bash
brew install ffmpeg
```

=== "Ubuntu / Debian"

text
```bash
sudo apt install ffmpeg
```

!!! tip
If you only need WAV output, you can skip installing ffmpeg entirely.


File: docs/getting-started/quickstart-cli.md

Quick Start: CLI

mlx-audio provides command-line tools for both text-to-speech generation and speech-to-text transcription.

Text-to-Speech

!!! note
These TTS quickstart examples use mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit.

Basic Generation

Generate speech from text with a single command:

bash
mlx_audio.tts.generate \
    --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit \
    --text "Hello, world!" \
    --voice Chelsie \
    --lang_code English

Play Audio Immediately

Add --play to hear the result without saving:

bash
mlx_audio.tts.generate \
    --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit \
    --text "Hello, world!" \
    --voice Chelsie \
    --lang_code English \
    --play

Voice and Language Selection

Choose a voice preset and provide a language hint:

bash
mlx_audio.tts.generate \
    --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit \
    --text "Welcome to MLX-Audio!" \
    --voice Ethan \
    --lang_code English

Save to a Directory

bash
mlx_audio.tts.generate \
    --model mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit \
    --text "Hello!" \
    --voice Chelsie \
    --lang_code English \
    --output_path ./my_audio

Voice Cloning (CSM)

Clone a voice from a reference audio file:

bash
mlx_audio.tts.generate \
    --model mlx-community/csm-1b \
    --text "Hello from Sesame." \
    --ref_audio ./reference_voice.wav \
    --play

Speech-to-Text

Transcribe with Whisper

bash
python -m mlx_audio.stt.generate \
    --model mlx-community/whisper-large-v3-turbo-asr-fp16 \
    --audio audio.wav \
    --output-path output \
    --format json \
    --verbose

Transcribe with Parakeet

bash
python -m mlx_audio.stt.generate \
    --model mlx-community/parakeet-tdt-0.6b-v3 \
    --audio speech.wav \
    --output-path output \
    --format json \
    --verbose

Transcribe with VibeVoice-ASR

bash
python -m mlx_audio.stt.generate \
    --model mlx-community/VibeVoice-ASR-bf16 \
    --audio meeting.wav \
    --output-path output \
    --format json \
    --max-tokens 8192 \
    --verbose

Add context/hotwords for better accuracy on domain-specific terms:

bash
python -m mlx_audio.stt.generate \
    --model mlx-community/VibeVoice-ASR-bf16 \
    --audio technical_talk.wav \
    --output-path output \
    --format json \
    --max-tokens 8192 \
    --context "MLX, Apple Silicon, PyTorch, Transformer" \
    --verbose

Common Flags

Flag Description
--model Hugging Face model ID or local path
--text Input text for TTS generation
--audio Input audio file for STT transcription
--voice Voice preset name (e.g., Chelsie, Ethan, casual_male)
--speed Speech speed multiplier (default: 1.0)
--lang_code Language hint (e.g., English, Chinese, or auto)
--play Play audio immediately after generation
--output_path Directory to save output files
--verbose Show detailed generation info
--format Output format for STT (json, etc.)

!!! note
Different models support different flags. Check the Models section for model-specific options.


File: docs/getting-started/quickstart-python.md

Quick Start: Python

Use the mlx-audio Python API to generate speech, transcribe audio, and process audio programmatically.

Text-to-Speech

!!! note
These TTS quickstart examples use mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit.

Basic Generation

python
from mlx_audio.tts.utils import load_model

# Load a TTS model
model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit")

# Generate speech
for result in model.generate(
    "Hello from MLX-Audio!",
    voice="Chelsie",
    lang_code="English",
):
    print(f"Generated {result.audio.shape[0]} samples")
    # result.audio contains the waveform as mx.array

Voice and Language Options

python
from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit")

for result in model.generate(
    text="Welcome to MLX-Audio!",
    voice="Ethan",
    lang_code="English",
):
    audio = result.audio

!!! tip "Qwen3-TTS Base Model"
Pass voice to pick a preset speaker and lang_code to hint the language, for example voice="Chelsie" and lang_code="English".

Voxtral TTS

python
from mlx_audio.tts.utils import load

model = load("mlx-community/Voxtral-4B-TTS-2603-mlx-bf16")

for result in model.generate(text="Hello, how are you today?", voice="casual_male"):
    print(result.audio_duration)

Qwen3-TTS

python
from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/Qwen3-TTS-12Hz-1.7B-Base-8bit")
results = list(model.generate(
    text="Hello, welcome to MLX-Audio!",
    voice="Chelsie",
    lang_code="English",
))

audio = results[0].audio  # mx.array

Speech-to-Text

Whisper

python
from mlx_audio.stt.generate import generate_transcription

result = generate_transcription(
    model="mlx-community/whisper-large-v3-turbo-asr-fp16",
    audio="audio.wav",
)
print(result.text)

Qwen3-ASR

python
from mlx_audio.stt import load

# Speech recognition
model = load("mlx-community/Qwen3-ASR-0.6B-8bit")
result = model.generate("audio.wav", language="English")
print(result.text)

Word-Level Alignment (Qwen3-ForcedAligner)

python
from mlx_audio.stt import load

aligner = load("mlx-community/Qwen3-ForcedAligner-0.6B-8bit")
result = aligner.generate("audio.wav", text="I have a dream", language="English")

for item in result:
    print(f"[{item.start_time:.2f}s - {item.end_time:.2f}s] {item.text}")

Parakeet (with Timestamps)

python
from mlx_audio.stt.utils import load

model = load("mlx-community/parakeet-tdt-0.6b-v3")
result = model.generate("audio.wav")

print(f"Text: {result.text}")

# Word-level timestamps
for sentence in result.sentences:
    print(f"[{sentence.start:.2f}s - {sentence.end:.2f}s] {sentence.text}")

Streaming Transcription

Several STT models support streaming for low-latency output:

=== "Voxtral Realtime"

text
```python
from mlx_audio.stt.utils import load

model = load("mlx-community/Voxtral-Mini-4B-Realtime-2602-4bit")

for chunk in model.generate("audio.wav", stream=True):
    print(chunk, end="", flush=True)
```

=== "Parakeet"

text
```python
from mlx_audio.stt.utils import load

model = load("mlx-community/parakeet-tdt-0.6b-v3")

for chunk in model.generate("long_audio.wav", stream=True):
    print(chunk.text, end="", flush=True)
```

=== "VibeVoice"

text
```python
from mlx_audio.stt.utils import load

model = load("mlx-community/VibeVoice-ASR-bf16")

for text in model.stream_transcribe(audio="speech.wav", max_tokens=4096):
    print(text, end="", flush=True)
```

Saving Audio

Save with STS Utilities

python
from mlx_audio.sts import save_audio

# Save an audio array to a WAV file
save_audio(audio_array, "output.wav", sample_rate=24000)

!!! note
Saving to MP3, FLAC, OGG, Opus, or Vorbis requires ffmpeg. WAV works without it.