## File: README.md

[Argmax](https://argmaxinc.com/blog) Open-Source SDK Swift is a collection of turn-key on-device inference frameworks:
- **WhisperKit** for speech-to-text with OpenAI Whisper
- **SpeakerKit** for speaker diarization with Pyannote
- **TTSKit** for text-to-speech with Qwen-TTS

> [!IMPORTANT]
> [Argmax Pro SDK](https://www.argmaxinc.com/blog/argmax-sdk-2) supports additional models and advanced features such as:
> - Real-time transcription with speakers
> - Frontier accuracy for your use case with custom vocabulary
> - Argmax Local Server for non-native apps
> - Android support with Argmax Pro SDK Kotlin
> 
> Further resources:
> - [Open-source vs Pro SDK](https://app.argmaxinc.com/docs/wiki/open-source-vs-pro-sdk)
> - [Try Pro SDK on TestFlight](https://testflight.apple.com/join/Q1cywTJw)
> - [Model Gallery](https://app.argmaxinc.com/docs/models)

## Table of Contents

- [Installation](#installation)
  - [Swift Package Manager](#swift-package-manager)
  - [Prerequisites](#prerequisites)
  - [Xcode Steps](#xcode-steps)
  - [Package.swift](#packageswift)
  - [Homebrew](#homebrew)
- [WhisperKit](#whisperkit)
  - [Quick Example](#quick-example)
  - [Memory-Efficient Loading for Large Files](#memory-efficient-loading-for-large-files)
  - [Model Selection](#model-selection)
  - [Generating Models](#generating-models)
  - [Swift CLI](#swift-cli)
  - [Local Server](#local-server)
    - [Building the Server](#building-the-server)
    - [Starting the Server](#starting-the-server)
    - [API Endpoints](#api-endpoints)
    - [Supported Parameters](#supported-parameters)
    - [Client Examples](#client-examples)
    - [Generating the API Specification](#generating-the-api-specification)
    - [Client Generation](#client-generation)
    - [API Limitations](#api-limitations)
    - [Fully Supported Features](#fully-supported-features)
- [TTSKit](#ttskit)
  - [Quick Example](#quick-example-1)
  - [Model Selection](#model-selection-1)
    - [Custom Voices](#custom-voices)
    - [Real-Time Streaming Playback](#real-time-streaming-playback)
  - [Generation Options](#generation-options)
    - [Style Instructions (1.7B only)](#style-instructions-17b-only)
  - [Saving Audio](#saving-audio)
  - [Progress Callbacks](#progress-callbacks)
  - [Swift CLI](#swift-cli-1)
  - [Demo App](#demo-app)
- [SpeakerKit](#speakerkit)
  - [Quick Example](#quick-example-2)
  - [Diarization Options](#diarization-options)
  - [Combining with Transcription](#combining-with-transcription)
  - [RTTM Output](#rttm-output)
  - [Swift CLI](#swift-cli-2)
- [Contributing \& Roadmap](#contributing--roadmap)
- [License](#license)
- [Citation](#citation)

## Installation

### Swift Package Manager

WhisperKit, TTSKit, and SpeakerKit are separate library products in the same Swift package. Add the package once and pick the products you need. You can also use the `ArgmaxOSS` umbrella product to import everything at once.

### Prerequisites

- macOS 14.0 or later.
- Xcode 16.0 or later.

### Xcode Steps

1. Open your Swift project in Xcode.
2. Navigate to `File` > `Add Package Dependencies...`.
3. Enter the package repository URL: `https://github.com/argmaxinc/argmax-oss-swift`.
4. Choose the version range or specific version.
5. When prompted to choose library products, select **ArgmaxOSS** (all kits), or individual kits: **WhisperKit**, **TTSKit**, **SpeakerKit**.

### Package.swift

Add the package dependency:

```swift
dependencies: [
    .package(url: "https://github.com/argmaxinc/argmax-oss-swift.git", from: "0.9.0"),
],
```

Then add the products you need as target dependencies:

```swift
.target(
    name: "YourApp",
    dependencies: [
        // Import everything at once:
        .product(name: "ArgmaxOSS", package: "argmax-oss-swift"),

        // Or pick individual kits:
        // .product(name: "WhisperKit", package: "argmax-oss-swift"),   // speech-to-text
        // .product(name: "TTSKit", package: "argmax-oss-swift"),       // text-to-speech
        // .product(name: "SpeakerKit", package: "argmax-oss-swift"),   // speaker diarization
    ]
),
```

### Homebrew

You can install the command line app using [Homebrew](https://brew.sh) by running the following command:

```bash
brew install whisperkit-cli
```  

## WhisperKit

To get started with WhisperKit, you need to initialize it in your project.

### Quick Example

This example demonstrates how to transcribe a local audio file:

```swift
import WhisperKit

// Initialize WhisperKit with default settings
Task {
    let pipe = try? await WhisperKit()
    let results = try? await pipe?.transcribe(audioPath: "path/to/your/audio.{wav,mp3,m4a,flac}")
    let transcription = results?.map(\.text).joined(separator: " ")
    print(transcription ?? "")
}
```

### Memory-Efficient Loading for Large Files

By default WhisperKit loads the whole audio file into memory before transcribing. For long recordings, `.incremental` streams it from disk in bounded-memory chunks instead:

```swift
import WhisperKit

let pipe = try await WhisperKit()

let options = AudioInputOptions(audioLoadingMode: .incremental)
let results = try await pipe.transcribe(
    audioPath: "path/to/large-audio.wav",
    audioInputOptions: options
)
print(results.map(\.text).joined(separator: " "))
```

It splits the audio at silence (VAD) boundaries, so the result matches a full-file transcription run with `chunkingStrategy: .vad` — only peak memory differs. Tune the chunking with `.incremental(chunkDuration:chunkBufferSize:)`, or pick channels with `AudioInputOptions(channelMode:)`.

From the CLI, add `--incremental-loading` (optionally `--incremental-chunk-duration` / `--incremental-chunk-buffer-size`):

```bash
swift run argmax-cli transcribe --model large-v3-v20240930_626MB --audio-path "path/to/large-audio.wav" --incremental-loading
```

### Model Selection

> [!NOTE]
> Argmax recommends `large-v3-v20240930_626MB` for maximum multilingual accuracy and `tiny` for the fastest debugging workflow.

| Whisper Version                  | WhisperKit Variant                                                                                                 | Description                                                                      |
|----------------------------------|------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------|
| Large v3 Turbo (compressed)      | [large-v3-v20240930_626MB](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-large-v3-v20240930_626MB) | Recommended across iOS and macOS for maximum accuracy                            |
| Large v3 Turbo                   | [large-v3-v20240930_turbo](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-large-v3-v20240930_turbo) | Recommended on macOS for maximum speed and accuracy                             |
| Base (multilingual)              | [base](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-base)                         |                                                                                  |
| Base (English-only)              | [base.en](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-base.en)                   |                                                                                  |
| Small (Multilingual)             | [small](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-small)                       |                                                                                  |
| Small (English-only)             | [small.en](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-small.en)                 |                                                                                  |
| Tiny (Multilingual)              | [tiny](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-tiny)                         |                                                                                  |
| Tiny (English-only)              | [tiny.en](https://huggingface.co/argmaxinc/whisperkit-coreml/tree/main/openai_whisper-tiny.en)                   | Smallest size, lowest accuracy. Only recommended for development & debugging.     |



WhisperKit automatically downloads the recommended model for the device if not specified. You can also select a specific model by passing in the model name:


```swift
let pipe = try? await WhisperKit(WhisperKitConfig(model: "large-v3-v20240930_626MB"))
```

This method also supports glob search, so you can use wildcards to select a model:

```swift
let pipe = try? await WhisperKit(WhisperKitConfig(model: "large-v3-v20240930_626MB"))
```

Note that the model search must return a single model from the source repo, otherwise an error will be thrown.

For a list of available models, see our [HuggingFace repo](https://huggingface.co/argmaxinc/whisperkit-coreml).

### Generating Models

WhisperKit also comes with the supporting repo [`whisperkittools`](https://github.com/argmaxinc/whisperkittools) which lets you create and deploy your own fine tuned versions of Whisper in CoreML format to HuggingFace. Once generated, they can be loaded by simply changing the repo name to the one used to upload the model:

```swift
let config = WhisperKitConfig(model: "large-v3-v20240930_626MB", modelRepo: "username/your-model-repo")
let pipe = try? await WhisperKit(config)
```

### Swift CLI

The Swift CLI allows for quick testing and debugging outside of an Xcode project. To install it, run the following:

```bash
git clone https://github.com/argmaxinc/argmax-oss-swift.git
cd argmax-oss-swift
```

Then, setup the environment and download your desired model.

```bash
make setup
make download-model MODEL=large-v3-v20240930_626MB
```

**Note**:

1. This will download only the model specified by `MODEL` (see what's available in our [HuggingFace repo](https://huggingface.co/argmaxinc/whisperkit-coreml), where we use the prefix `openai_whisper-{MODEL}`)
2. Before running `download-model`, make sure [git-lfs](https://git-lfs.com) is installed

If you would like download all available models to your local folder, use this command instead:

```bash
make download-models
```

You can then run them via the CLI with:

```bash
swift run argmax-cli transcribe --model-path "Models/whisperkit-coreml/openai_whisper-large-v3-v20240930_626MB" --audio-path "path/to/your/audio.{wav,mp3,m4a,flac}"
```

Which should print a transcription of the audio file. If you would like to stream the audio directly from a microphone, use:

```bash
swift run argmax-cli transcribe --model-path "Models/whisperkit-coreml/openai_whisper-large-v3-v20240930_626MB" --stream
```

### Local Server

The Argmax CLI includes a local server that implements the OpenAI Audio API, allowing you to use existing OpenAI SDK clients or generate new ones. The server supports transcription and translation with **output streaming** capabilities (real-time transcription results as they're generated).

> [!NOTE]
> [Argmax Pro Local Server](https://www.argmaxinc.com/blog/argmax-local-server) provides a real-time streaming transcription with a WebSocket local server that is API-compatible with cloud-based providers such as Deepgram.


#### Building the Server

```bash
# Build with server support
make build-local-server

# Or manually with the build flag
BUILD_ALL=1 swift build --product argmax-cli
```

#### Starting the Server

```bash
# Start server with default settings
BUILD_ALL=1 swift run argmax-cli serve

# Custom host and port
BUILD_ALL=1 swift run argmax-cli serve --host 0.0.0.0 --port 8080

# With specific model and verbose logging
BUILD_ALL=1 swift run argmax-cli serve --model tiny --verbose

# See all configurable parameters
BUILD_ALL=1 swift run argmax-cli serve --help
```

#### API Endpoints

- **POST** `/v1/audio/transcriptions` - Transcribe audio to text
- **POST** `/v1/audio/translations` - Translate audio to English

#### Supported Parameters

| Parameter | Description | Default |
|-----------|-------------|---------|
| `file` | Audio file (wav, mp3, m4a, flac) | Required |
| `model` | Model identifier | Server default |
| `language` | Source language code | Auto-detect |
| `prompt` | Text to guide transcription | None |
| `response_format` | Output format (json, verbose_json) | verbose_json |
| `temperature` | Sampling temperature (0.0-1.0) | 0.0 |
| `timestamp_granularities[]` | Timing detail (word, segment) | segment |
| `stream` | Enable streaming | false |

#### Client Examples

**Python Client (OpenAI SDK)**
```bash
cd Examples/ServeCLIClient/Python
uv sync
python whisperkit_client.py transcribe --file audio.wav --language en
python whisperkit_client.py translate --file audio.wav
```

Quick Python example:
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:50060/v1")
result = client.audio.transcriptions.create(
    file=open("audio.wav", "rb"),
    model="tiny"  # Model parameter is required
)
print(result.text)
```

**Swift Client (Generated from OpenAPI Spec, see ServeCLIClient/Swift/updateClient.sh)**
```bash
cd Examples/ServeCLIClient/Swift
swift run whisperkit-client transcribe audio.wav --language en
swift run whisperkit-client translate audio.wav
```

**CurlClient (Shell Scripts)**
```bash
cd Examples/ServeCLIClient/Curl
chmod +x *.sh
./transcribe.sh audio.wav --language en
./translate.sh audio.wav --language es
./test.sh  # Run comprehensive test suite
```

#### Generating the API Specification

The server's OpenAPI specification and code are generated from the official OpenAI API:

```bash
# Generate latest spec and server code
make generate-server
```

#### Client Generation

You can generate clients for any language using the OpenAPI specification, for example:

```bash
# Generate Python client
swift run swift-openapi-generator generate scripts/specs/localserver_openapi.yaml \
  --output-directory python-client \
  --mode client \
  --mode types

# Generate TypeScript client
npx @openapitools/openapi-generator-cli generate \
  -i scripts/specs/localserver_openapi.yaml \
  -g typescript-fetch \
  -o typescript-client
```

#### API Limitations

Compared to the official OpenAI API, the local server has these limitations:

- **Response formats**: Only `json` and `verbose_json` supported (no plain text, SRT, VTT formats)
- **Model selection**: Client must launch server with desired model via `--model` flag

#### Fully Supported Features

The local server fully supports these OpenAI API features:

- **Include parameters**: `logprobs` parameter for detailed token-level log probabilities
- **Streaming responses**: Server-Sent Events (SSE) for real-time transcription
- **Timestamp granularities**: Both `word` and `segment` level timing
- **Language detection**: Automatic language detection or manual specification
- **Temperature control**: Sampling temperature for transcription randomness
- **Prompt text**: Text guidance for transcription style and context

## TTSKit

TTSKit is an on-device text-to-speech framework built on Core ML. It runs [Qwen3-TTS](https://github.com/QwenLM/Qwen3-TTS) models entirely on Apple silicon with real-time streaming playback, no server required.

- macOS 15.0 or later.
- iOS 18.0 or later.

### Quick Example

This example demonstrates how to generate speech from text:

```swift
import TTSKit

Task {
    let tts = try await TTSKit()
    let result = try await tts.generate(text: "Hello from TTSKit!")
    print("Generated \(result.audioDuration)s of audio at \(result.sampleRate)Hz")
}
```

`TTSKit()` automatically downloads the default 0.6B model on first run. The tokenizer and CoreML models are loaded lazily on the first `generate()` call.

### Model Selection

TTSKit ships two model sizes. You can select the model by passing a variant to `TTSKitConfig`:

```swift
// Fast, runs on all platforms (~1 GB download)
let tts = try await TTSKit(TTSKitConfig(model: .qwen3TTS_0_6b))

// Higher quality, macOS only (~2.2 GB download, supports style instructions)
let tts = try await TTSKit(TTSKitConfig(model: .qwen3TTS_1_7b))
```

Models are hosted on [HuggingFace](https://huggingface.co/argmaxinc/ttskit-coreml) and cached locally after the first download.

#### Custom Voices

You can choose from 9 built-in voices and 10 languages:

```swift
let result = try await tts.generate(
    text: "こんにちは世界",
    speaker: .onoAnna,
    language: .japanese
)
```

**Voices:** `.ryan`, `.aiden`, `.onoAnna`, `.sohee`, `.eric`, `.dylan`, `.serena`, `.vivian`, `.uncleFu`

**Languages:** `.english`, `.chinese`, `.japanese`, `.korean`, `.german`, `.french`, `.russian`, `.portuguese`, `.spanish`, `.italian`

#### Real-Time Streaming Playback

`play` streams audio to the device speakers frame-by-frame as it is generated:

```swift
try await tts.play(text: "This starts playing before generation finishes.")
```

You can control how much audio is buffered before playback begins. The default `.auto` strategy measures the first generation step and pre-buffers just enough to avoid underruns:

```swift
try await tts.play(
    text: "Long passage...",
    playbackStrategy: .auto
)
```

Other strategies include `.stream` (immediate, no buffer), `.buffered(seconds:)` (fixed pre-buffer), and `.generateFirst` (generate all audio first, then play).

### Speech Decoder Mode

TTSKit's default speech decoder bundles two functions, selectable via `TTSKitConfig.speechDecoderMode`:

| Mode | RVQ frames / call | Audio / call | Use case |
|------|-------------------|--------------|----------|
| `.latencyOptimized` (default) | 1 | ~80 ms | Lowest time-to-first-audio for streaming. |
| `.throughputOptimized` | 4 | ~320 ms | Amortizes decoder overhead for higher throughput, at the cost of a ~4× larger first-buffer latency. |

```swift
// Default: latency-optimized (lowest time-to-first-audio)
let tts = try await TTSKit()

// Opt into throughput-optimized generation
let config = TTSKitConfig(speechDecoderMode: .throughputOptimized)
let throughputTTS = try await TTSKit(config)
```

The mode is read once when models are loaded; set it before constructing `TTSKit` (or reload the model to switch at runtime).

### Generation Options

You can customize sampling, chunking, and concurrency via `GenerationOptions`:

```swift
// Defaults recommended by Qwen
var options = GenerationOptions()
options.temperature = 0.9
options.topK = 50
options.repetitionPenalty = 1.05
options.maxNewTokens = 245

// Long text is automatically split at sentence boundaries
options.chunkingStrategy = .sentence
options.concurrentWorkerCount = nil  // nil = all chunks run concurrently with a good default for the device

let result = try await tts.generate(text: longArticle, options: options)
```

#### Style Instructions (1.7B only)

The 1.7B model accepts a natural-language style instruction that controls prosody:

```swift
var options = GenerationOptions()
options.instruction = "Speak slowly and warmly, like a storyteller."

let result = try await tts.generate(
    text: "Once upon a time...",
    speaker: .ryan,
    options: options
)
```

### Saving Audio

Generated audio can be saved to WAV or M4A:

```swift
let result = try await tts.generate(text: "Save me!")
let outputDir = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]

// Save as .wav or .m4a (AAC)
try await AudioOutput.saveAudio(result.audio, toFolder: outputDir, filename: "output", format: .m4a)
```

### Progress Callbacks

You can receive per-step audio during generation. Return `false` from the callback to cancel early:

```swift
let result = try await tts.generate(text: "Hello!") { progress in
    print("Audio chunk: \(progress.audio.count) samples")
    if let stepTime = progress.stepTime {
        print("First step took \(stepTime)s")
    }
    return true  // return false to cancel
}
```

### Swift CLI

The TTS command is available through the `argmax-cli` tool. You can generate speech and optionally play it back in real time:

```bash
swift run argmax-cli tts --text "Hello from the command line" --play
swift run argmax-cli tts --text "Save to file" --output-path output.wav
swift run argmax-cli tts --text "日本語テスト" --speaker ono-anna --language japanese
swift run argmax-cli tts --text-file article.txt --model 1.7b --instruction "Read cheerfully"
swift run argmax-cli tts --help
```

### Demo App

The [TTSKitExample](Examples/TTS/TTSKitExample/) example app showcases real-time streaming, model management, waveform visualization, and generation history on macOS and iOS. See the [TTSKitExample README](Examples/TTS/TTSKitExample/README.md) for build instructions.

## SpeakerKit

SpeakerKit is an on-device speaker diarization framework built on Core ML. It runs [Pyannote v4 (community-1)](https://huggingface.co/argmaxinc/speakerkit-coreml) on Apple silicon to label speakers in audio. Read the [blog post](https://www.argmaxinc.com/blog/speakerkit) for architecture details and benchmarks.

- macOS 13.0 or later.
- iOS 16.0 or later.

### Quick Example

This example demonstrates how to diarize an audio file:

```swift
import SpeakerKit

Task {
    let speakerKit = try await SpeakerKit()

    let audioArray = try AudioProcessor.loadAudioAsFloatArray(fromPath: "audio.wav")
    let result = try await speakerKit.diarize(audioArray: audioArray)

    print("Detected \(result.speakerCount) speakers")
    for segment in result.segments {
        print(segment)
    }
}
```

`SpeakerKit()` uses `PyannoteConfig()` defaults, automatically downloading models from [HuggingFace](https://huggingface.co/argmaxinc/speakerkit-coreml) on first run. The segmenter and embedder CoreML models are loaded lazily (unless `load` is set on config) on the first `diarize()` call.

### Diarization Options

You can control speaker detection via `PyannoteDiarizationOptions`:

```swift
let audioArray = try AudioProcessor.loadAudioAsFloatArray(fromPath: "audio.wav")
let options = PyannoteDiarizationOptions(
    numberOfSpeakers: 2,               // nil = automatic detection
    clusterDistanceThreshold: 0.6,     // clustering threshold
    useExclusiveReconciliation: false   // exclusive speaker assignment per frame
)
let result = try await speakerKit.diarize(audioArray: audioArray, options: options)
```

For local models, skip the download step:

```swift
let config = PyannoteConfig(modelFolder: "/path/to/models")
let speakerKit = try await SpeakerKit(config)
```

### Combining with Transcription

SpeakerKit can merge diarization results with WhisperKit transcriptions to produce speaker-attributed segments:

```swift
import WhisperKit
import SpeakerKit

let whisperKit = try await WhisperKit()
let speakerKit = try await SpeakerKit()

let audioArray = try AudioProcessor.loadAudioAsFloatArray(fromPath: "audio.wav")
let transcription = try await whisperKit.transcribe(audioArray: audioArray)
let diarization = try await speakerKit.diarize(audioArray: audioArray)

let speakerSegments = diarization.addSpeakerInfo(to: transcription)

for group in speakerSegments {
    for segment in group {
        print("\(segment.speaker): \(segment.text)")
    }
}
```

Two strategies are available for matching speakers to transcription:
- `.subsegment` (default) -- splits segments at word gaps, then assigns speakers
- `.segment` -- assigns a speaker to each transcription segment as a whole

### RTTM Output

Generate RTTM output:

```swift
let speakerKit = try await SpeakerKit()

let audioArray = try AudioProcessor.loadAudioAsFloatArray(fromPath: "meeting.wav")
let diarization = try await speakerKit.diarize(audioArray: audioArray)

let rttmLines = SpeakerKit.generateRTTM(from: diarization, fileName: "meeting")
for line in rttmLines {
    print(line)
}
```

### Swift CLI

The diarization commands are available through the `argmax-cli` tool:

```bash
# Standalone diarization
swift run argmax-cli diarize --audio-path audio.wav --verbose

# Save RTTM output
swift run argmax-cli diarize --audio-path audio.wav --rttm-path output.rttm

# Specify number of speakers
swift run argmax-cli diarize --audio-path audio.wav --num-speakers 3

# Transcription with diarization
swift run argmax-cli transcribe --audio-path audio.wav --diarization

# See all options
swift run argmax-cli diarize --help
```

## Contributing & Roadmap

Our goal is to make this SDK better and better over time and we'd love your help! Just search the code for "TODO" for a variety of features that are yet to be built. Please refer to our [contribution guidelines](CONTRIBUTING.md) for submitting issues, pull requests, and coding standards, where we also have a public roadmap of features we are looking forward to building in the future.

**External dependencies:** `Sources/ArgmaxCore/External/` contains a copy of [swift-transformers](https://github.com/huggingface/swift-transformers) (Hub and Tokenizers modules, v1.1.6) with Jinja-dependent code removed. When updating to a newer version, copy the fresh sources over that directory and re-apply the patches marked with `// Argmax-modification:` (`grep -r "Argmax-modification:" Sources/ArgmaxCore/External/`). The matching upstream tests are vendored under `Tests/ArgmaxCoreTests/External/` using the same convention.

## License

Argmax OSS is released under the MIT License. See [LICENSE](LICENSE) for more details.

This project incorporates third-party software under their own license terms. See [NOTICES](NOTICES) for attributions.

## Citation

If you use this SDK for something cool or just find it useful, please drop us a note at [info@argmaxinc.com](mailto:info@argmaxinc.com)!

If you use WhisperKit, SpeakerKit or TTSKit for academic work, please cite the project using the following BibTeX:

```bibtex
@misc{whisperkit-argmax,
   title = {Argmax OSS: On-device Speech AI with WhisperKit, SpeakerKit and TTSKit},
   author = {Argmax, Inc.},
   year = {2024},
   URL = {https://github.com/argmaxinc/argmax-oss-swift}
}
```

---

## File: Examples/TTS/TTSKitExample/README.md

# TTSKit Example App

TTSKitExample is the demo app for TTSKit, an on-device text-to-speech framework powered by Core ML. It runs on macOS and iOS with no server required.

## Requirements

- macOS 15.2 or later.
- iOS 18.2 or later.
- Xcode 16.3 or later.


## Building & Running

1. Open `Examples/TTS/TTSKitExample/TTSKitExample.xcodeproj` in Xcode.
2. Select the **TTSKitExample** scheme and your target device.
3. **Set up signing:** In the project navigator, select the *TTSKitExample* target. Go to the **Signing & Capabilities** tab, then:
   - Choose your Apple Developer Team from the **Team** dropdown.
   - Make sure a unique bundle identifier is set.
   - Xcode will automatically handle provisioning; resolve any signing issues if prompted.
4. Build and run (`Cmd+R`). Dependencies resolve automatically via SPM.


On first launch, the app prompts you to download a model. Downloads are cached in your Documents directory and reused across launches.

### Private HuggingFace Repos

If you use a private model fork, set a token in `ViewModel.swift`:

```swift
let config = TTSKitConfig(model: selectedPreset, modelToken: "hf_...")
```

## Features

- **Model management.** Download, prewarm, load, and unload models from the sidebar.
- **Two model sizes.** 0.6B (fast, all platforms) and 1.7B (higher quality, style instructions, macOS only).
- **Real-time streaming.** Audio starts playing before generation completes.
- **Adaptive buffering.** The `.auto` strategy measures first-step speed and pre-buffers accordingly.
- **9 voices.** Ryan, Aiden, Ono Anna, Sohee, Eric, Dylan, Serena, Vivian, Uncle Fu.
- **10 languages.** English, Chinese, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian.
- **Style instructions.** Natural-language prosody control (1.7B only).
- **Generation history.** All outputs saved as M4A with embedded metadata, reconstructed at launch.
- **Favorites.** Star generations to pin them at the top of the sidebar.
- **Waveform.** Live visualization that updates during streaming.
- **Settings.** Temperature, top-K, repetition penalty, max tokens, chunking, compute units.
- **Copy audio.** One-tap copy of the M4A to clipboard.

## Project Structure

```
TTSKitExample/
├── TTSKitExampleApp.swift        entry point
├── ContentView.swift             root NavigationSplitView
├── SidebarView.swift             model management + history
├── DetailView.swift              input form, waveform, playback
├── ModelManagementView.swift     download / load / unload
├── GenerationSettingsView.swift  advanced options sheet
├── ViewModel.swift               @Observable view model
├── AudioMetadata.swift           Codable metadata embedded in .m4a files
├── WaveformView.swift            live waveform bar chart
└── ComputeUnitsView.swift        per-component compute unit picker
```

Persistence is file-based. Each generation is a self-contained `.m4a` file with all metadata (text, speaker, timings) embedded as JSON in the iTunes comment atom. No database is needed. The history is reconstructed by scanning the Documents directory at launch.

---

## File: Examples/ServeCLIClient/Swift/README.md

# WhisperKit Swift Client

A simple Swift client for the WhisperKit local server.

## Quick Start

1. **Start the WhisperKit server** (in another terminal):
   ```bash
   whisperkit-cli serve
   ```

2. **Build the client**:
   ```bash
   swift build
   ```

3. **Run commands**:
   ```bash
   # Transcribe an audio file
   swift run whisperkit-client transcribe audio.wav
   
   # Translate an audio file to English
   swift run whisperkit-client translate audio.wav
   
   # Test with sample files
   swift run whisperkit-client test
   ```

## Available Commands

- `transcribe <audio-file>` - Transcribe audio to text
- `translate <audio-file>` - Translate audio to English
- `test` - Test transcription and translation on sample files

## Options

- `--language, -l` - Source language for transcription (default: auto-detect)
- `--model, -m` - Model to use (default: tiny)
- `--response-format` - Response format: json, verbose_json (default: verbose_json)
- `--timestamp-granularities` - Comma-separated: word,segment (default: segment)
- `--stream` - Enable streaming output
- `--server-url, -s` - Server URL (default: http://localhost:50060/v1)

## Examples

```bash
# Transcribe in Spanish
swift run whisperkit-client transcribe -l es audio.wav

# Transcribe with word-level timestamps
swift run whisperkit-client transcribe --timestamp-granularities "word,segment" audio.wav

# Translate from Spanish to English
swift run whisperkit-client translate -l es audio.wav

# Use custom server
swift run whisperkit-client transcribe -s http://192.168.1.100:50060 audio.wav

# Stream transcription
swift run whisperkit-client transcribe --stream audio.wav
```

## Project Structure

```
Sources/
├── CLI.swift           # All CLI commands and client logic
└── Generated/          # Auto-generated OpenAPI client code
    ├── Client.swift
    └── Types.swift
```

## Current Limitations

- **Response Format**: The `--response-format` parameter is not fully working due to OpenAPI schema discrimination issues. The client always receives basic JSON responses instead of verbose JSON with segments and word timestamps.
- **Word Timestamps**: Word-level timestamps are not displayed due to the response format issue above.
- **Basic Functionality**: Basic transcription and translation work correctly.

> **Note**: This is a known issue with the Swift OpenAPI generator's handling of `oneOf` schemas with discriminators. The server correctly sends verbose JSON responses, but the Swift client cannot properly parse them. Consider using the Python client or CurlClient for full functionality.

## Updating Generated Code

When the server spec changes, regenerate the client code:

```bash
./updateClient.sh
```

This will update the files in `Sources/Generated/` from `scripts/specs/localserver_openapi.yaml`.

---

## File: Examples/ServeCLIClient/Python/README.md

# WhisperKit Python Client

A simple Python client for the WhisperKit local server using OpenAI's SDK.

## Quick Start

1. **Start the WhisperKit server** (in another terminal):
   ```bash
   whisperkit-cli serve
   ```

2. **Install dependencies**:
   ```bash
   uv sync
   ```

3. **Run commands**:
   ```bash
   # Transcribe an audio file
   python whisperkit_client.py transcribe audio.wav
   
   # Translate an audio file to English
   python whisperkit_client.py translate audio.wav
   
   # Test with sample files
   python whisperkit_client.py test
   ```

## Available Commands

- `transcribe <audio-file>` - Transcribe audio to text
- `translate <audio-file>` - Translate audio to English
- `test` - Test transcription and translation on sample files

## Options

- `--server, -s` - Server URL (default: http://localhost:50060)
- `--model, -m` - Model to use (default: tiny)
- `--language, -l` - Source language for transcription (default: auto-detect)
- `--response-format` - Response format: json, verbose_json (default: verbose_json)
- `--timestamp-granularities` - Comma-separated: word,segment (default: segment)
- `--stream` - Enable streaming output
- `--debug` - Show raw JSON response for debugging


## Examples

```bash
# Transcribe in Spanish
python whisperkit_client.py transcribe -l es audio.wav

# Translate to English (auto-detects source language)
python whisperkit_client.py translate audio.wav

# Use custom server and model
python whisperkit_client.py -s http://192.168.1.100:50060 -m large transcribe audio.wav

# Transcribe with word-level timestamps
python whisperkit_client.py transcribe --timestamp-granularities "word,segment" audio.wav

# Stream transcription
python whisperkit_client.py transcribe --stream audio.wav

# Debug mode to see raw JSON
python whisperkit_client.py transcribe --debug audio.wav

# Test with sample files
python whisperkit_client.py test
```

## Project Structure

```
Examples/ServeCLIClient/Python/
├── whisperkit_client.py    # Main CLI script with all functionality
├── test_transcribe.py      # Test script for transcription
├── test_translate.py       # Test script for translation
├── requirements.txt         # Python dependencies
├── uv.lock                 # Locked dependency versions
└── README.md               # This file
```

## Dependencies

- `openai` - OpenAI Python SDK for API communication
- `uv` - Fast Python package manager

## Testing

The client automatically finds test audio files from `Tests/WhisperKitTests/Resources/` in the main project directory.

```bash
# Run tests on sample files
python whisperkit_client.py test

# Or run individual test scripts
python test_transcribe.py
python test_translate.py
```

## Alternative Clients

For lightweight testing without Python dependencies, see the [CurlClient](../Curl/README.md) which provides shell script implementations using curl.

---

## File: Examples/ServeCLIClient/Curl/README.md

# WhisperKit CurlClient

A simple, lightweight client for the WhisperKit Local Server using shell scripts and curl.

## Quick Start

1. **Make scripts executable:**
   ```bash
   chmod +x *.sh
   ```

2. **Start the WhisperKit server:**
   ```bash
   whisperkit-cli serve --model tiny
   ```

3. **Use the scripts:**
   ```bash
   # Transcribe audio
   ./transcribe.sh audio.wav
   
   # Translate audio to English
   ./translate.sh audio.wav --language es
   
   # Run test suite
   ./test.sh
   ```

## Scripts

### `transcribe.sh`
Transcribes audio files to text.

**Basic usage:**
```bash
./transcribe.sh audio.wav
./transcribe.sh audio.wav --language en --timestamp-granularities word,segment
./transcribe.sh audio.wav --stream true --logprobs
```

### `translate.sh`
Translates audio files to English.

**Basic usage:**
```bash
./translate.sh audio.wav
./translate.sh audio.wav --language es
./translate.sh audio.wav --stream true --logprobs
```

### `test.sh`
Runs comprehensive tests on sample files.

## Options

- `-h, --help` - Show help
- `-s, --server ` - Server URL (default: http://localhost:50060/v1)
- `-l, --language <lang>` - Source language (e.g., en, es, ja)
- `-f, --response-format <format>` - Response format: json, verbose_json
- `--timestamp-granularities <types>` - Timestamp granularities: word,segment
- `--stream ` - Enable streaming (default: false)
- `--logprobs` - Include logprobs in response (default: false)
- `--temperature <value>` - Sampling temperature 0.0-1.0 (default: 0.0)
- `--verbose` - Show verbose curl output

## Prerequisites

- `curl` (usually pre-installed)
- `bash` shell
- WhisperKit Local Server running

---

## File: fastlane/README.md

fastlane documentation
----

# Installation

Make sure you have the latest version of the Xcode command line tools installed:

```sh
xcode-select --install
```

For _fastlane_ installation instructions, see [Installing _fastlane_](https://docs.fastlane.tools/#installing-fastlane)

# Available Actions

## iOS

### ios list_devices

```sh
[bundle exec] fastlane ios list_devices
```

List all connected devices

### ios benchmark

```sh
[bundle exec] fastlane ios benchmark
```

Benchmark devices with options

### ios extract_results

```sh
[bundle exec] fastlane ios extract_results
```

Extract benchmark results

### ios upload_results

```sh
[bundle exec] fastlane ios upload_results
```

Upload benchmark results

----

This README.md is auto-generated and will be re-generated every time [_fastlane_](https://fastlane.tools) is run.

More information about _fastlane_ can be found on [fastlane.tools](https://fastlane.tools).

The documentation of _fastlane_ can be found on [docs.fastlane.tools](https://docs.fastlane.tools).