seamless_communication Documentation

Technical Documentation: facebookresearch/seamless_communication

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Seamless Intro

Seamless is a family of AI models that enable more natural and authentic communication across languages. SeamlessM4T is a massive multilingual multimodal machine translation model supporting around 100 languages. SeamlessM4T serves as foundation for SeamlessExpressive, a model that preserves elements of prosody and voice style across languages and SeamlessStreaming, a model supporting simultaneous translation and streaming ASR for around 100 languages. SeamlessExpressive and SeamlessStreaming are combined into Seamless, a unified model featuring multilinguality, real-time and expressive translations.

Links Copy Chapter

Demos Copy Section

Papers Copy Section

Seamless

EMMA

SONAR

Blog Copy Section

AI at Meta Blog

Tutorial Copy Chapter

An exhaustive tutorial given at the NeurIPS 2023 - Seamless EXPO, which is a one-stop shop to learn how to use the entire suite of Seamless models. Please feel free to play with the notebook.

SeamlessM4T Copy Chapter

SeamlessM4T is our foundational all-in-one Massively Multilingual and Multimodal Machine Translation model delivering high-quality translation for speech and text in nearly 100 languages.

SeamlessM4T models support the tasks of:

• Speech-to-speech translation (S2ST)
• Speech-to-text translation (S2TT)
• Text-to-speech translation (T2ST)
• Text-to-text translation (T2TT)
• Automatic speech recognition (ASR)

:star2: We are releasing SeamlessM4T v2, an updated version with our novel UnitY2 architecture. This new model improves over SeamlessM4T v1 in quality as well as inference latency in speech generation tasks.

To learn more about the collection of SeamlessM4T models, the approach used in each, their language coverage and their performance, visit the SeamlessM4T README or 🤗 Model Card.

[!NOTE]
Seamless M4T is also available in the 🤗 Transformers library. Visit this section for more details.

SeamlessExpressive Copy Chapter

SeamlessExpressive is a speech-to-speech translation model that captures certain underexplored aspects of prosody such as speech rate and pauses, while preserving the style of one's voice and high content translation quality.

To learn more about SeamlessExpressive models, visit the SeamlessExpressive README or 🤗 Model Card

SeamlessStreaming Copy Chapter

SeamlessStreaming is a streaming translation model. The model supports speech as input modality and speech/text as output modalities.

The SeamlessStreaming model supports the following tasks:

• Speech-to-speech translation (S2ST)
• Speech-to-text translation (S2TT)
• Automatic speech recognition (ASR)

To learn more about SeamlessStreaming models, visit the SeamlessStreaming README or 🤗 Model Card

Seamless Copy Chapter

The Seamless model is the unified model for expressive streaming speech-to-speech translations.

What's new Copy Chapter

• [12/18/2023] We are open-sourcing our Conformer-based W2v-BERT 2.0 speech encoder as described in Section 3.2.1 of the paper, which is at the core of our Seamless models.
• [12/14/2023] We are releasing the Seamless tutorial given at NeurIPS 2023.

Quick Start

Installation Copy Chapter

[!NOTE]
One of the prerequisites is fairseq2 which has pre-built packages available only
for Linux x86-64 and Apple-silicon Mac computers. In addition it has a dependency on libsndfile which
might not be installed on your machine. If you experience any installation issues, please refer to its
README for further instructions.

pip install .

[!NOTE]
Transcribing inference audio for computing metric uses Whisper, which is automatically installed. Whisper in turn requires the command-line tool ffmpeg to be installed on your system, which is available from most package managers.

Running inference Copy Chapter

SeamlessM4T Inference Copy Section

Here’s an example of using the CLI from the root directory to run inference.

S2ST task:

m4t_predict <path_to_input_audio> --task s2st --tgt_lang <tgt_lang> --output_path <path_to_save_audio>

T2TT task:

m4t_predict <input_text> --task t2tt --tgt_lang <tgt_lang> --src_lang <src_lang>

Please refer to the inference README for detailed instruction on how to run inference and the list of supported languages on the source, target sides for speech, text modalities.

For running S2TT/ASR natively (without Python) using GGML, please refer to the unity.cpp section.

SeamlessExpressive Inference Copy Section

[!NOTE]
Please check the section on how to download the model.

Here’s an example of using the CLI from the root directory to run inference.

expressivity_predict <path_to_input_audio> --tgt_lang <tgt_lang> --model_name seamless_expressivity --vocoder_name vocoder_pretssel --output_path <path_to_save_audio>

SeamlessStreaming and Seamless Inference Copy Section

Streaming Evaluation README has detailed instructions for running evaluations for the SeamlessStreaming and Seamless models. The CLI has an --no-scoring option that can be used to skip the scoring part and just run inference.

Please check the inference README for more details.

Running SeamlessStreaming Demo Copy Chapter

You can duplicate the SeamlessStreaming HF space to run the streaming demo.

You can also run the demo locally, by cloning the space from here. See the README of the SeamlessStreaming HF repo for more details on installation.

Running SeamlessM4T & SeamlessExpressive [Gradio](https://github.com/gradio-app/gradio) demos locally Copy Chapter

To launch the same demo Space we host on Hugging Face locally:

cd demo
pip install -r requirements.txt
python app.py

Resources and usage

Model Copy Chapter

SeamlessM4T models Copy Section

SeamlessExpressive models Copy Section

🤗 Model card

To access and download SeamlessExpressive, please request the model artifacts through this request form. Upon approval, you will then receive an email with download links to each model artifact.

Please note that SeamlessExpressive is made available under its own License and Acceptable Use Policy.

SeamlessStreaming models Copy Section

Seamless models Copy Section

Seamless model is simply the SeamlessStreaming model with the non-expressive vocoder_v2 swapped out with the expressive vocoder_pretssel.
Please check out above section on how to acquire vocoder_pretssel checkpoint.

W2v-BERT 2.0 speech encoder Copy Section

Here's how you should do a foward pass through the speech encoder:

import torch

from fairseq2.data.audio import AudioDecoder, WaveformToFbankConverter
from fairseq2.memory import MemoryBlock
from fairseq2.nn.padding import get_seqs_and_padding_mask
from fairseq2.data import Collater
from pathlib import Path
from seamless_communication.models.conformer_shaw import load_conformer_shaw_model


audio_wav_path, device, dtype = ...
audio_decoder = AudioDecoder(dtype=torch.float32, device=device)
fbank_converter = WaveformToFbankConverter(
    num_mel_bins=80,
    waveform_scale=2**15,
    channel_last=True,
    standardize=True,
    device=device,
    dtype=dtype,
)
collater = Collater(pad_value=1)

model = load_conformer_shaw_model("conformer_shaw", device=device, dtype=dtype)
model.eval()

with Path(audio_wav_path).open("rb") as fb:
    block = MemoryBlock(fb.read())

decoded_audio = audio_decoder(block)
src = collater(fbank_converter(decoded_audio))["fbank"]
seqs, padding_mask = get_seqs_and_padding_mask(src)

with torch.inference_mode():
  seqs, padding_mask = model.encoder_frontend(seqs, padding_mask)
  seqs, padding_mask = model.encoder(seqs, padding_mask)

Evaluation Copy Chapter

SeamlessM4T Evaluation Copy Section

To reproduce our results, or to evaluate using the same metrics over your own test sets, please check out the README here.

SeamlessExpressive Evaluation Copy Section

Below is the script for efficient batched evaluation.

export MODEL_DIR="/path/to/SeamlessExpressive/model"
export TEST_SET_TSV="input.tsv" # Your dataset in a TSV file, with headers "id", "audio"
export TGT_LANG="spa" # Target language to translate into, options including "fra", "deu", "eng" ("cmn" and "ita" are experimental)
export OUTPUT_DIR="tmp/" # Output directory for generated text/unit/waveform
export TGT_TEXT_COL="tgt_text" # The column in your ${TEST_SET_TSV} for reference target text to calcuate BLEU score. You can skip this argument.
export DFACTOR="1.0" # Duration factor for model inference to tune predicted duration (preddur=DFACTOR*preddur) per each position which affects output speech rate. Greater value means slower speech rate (default to 1.0). See expressive evaluation README for details on duration factor we used.
expressivity_evaluate ${TEST_SET_TSV} \
  --gated-model-dir ${MODEL_DIR} --task s2st --tgt_lang ${TGT_LANG} \
  --audio_root_dir "" --output_path ${OUTPUT_DIR} --ref_field ${TGT_TEXT_COL} \
  --model_name seamless_expressivity --vocoder_name vocoder_pretssel \
  --text_unk_blocking True --duration_factor ${DFACTOR}

Please check out this README section

SeamlessStreaming and Seamless Evaluation Copy Section

Streaming Evaluation README has detailed instructions for running evaluations on the SeamlessStreaming and Seamless models.

Unity.cpp Copy Chapter

To enable Seamless Communication Everywhere, we implemented unity.cpp so users could run SeamlessM4T models in GGML - a C tensor library allowing easier integration on verbose platforms.

To transcribe/translte a given audio,

./ggml/bin/unity --model seamlessM4T_medium.ggml input.wav

For details of build and more usage please check out unity.cpp

Expressive Datasets Copy Chapter

We created two expressive speech-to-speech translation datasets, mExpresso and mDRAL, between English and five other languages -- French, German, Italian, Mandarin and Spanish. We currently open source the speech-to-text of mExpresso for out-of-English directions, and we will open source the remaining part of the datasets soon. For details, please check out README

SeamlessAlignExpressive Copy Section

We’re introducing the first expressive speech alignment procedure. Starting with raw data, the expressive alignment procedure automatically discovers pairs of audio segments sharing not only the same meaning, but the same overall expressivity. To showcase this procedure, we are making metadata available to create a benchmarking dataset called SeamlessAlignExpressive, that can be used to validate the quality of our alignment method. SeamlessAlignExpressive is the first large-scale (11k+ hours) collection of multilingual audio alignments for expressive translation. More details can be found on the SeamlessAlignExpressive README.

Converting raw audio to units Copy Chapter

Please check out the README here. Note that SeamlessM4T v1 model uses reduced units and other models use non-reduced units.

Libraries

Seamless Communication depends on 4 libraries developed by Meta.

[fairseq2](https://github.com/facebookresearch/fairseq2) Copy Chapter

fairseq2 is our next-generation open-source library of sequence modeling components that provides researchers and developers with building blocks for machine translation, language modeling, and other sequence generation tasks. All SeamlessM4T models in this repository are powered by fairseq2.

[SONAR and BLASER 2.0](https://github.com/facebookresearch/SONAR) Copy Chapter

SONAR, Sentence-level multimOdal and laNguage-Agnostic Representations is a new multilingual and -modal sentence embedding space which outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks. SONAR provides text and speech encoders for many languages. SeamlessAlign was mined based on SONAR embeddings.

BLASER 2.0 is our latest model-based evaluation metric for multimodal translation. It is an extension of BLASER, supporting both speech and text. It operates directly on the source signal, and as such, does not require any intermediate ASR system like ASR-BLEU. As in the first version, BLASER 2.0 leverages the similarity between input and output sentence embeddings. SONAR is the underlying embedding space for BLASER 2.0. Scripts to run evaluation with BLASER 2.0 can be found in the SONAR repo.

[stopes](https://github.com/facebookresearch/stopes) Copy Chapter

As part of the seamless communication project, we've extended the stopes library. Version 1 provided a text-to-text mining tool to build training dataset for translation models. Version 2 has been extended thanks to SONAR, to support tasks around training large speech translation models. In particular, we provide tools to read/write the fairseq audiozip datasets and a new mining pipeline that can do speech-to-speech, text-to-speech, speech-to-text and text-to-text mining, all based on the new SONAR embedding space.

[SimulEval](https://github.com/facebookresearch/SimulEval) Copy Chapter

SimulEval is a library used for evaluating simulaneous translation models. SimulEval also provides a backend for generation using partial/incremental inputs with flexible/extensible states, which is used to implement streaming inference. Users define agents which implement SimulEval's interface, which can be connected together in a pipeline. You can find agents implemented for SeamlessStreaming here.

[Legacy] SeamlessM4T v1 instructions Copy Chapter

Finetuning SeamlessM4T v1 models

Please check out the README here.

On-device models

Apart from Seamless-M4T large (2.3B) and medium (1.2B) models, we are also releasing a small model (281M) targeted for on-device inference. To learn more about the usage and model details check out the README here.

SeamlessAlign mined dataset

We open-source the metadata to SeamlessAlign, the largest open dataset for multimodal translation, totaling 270k+ hours of aligned Speech and Text data. The dataset can be rebuilt by the community based on the SeamlessAlign readme.

Citation

If you use Seamless in your work or any models/datasets/artifacts published in Seamless, please cite :

@inproceedings{seamless2023,
   title="Seamless: Multilingual Expressive and Streaming Speech Translation",
   author="{Seamless Communication}, Lo{\"i}c Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rakotoarison, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R. Costa-juss{\`a}, Maha Elbayad, Hongyu Gong, Francisco Guzm{\'a}n, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alex Mourachko, Benjamin Peloquin, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, Mary Williamson",
  journal={ArXiv},
  year={2023}
}

License

We have three license categories.

The following non-generative components are MIT licensed as found in MIT_LICENSE:

• W2v-BERT 2.0 speech encoder
• Code
• Text only part of the mExpresso dataset found in the SeamlessExpressive README.
• UnitY2 forced alignment extractor found in the UnitY2 Aligner README.
• Speech toxicity tool with the etox dataset found in the ETOX README.
• MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector Mutox README

The following models are CC-BY-NC 4.0 licensed as found in the LICENSE:

• SeamlessM4T models (v1 and v2).
• SeamlessStreaming models.

The following models are Seamless licensed as found in SEAMLESS_LICENSE:

• Seamless models.
• SeamlessExpressive models.

Overview (`ggml/README.md`) Copy Section

unity.cpp

Introduction Copy Chapter

GGML is an open source library in C to enable large model inference on various hardware platforms. We implemented unity.cpp in ggml. Now it supports SeamlessM4T model for X2T tasks - Speech-to-text translation (S2TT), Acoustic speech recognition (ASR), Text-to-text translation (T2TT).

The project is still active in development. Contributions are welcome!

Build Copy Chapter

To build the interactive console for S2TT & ASR & T2TT,

cd seamless_communication/ggml
mkdir build; cd build
cmake -DGGML_OPENBLAS=ON \
    -DBUILD_SHARED_LIBS=On \
	  -DCMAKE_BUILD_TYPE=Release \
	  -DCMAKE_CXX_FLAGS="-g2 -fno-omit-frame-pointer" \
    ..
make -j4 unity # Interactive Console

For more build commands see Makefile.

CLI usage Copy Chapter

S2TT Copy Section

Command to launch an interactive console for S2TT & ASR, note that the model already includes vocabulary needed to detokenize.

OPENBLAS_NUM_THREADS=8 ./bin/unity --model seamlessM4T_medium.ggml

In the console, enter "wav_file tgt_lang" - the path of local waveform file and target language, separated by space. Note that the first run would include some “warm up” time so could be slow.

T2TT Copy Section

Launching command:

OPENBLAS_NUM_THREADS=8 ./bin/unity --model nllb-200_dense_1b.ggml --text

In the console, enter "input_text tgt_lang" - input text and target langauge, separated by space. Note that the language code should align with NLLB BCP-47 code, NOT 3-letter language code as S2TT task with Seamless. Unifying this is on todo list.

Model downloads Copy Section

Converted ggml models could be downloaded from

For more details of NLLB models, please check https://github.com/facebookresearch/fairseq/tree/nllb.

Fairseq2 model conversion Copy Chapter

Models from fairseq2 checkpoints could be converted to ggml automatically with ggml_convert.py.

python ggml_convert.py -m MODEL_NAME

where MODEL_NAME corresponds to asset cards in fairseq2 / seamless_communication, e.g. seamlessM4T_medium, seamlessM4T_large

Python bindings Copy Chapter

We also utilize ggml python bindings for better dev experience. For examples of running unity.cpp in python, refer to tests in test_unity_cpp.py.

[Optional]Dependencies Copy Chapter

OpenBLAS Copy Section

We strongly suggest building with OpenBLAS, as we've seen 8x speedup on test machine.

libsndfile Copy Section

This is needed only for the console to load waveform, but not the library.

Overview (`docs/expressive/README.md`) Copy Section

SeamlessExpressive

SeamlessExpressive model consists of two main modules: (1) Prosody UnitY2, which is a prosody-aware speech-to-unit translation model based on UnitY2 architecture; and (2) PRETSSEL, which is a unit-to-speech model featuring cross-lingual expressivity preservation.

Prosody UnitY2 Copy Chapter

Prosody UnitY2 is an expressive speech-to-unit translation model, injecting expressivity embedding from PRETSSEL into the unit generation. It could transfer phrase-level prosody such as speech rate or pauses.

PRETSSEL Copy Chapter

Paralinguistic REpresentation-based
TextleSS acoustic modEL (PRETSSEL) is an expressive unit-to-speech generator, and it can efficiently disentangle semantic and expressivity components from speech. It transfers utterance-level expressivity like the style of one's voice.

Benchmark Datasets

mExpresso (Multilingual Expresso) Copy Chapter

mExpresso is an expressive S2ST dataset that includes seven styles of read speech (i.e., default, happy, sad, confused, enunciated, whisper and laughing) between English and five other languages -- French, German, Italian, Mandarin and Spanish. We create the dataset by expanding a subset of read speech in Expresso Dataset. We first translate the English transcriptions into other languages, including the emphasis markers in the transcription, and then the gender matched bilingual speakers read the translation in the style suggested by the markers.

We are currently open source the text translation of the other language to enable evaluating English to other directions. We will open source the audio files in the near future.

Text translation in other languages can be Downloaded.

Statistics of mExpresso Copy Section

Create mExpresso S2T dataset by downloading and combining with English Expresso Copy Section

Run the following command to create English to other langauges speech-to-text dataset from scratch. It will first download the English Expresso dataset, downsample the audio to 16k Hz, and join with the text translation to form the manifest.

python3 -m seamless_communication.cli.expressivity.data.prepare_mexpresso \
    <OUTPUT_FOLDER>

The output manifest will be located at <OUTPUT_FOLDER>/{dev,test}_mexpresso_eng_{spa,fra,deu,ita,cmn}.tsv

Automatic evaluation

Python package dependencies (on top of seamless_communication, coming from stopes pipelines):

• Unidecode
• scipy
• phonemizer
• s3prl
• syllables
• ipapy
• pkuseg
• nltk
• fire
• inflect

pip install Unidecode scipy phonemizer s3prl syllables ipapy pkuseg nltk fire inflect

As described in Section 4.3 we use following automatic metrics:

1. ASR-BLEU: refer to /src/seamless_communication/cli/eval_utils to see how the OpenAI whisper ASR model is used to extract transcriptions from generated audios.

2. Vocal Style Similarity: refer to stopes/eval/vocal_style_similarity for implementation details.

3. AutoPCP: refer to stopes/eval/auto_pcp for implementation details.

4. Pause and Rate scores: refer to stopes/eval/local_prosody for implementation details. Rate score corresponds to the syllable speech rate spearman correlation between source and predicted speech. Pause score corresponds to the weighted mean joint score produced by stopes/eval/local_prosody/compare_utterances.py script from stopes repo.

Evaluation results: mExpresso Copy Chapter

Please see mExpresso section on how to download evaluation data

Important Notes:

• We used empirically chosen duration factors per each tgt language towards the best perceptual quality: 1.0 (default) for cmn, spa, ita; 1.1 for deu; 1.2 for fra. Same settings were used to report results in the "Seamless: Multilingual Expressive and Streaming Speech Translation" paper.

• Results here slightly differs from ones shown in the paper due to several descrepancies in the pipeline: results reported here use pipeline w/ fairseq2 backend for model's inference and pipeline includes watermarking.

Step-by-step evaluation Copy Section

Pre-requisite: all steps described here assume that the generation/inference has been completed following steps.

For stopes installation please refer to stopes/eval.

The resulting directory of generated outputs:

export SPLIT="dev_mexpresso_eng_spa" # example, change for your split
export TGT_LANG="spa"
export SRC_LANG="eng"
export GENERATED_DIR="path_to_generated_output_for_given_data_split"
export GENERATED_TSV="generate-${SPLIT}.tsv"
export STOPES_ROOT="path_to_stopes_code_repo"
export SC_ROOT="path_to_this_repo"

ASR-BLEU evaluation

python ${SC_ROOT}/src/seamless_communication/cli/expressivity/evaluate/run_asr_bleu.py \
    --generation_dir_path=${GENERATED_DIR} \
    --generate_tsv_filename=generate-${SPLIT}.tsv \
    --tgt_lang=${TGT_LANG}

• generate-${SPLIT}.tsv is an expected output from inference described in pre-requisite

After completion resulting ASR-BLEU score is written in ${GENERATED_DIR}/s2st_asr_bleu_normalized.json.

Vocal Style Similarity

Download & set WavLM finetuned ckpt path (${SPEECH_ENCODER_MODEL_PATH}) as described in stopes README to reproduce our vocal style similarity eval.

python -m stopes.modules +vocal_style_similarity=base \
    launcher.cluster=local \
    vocal_style_similarity.model_type=valle \
    +vocal_style_similarity.model_path=${SPEECH_ENCODER_MODEL_PATH} \
    +vocal_style_similarity.input_file=${GENERATED_DIR}/${GENERATED_TSV} \
    +vocal_style_similarity.output_file=${GENERATED_DIR}/vocal_style_sim_result.txt \
    vocal_style_similarity.named_columns=true \
    vocal_style_similarity.src_audio_column=src_audio \
    vocal_style_similarity.tgt_audio_column=hypo_audio

• We report average number from all utterance scores written in ${GENERATED_DIR}/vocal_style_sim_result.txt.

AutoPCP

python -m stopes.modules +compare_audios=AutoPCP_multilingual_v2 \
    launcher.cluster=local \
    +compare_audios.input_file=${GENERATED_DIR}/${GENERATED_TSV} \
    compare_audios.src_audio_column=src_audio \
    compare_audios.tgt_audio_column=hypo_audio \
    +compare_audios.named_columns=true \
    +compare_audios.output_file=${GENERATED_DIR}/autopcp_result.txt

• We report average number from all utterance scores written in ${GENERATED_DIR}/autopcp_result.txt.

Pause and Rate

This stage includes 3 steps: (1) src lang annotation, (2) tgt lang annotation, (3) pairwise comparison

# src lang pause&rate annotation
python ${STOPES_ROOT}/stopes/eval/local_prosody/annotate_utterances.py \
    +data_path=${GENERATED_DIR}/${GENERATED_TSV} \
    +result_path=${GENERATED_DIR}/${SRC_LANG}_speech_rate_pause_annotation.tsv \
    +audio_column=src_audio \
    +text_column=src_text \
    +speech_units=[syllable] \
    +vad=true \
    +net=true \
    +lang=$SRC_LANG \
    +forced_aligner=fairseq2_nar_t2u_aligner

# tgt lang pause&rate annotation
python ${STOPES_ROOT}/stopes/eval/local_prosody/annotate_utterances.py \
    +data_path=${GENERATED_DIR}/${GENERATED_TSV} \
    +result_path=${GENERATED_DIR}/${TGT_LANG}_speech_rate_pause_annotation.tsv \
    +audio_column=hypo_audio \
    +text_column=s2t_out \
    +speech_units=[syllable] \
    +vad=true \
    +net=true \
    +lang=$TGT_LANG \
    +forced_aligner=fairseq2_nar_t2u_aligner

# pair wise comparison
python ${STOPES_ROOT}/stopes/eval/local_prosody/compare_utterances.py \
    +src_path=${GENERATED_DIR}/${SRC_LANG}_speech_rate_pause_annotation.tsv \
    +tgt_path=${GENERATED_DIR}/${TGT_LANG}_speech_rate_pause_annotation.tsv \
    +result_path=${GENERATED_DIR}/${SRC_LANG}_${TGT_LANG}_pause_scores.tsv \
    +pause_min_duration=0.1

• For Rate reporting, please see the aggregation function get_rate in ${SC_ROOT}/src/seamless_communication/cli/expressivity/evaluate/post_process_pauserate.py;
• For Pause reporting, please see the aggregation function get_pause in ${SC_ROOT}/src/seamless_communication/cli/expressivity/evaluate/post_process_pauserate.py.

Overview (`docs/expressive/seamless_align_expressive_README.md`) Copy Section

SeamlessAlignExpressive

Building upon our past work with WikiMatrix, CCMatrix, NLLB, SpeechMatrix and SeamlessM4T, we’re introducing the first expressive speech alignment procedure. Starting with raw data, the expressive alignment procedure automatically discovers pairs of audio segments sharing not only the same meaning, but the same overall expressivity. To showcase this procedure, we are making metadata available to create a benchmarking dataset called SeamlessAlignExpressive, that can be used to validate the quality of our alignment method. SeamlessAlignExpressive is the first large-scale collection of multilingual audio alignments for expressive translation for benchmarking.

Format Copy Chapter

The metadata files are space separated, gzip files. Each file corresponds to one alignment direction. File naming convention: we use 2 letters with an 'A': e.g. frA, enA, deA.

For example, the direction deA-enA corresponds to information for reconstructing German speech to English speech alignments.

Each line has 9 columns.

The columns correspond to:
- direction: direction, e.g. enA-deA
- side: side, e.g. enA or deA
- line_no: alignment number
- cc_warc: The public CC warc file reference containing the public audio url
- duration: original file duration
- audio_speech_segment_url: public audio reference
- audio_speech_start_frame: start frame when the audio is resampled at 16kHz
- audio_speech_end_frame: end frame when the audio is resampled at 16kHz
- laser_score: score of the alignment

Data Copy Chapter

deA-enA enA-esA enA-frA enA-itA enA-zhA

Overview (`docs/m4t/README.md`) Copy Section

SeamlessM4T

SeamlessM4T is our foundational all-in-one Massively Multilingual and Multimodal Machine Translation model delivering high-quality translation for speech and text in nearly 100 languages.

SeamlessM4T models support:

• :microphone: 101 languages for speech input.
• :speech_balloon: 96 Languages for text input/output.
• :speaker: 35 languages for speech output.

This unified model enables multiple tasks without relying on multiple separate models:

• Speech-to-speech translation (S2ST)
• Speech-to-text translation (S2TT)
• Text-to-speech translation (T2ST)
• Text-to-text translation (T2TT)
• Automatic speech recognition (ASR).

[!NOTE]
SeamlessM4T v2 and v1 are also supported in the 🤗 Transformers library, more on it in the dedicated section below.

SeamlessM4T v1 Copy Chapter

The v1 version of SeamlessM4T is a multitask adaptation of the UnitY architecture (Inaguma et al., 2023).
UnitY is a two-pass direct S2ST architecture which first generates textual representations and subsequently predicts discrete acoustic units.

SeamlessM4T v2 Copy Chapter

The v2 version of SeamlessM4T is a multitask adaptation of our novel UnitY2 architecture.
Unity2 with its hierarchical character-to-unit upsampling and non-autoregressive text-to-unit decoding considerably improves over SeamlessM4T v1 in quality and inference speed.

SeamlessM4T models Copy Chapter

We provide the extensive evaluation results of seamlessM4T-Large and SeamlessM4T-Medium reported in the paper (as averages) in the metrics files above.

The evaluation data ids for FLEURS, CoVoST2 and CVSS-C can be found here

Using SeamlessM4T models Copy Chapter

`m4t_predict` with CLI: Copy Section

Inference is run with the CLI, from the root directory of the repository.

The model can be specified with --model_name seamlessM4T_v2_large, seamlessM4T_large or seamlessM4T_medium:

# S2ST:
m4t_predict <path_to_input_audio> --task s2st --tgt_lang <tgt_lang> --output_path <path_to_save_audio> --model_name seamlessM4T_v2_large

# S2T:
m4t_predict <path_to_input_audio> --task s2tt --tgt_lang <tgt_lang> --model_name seamlessM4T_v2_large

# T2TT:
m4t_predict <input_text> --task t2tt --tgt_lang <tgt_lang> --src_lang <src_lang> --model_name seamlessM4T_v2_large

# T2ST:
m4t_predict <input_text> --task t2st --tgt_lang <tgt_lang> --src_lang <src_lang> --output_path <path_to_save_audio> --model_name seamlessM4T_v2_large

# ASR:
m4t_predict <path_to_input_audio> --task asr --tgt_lang <tgt_lang> --model_name seamlessM4T_v2_large

Inference with `Translator`: Copy Section

Inference calls for the Translator object instantiated with a multitask UnitY or UnitY2 model with the options:

• seamlessM4T_v2_large
• seamlessM4T_large
• seamlessM4T_medium

and a vocoder:

• vocoder_v2 for seamlessM4T_v2_large.
• vocoder_36langs for seamlessM4T_large or seamlessM4T_medium.

import torch
from seamless_communication.inference import Translator


# Initialize a Translator object with a multitask model, vocoder on the GPU.
translator = Translator("seamlessM4T_large", "vocoder_36langs", torch.device("cuda:0"), torch.float16)

Now predict() can be used to run inference as many times on any of the supported tasks.

Given an input audio with or an input text in <src_lang>,
we first set the text_generation_opts, unit_generation_opts and then translate into <tgt_lang> as follows:

S2ST and T2ST (speech output):

# S2ST
text_output, speech_output = translator.predict(
    input=<path_to_input_audio>,
    task_str="S2ST",
    tgt_lang=<tgt_lang>,
    text_generation_opts=text_generation_opts,
    unit_generation_opts=unit_generation_opts
)

# T2ST
text_output, speech_output = translator.predict(
    input=<input_text>,
    task_str="T2ST",
    tgt_lang=<tgt_lang>,
    src_lang=<src_lang>,
    text_generation_opts=text_generation_opts,
    unit_generation_opts=unit_generation_opts
)

Note that <src_lang> must be specified for T2ST.

The generated units are synthesized and the output audio file is saved with:

# Save the translated audio output:
import torchaudio
torchaudio.save(
    <path_to_save_audio>,
    speech_output.audio_wavs[0][0].cpu(),
    sample_rate=speech_output.sample_rate,
)

S2TT, T2TT and ASR (text output):

# S2TT
text_output, _ = translator.predict(
    input=<path_to_input_audio>,
    task_str="S2TT",
    tgt_lang=<tgt_lang>,
    text_generation_opts=text_generation_opts,
    unit_generation_opts=None
)

# ASR
# This is equivalent to S2TT with `<tgt_lang>=<src_lang>`.
    text_output, _ = translator.predict(
    input=<path_to_input_audio>,
    task_str="ASR",
    tgt_lang=<src_lang>,
    text_generation_opts=text_generation_opts,
    unit_generation_opts=None
)

# T2TT
text_output, _ = translator.predict(
    input=<input_text>,
    task_str="T2TT",
    tgt_lang=<tgt_lang>,
    src_lang=<src_lang>,
    text_generation_opts=text_generation_opts,
    unit_generation_opts=None
)

Note that <src_lang> must be specified for T2TT

To reproduce the seamless papers results (v1 or v2), or to evaluate using the same metrics over your own test sets, please check out the Evaluation README here.

Inference with 🤗 `Transformers` Copy Chapter

SeamlessM4T is available in the 🤗 Transformers library, requiring minimal dependencies. Steps to get started:

1. First install the 🤗 Transformers library from main and sentencepiece:

pip install git+https://github.com/huggingface/transformers.git sentencepiece

2. Run the following Python code to generate speech samples. Here the target language is Russian:

import torchaudio
from transformers import AutoProcessor, SeamlessM4Tv2Model

processor = AutoProcessor.from_pretrained("facebook/seamless-m4t-v2-large")
model = SeamlessM4Tv2Model.from_pretrained("facebook/seamless-m4t-v2-large")

# from text
text_inputs = processor(text="Hello, my dog is cute", src_lang="eng", return_tensors="pt")
audio_array_from_text = model.generate(**text_inputs, tgt_lang="rus")[0].cpu().squeeze()

# from audio
audio, orig_freq = torchaudio.load("https://www2.cs.uic.edu/~i101/SoundFiles/preamble10.wav")
audio = torchaudio.functional.resample(audio, orig_freq=orig_freq, new_freq=16_000) # must be a 16 kHz waveform array
audio_inputs = processor(audios=audio, return_tensors="pt")
audio_array_from_audio = model.generate(**audio_inputs, tgt_lang="rus")[0].cpu().squeeze()

3. Listen to the audio samples either in an ipynb notebook:

from IPython.display import Audio

sample_rate = model.config.sampling_rate
Audio(audio_array_from_text, rate=sample_rate)
Audio(audio_array_from_audio, rate=sample_rate)

Or save them as a .wav file using a third-party library, e.g. torchaudio:

torchaudio.save(
    <path_to_save_audio>,
    audio_array_from_audio,  # or audio_array_from_text
    sample_rate=model.config.sampling_rate,
)

2. (bis) To run inference for text generating tasks (T2TT, ASR or S2TT), it is recommended to use dedicated models. With that, only the required sub-modules will be loaded. For exmaple, text-to-text translation from English to Bulgarian, is performed as follows:

from transformers import AutoProcessor, SeamlessM4Tv2ForTextToText
processor = AutoProcessor.from_pretrained("facebook/seamless-m4t-v2-large")
model = SeamlessM4Tv2ForTextToText.from_pretrained("facebook/seamless-m4t-v2-large")

src_lang, tgt_lang = "eng", "bul"
text_inputs = processor(text='Hello, my dog is cute', src_lang=src_lang, return_tensors="pt")
decoder_input_ids = model.generate(**text_inputs, tgt_lang=tgt_lang)[0].tolist()
translated_text = processor.decode(decoder_input_ids, skip_special_tokens=True)
print(f"{tgt_lang}: {translated_text}")

[!NOTE]
For more details on using the SeamlessM4T model for inference using the 🤗 Transformers library, refer to the
SeamlessM4T v2 docs, the
SeamlessM4T v1 docs or to this hands-on Google Colab.

Finetuning SeamlessM4T models Copy Chapter

Please check out the Finetuning README here.

Supported Languages: Copy Chapter

Listed below, are the languages supported by SeamlessM4T-large (v1/v2).
The source column specifies whether a language is supported as source speech (Sp) and/or source text (Tx).
The target column specifies whether a language is supported as target speech (Sp) and/or target text (Tx).

Note that seamlessM4T-medium supports 200 languages in the text modality, and is based on NLLB-200 (see full list in asset card)

Citation Copy Chapter

For UnitY, please cite :

@inproceedings{inaguma-etal-2023-unity,
    title="{U}nit{Y}: Two-pass Direct Speech-to-speech Translation with Discrete Units",
    author="Inaguma, Hirofumi  and Popuri, Sravya  and Kulikov, Ilia  and Chen, Peng-Jen  and Wang, Changhan  and Chung, Yu-An  and Tang, Yun  and Lee, Ann  and Watanabe, Shinji  and Pino, Juan",
    booktitle="Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    year="2023",
    url="https://aclanthology.org/2023.acl-long.872",
}

For SeamlessM4T v1, please cite :

@article{seamlessm4t2023,
  title={SeamlessM4T: Massively Multilingual \& Multimodal Machine Translation},
  author={{Seamless Communication}, Lo\"{i}c Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye,  Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-juss\`{a} \footnotemark[3], Onur \,{C}elebi,Maha Elbayad,Cynthia Gao, Francisco Guzm\'an, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang},
  journal={ArXiv},
  year={2023}
}

For SeamlessM4T v2, please cite :

@inproceedings{seamless2023,
   title="Seamless: Multilingual Expressive and Streaming Speech Translation",
   author="{Seamless Communication}, Lo{\"i}c Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rakotoarison, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R. Costa-juss{\`a}, Maha Elbayad, Hongyu Gong, Francisco Guzm{\'a}n, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alex Mourachko, Benjamin Peloquin, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, Mary Williamson",
  journal={ArXiv},
  year={2023}
}

Overview (`docs/m4t/on_device_README.md`) Copy Section

On-device Models [Experimental]

Apart from SeamlessM4T-LARGE (2.3B) and SeamlessM4T-MEDIUM (1.2B) models, we are also developing a small model (281M) targeting for on-device inference.
This folder contains an example to run an exported small model covering most tasks (ASR/S2TT/S2ST). The model could be executed on popular mobile devices with Pytorch Mobile (https://pytorch.org/mobile/home/).

Updates Copy Chapter

[2023/8/23] Uploaded new on-device models with several fixes to reduce size and avoid OOM. Metrics should be close to what's reported below, will rerun eval and update.

Overview Copy Chapter

UnitY-Small-S2T is a pruned version of UnitY-Small without 2nd pass unit decoding.

Inference Copy Chapter

To use exported model, users don't need seamless_communication or fairseq2 dependency.

import torchaudio
import torch
audio_input, _ = torchaudio.load(TEST_AUDIO_PATH) # Load waveform using torchaudio

s2t_model = torch.jit.load("unity_on_device_s2t.ptl") # Load exported S2T model
with torch.no_grad():
    text = s2t_model(audio_input, tgt_lang=TGT_LANG) # Forward call with tgt_lang specified for ASR or S2TT
print(text) # Show text output 

s2st_model = torch.jit.load("unity_on_device.ptl")
with torch.no_grad():
    text, units, waveform = s2st_model(audio_input, tgt_lang=TGT_LANG) # S2ST model also returns waveform
print(text)
torchaudio.save(f"{OUTPUT_FOLDER}/result.wav", waveform.unsqueeze(0), sample_rate=16000) # Save output waveform to local file

Also running the exported model doesn't need python runtime. For example, you could load this model in C++ following this tutorial, or building your own on-device applications similar to this example

Metrics Copy Chapter

S2TT BLEU / S2ST ASR-BLEU on FLEURS Copy Section

For ASR-BLEU, we follow the same protocol as SeamlessM4T Large/Medium models: We used Whisper-large-v2 for Eng-X and Whisper-medium for X-Eng when evaluating ASR BLEU.

ASR WER on FLEURS Copy Section

Overview (`docs/m4t/seamless_align_README.md`) Copy Section

Seamless - Speech to Speech and Speech to Text Metadata

This document contains metadata information for reconstructing the dataset we used for training our models.

Format Copy Chapter

The metadata format is similar to NLLB bitext format with some small differences.

The metadata files are tab separated, gzip files. Each file corresponds to one alignment direction.

File naming convention:

• for text, we use 3 letters: e.g. fra, eng, tur
• for audio, we use 2 letters and a 'A': e.g. frA, enA, trA

For example, the direction eng-trA corresponds to information for reconstructing English text with Turkish speech alignments. Similarly, enA-jpn corresponds to "English speech with Japanese text", and enA-frA corresponds to "English speech with French speech".

Each line has 11 columns.

For Audio, the columns correspond to:

- `cc_warc`: The warc file reference containing the public audio url
- `cc_sha`: not used
- `audio_speeh_segment_url`: space separated audio reference. See below.
- `cc_lineno`: not used
- `paragraph_digest`: expected duration of the whole audio file (without start/end frame trimming)
- `sentence_digest`: not used
- `text_lid_score`: not used
- `laser_score`: score of the alignment
- `direction`: direction, e.g. `enA-jpn`
- `side`: side, e.g. `enA` or `jpn`
- `line_no`: alignment number

audio_speech_segment_url is a space separated audio reference. It has the following format:
<start_frame> <end_frame>, where start_frame and end_frame correspond to the segment that needs to be extracted from the audio file that is referenced at ``, resampled at 16000 Hz.

For text, the columns are similar to NLLB format (except being tab separated here):

• If the metadata comes from Common Crawl:

  • cc_warc: the reference to the Common Crawl WET file
  • cc_sha: the document sha1 in the WET file
  • cc_document_url: the url of the document referenced in the WET file
  • cc_lineno: the line number in the document referenced in the WET file
  • paragraph_digest: xxhash.xxh3_64_intdigest of the paragraph
  • sentence_digest: xxhash.xxh3_64_intdigest of the sentence
  • text_lid_score: language identification score, when available
  • laser_score: score of the alignment
  • direction: direction, e.g. enA-jpn
  • side: side, e.g. enA or jpn
  • line_no: alignment number

• If the metadata comes from other corpus:

  • corpus: corpus name
  • cc_sha: not used
  • cc_document_url: not used
  • lineno: line number in the document
  • paragraph_digest: xxhash.xxh3_64_intdigest of the paragraph
  • sentence_digest: xxhash.xxh3_64_intdigest of the sentence
  • text_lid_score: language identification score, when available
  • laser_score: score of the alignment
  • direction: direction, e.g. enA-jpn
  • side: side, e.g. enA or jpn
  • line_no: alignment number

Data Copy Chapter

Update: 30 Nov 2023

We are publishing an extension of the previous speech to speech release.

afA-enA amA-enA arA-enA asA-enA azA-enA beA-enA bgA-enA bnA-enA bsA-enA caA-enA csA-enA cyA-enA daA-enA deA-enA elA-enA enA-esA enA-etA enA-fiA enA-frA enA-gaA enA-glA enA-guA enA-heA enA-hiA enA-hrA enA-huA enA-hyA enA-idA enA-isA enA-itA enA-jaA enA-jvA enA-kaA enA-kiA enA-kkA enA-knA enA-koA enA-kyA enA-lgA enA-loA enA-ltA enA-lvA enA-mkA enA-mlA enA-mnA enA-mrA enA-msA enA-mtA enA-neA enA-nlA enA-noA enA-orA enA-paA enA-pbA enA-plA enA-psA enA-ptA enA-rnA enA-ruA enA-sdA enA-skA enA-slA enA-srA enA-svA enA-swA enA-taA enA-teA enA-tgA enA-thA enA-trA enA-ukA enA-urA enA-uzA enA-viA enA-yoA enA-zhA

Update: 25 Sep 2023

We are publishing updated metadata with the expected duration of the original audio file in the column paragraph_digest (originally not used for audio).

arb-enA ben-enA cat-enA dan-enA enA-est enA-fin enA-jpn enA-mlt enA-nld enA-pol enA-por enA-ron enA-slk enA-swe enA-swh enA-tur enA-ukr enA-urd enA-vie arA-enA arA-eng beA-enA caA-enA caA-eng csA-enA csA-eng cyA-enA cyA-eng daA-enA daA-eng deA-enA deA-eng enA-esA enA-fiA enA-frA enA-hiA enA-idA enA-itA enA-knA enA-koA enA-mtA enA-nlA enA-plA enA-ptA enA-rnA enA-ruA enA-skA enA-svA enA-swA enA-taA enA-teA enA-tgA enA-thA enA-trA enA-ukA enA-urA enA-uzA enA-viA enA-zhA eng-esA eng-fiA eng-frA eng-hiA eng-idA eng-itA eng-knA eng-koA eng-mtA eng-nlA eng-plA eng-ptA eng-rnA eng-ruA eng-skA eng-swA eng-taA eng-teA eng-tgA eng-thA eng-trA eng-ukA eng-urA eng-uzA eng-viA eng-zhA

You can find the legacy metadata (without duration information) here:

Legacy Data Copy Section

arb-enA ben-enA cat-enA dan-enA enA-est enA-fin enA-jpn enA-mlt enA-nld enA-pol enA-por enA-ron enA-slk enA-swe enA-swh enA-tur enA-ukr enA-urd enA-vie arA-enA arA-eng beA-enA caA-enA caA-eng csA-enA csA-eng cyA-enA cyA-eng daA-enA daA-eng deA-enA deA-eng enA-esA enA-fiA enA-frA enA-hiA enA-idA enA-itA enA-knA enA-koA enA-mtA enA-nlA enA-plA enA-ptA enA-rnA enA-ruA enA-skA enA-svA enA-swA enA-taA enA-teA enA-tgA enA-thA enA-trA enA-ukA enA-urA enA-uzA enA-viA enA-zhA eng-esA eng-fiA eng-frA eng-hiA eng-idA eng-itA eng-knA eng-koA eng-mtA eng-nlA eng-plA eng-ptA eng-rnA eng-ruA eng-skA eng-swA eng-taA eng-teA eng-tgA eng-thA eng-trA eng-ukA eng-urA eng-uzA eng-viA eng-zhA

Download script Copy Chapter

You can use the wet_lines script to download and gather aligned text information from the metadata. This script can be found here.

Example usage: Copy Section

zcat seamless.dataset.metadata.public.enA-swA.tsv.gz | egrep ^crawl-data | tr '\t' ' ' | wet_lines

Based on metadata information it receives from stdin, wet_lines will download the corpora, find the paragraph and print the input with an additional column which corresponds to the text of the paragraph.

In order to retrieve the sentences from these paragraphs, one can use the sentence splitter available here. It will print the input (metadata + paragraph) with an additional column which corresponds to the text of the sentence.

Reconstructing sentences from metadata: Copy Section

xzcat metadatafile.xz | egrep ^crawl-data | wet_lines | python -c "from sentence_cleaner_splitter.cleaner_splitter import *; split_clean()"

Overview (`docs/m4t/unity2_aligner_README.md`) Copy Section

UnitY2 forced alignment extractor

Please refer to Section 3.3.2 of the "Seamless: Multilingual Expressive and Streaming Speech Translation" paper to read more details about aligner design & training.

We provide a light-weight wrapper to extract alignments between given text and acoustic unit sequences. Unit extractor is also available from the wrapper itself.

Alignment extractor codebase Copy Chapter

The entire codebase is located in /src/seamless_communication/models/aligner. It is built using fairseq2 library. This time we release a mutlilingual (38 languages following SeamlessM4Tv2 target languages) checkpoint to load the alignment toolkit. This checkpoint corresponds to nar_t2u_aligner asset card.

Usage examples Copy Chapter

For large-scale alignment extraction offline unit extraction is preferred. Refer to /src/seamless_communication/cli/m4t/audio_to_units for more details on offline unit extraction.

Alignment extractor initialization:

from seamless_communication.models.aligner.alignment_extractor import AlignmentExtractor
from fairseq2.typing import Device
import torch

extractor = AlignmentExtractor(
    aligner_model_name_or_card="nar_t2u_aligner",
    unit_extractor_model_name_or_card="xlsr2_1b_v2",
    unit_extractor_output_layer=35,
    unit_extractor_kmeans_model_uri="https://dl.fbaipublicfiles.com/seamlessM4T/models/unit_extraction/kmeans_10k.npy",
)

• large unit extractor checkpoint will be downloaded, this takes time

• by default cpu device is used, but fp16 (dtype=torch.float16) & cuda (device=Device("cuda")) are supported, see class constructor for details

Extracting alignment

Ru audio example:

• audio link: https://models.silero.ai/denoise_models/sample0.wav (thanks Silero team for public audio samples)

• ru_transcription: первое что меня поразило это необыкновенно яркий солнечный свет похожий на электросварку

alignment_durations, _, tokenized_text_tokens = extractor.extract_alignment("sample0.wav", ru_transcription, plot=True, add_trailing_silence=True)

• audio will be resampled to 16kHz for unit extraction

• alignment_durations contains number of units (20ms frames) aligned per each token from tokenized_text_tokens.

• add_trailing_silence sets extra silence token in the end of the given text sequence. That is useful when there is no terminal punctuation provided in the text itself.

Ru alignment plot:

En audio example:

• audio link: https://dl.fbaipublicfiles.com/seamlessM4T/LJ037-0171_sr16k.wav

• en_transcription: the examination and testimony of the experts enabled the commision to conclude that five shots may have been fired.

alignment_durations, _, tokenized_text_tokens = extractor.extract_alignment("LJ037-0171_sr16k.wav", en_transcription, plot=True, add_trailing_silence=False)

• here we set add_trailing_silence to False since terminal punctuation exists, but True will also work

En alignment plot:

Integration test Copy Chapter

If you encounter issues with produced alignments, please run integration test with the alignment extraction toolkit to make sure that your environment works good.

Run from the repo root:

pytest -vv tests/integration/models/test_unity2_aligner.py

Overview (`docs/streaming/README.md`) Copy Section

SeamlessStreaming

SeamlessStreaming is a multilingual streaming translation model. It supports:

• Streaming Automatic Speech Recognition on 96 languages.
• Simultaneous translation on 101 source languages for speech input.
• Simultaneous translation on 96 target languages for text output.
• Simultaneous translation on 36 target languages for speech output.

Check out the SeamlessM4T README for more details on supported languages.

SeamlessStreaming models Copy Chapter

The evaluation data ids for FLEURS, CoVoST2 and CVSS-C can be found here

Evaluating SeamlessStreaming models Copy Chapter

To reproduce our results, or to evaluate using the same metrics over your own test sets, please check out the Evaluation README here. Streaming evaluation depends on the SimulEval library.

Citation Copy Chapter

For EMMA, please cite :

@article{ma_efficient_2023,
  author={Ma, Xutai and Sun, Anna and Ouyang, Siqi and Inaguma, Hirofumi and Tomasello, Paden},
  title={Efficient Monotonic Multihead Attention},
  year={2023},
  url={https://ai.meta.com/research/publications/efficient-monotonic-multihead-attention/},
}

For SeamlessStreaming, please cite :

@inproceedings{seamless2023,
   title="Seamless: Multilingual Expressive and Streaming Speech Translation",
   author="{Seamless Communication}, Lo{\"i}c Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rakotoarison, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R. Costa-juss{\`a}, Maha Elbayad, Hongyu Gong, Francisco Guzm{\'a}n, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alex Mourachko, Benjamin Peloquin, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, Mary Williamson",
  journal={ArXiv},
  year={2023}
}

Overview (`ggml/examples/python/README.md`) Copy Section

Simple autogenerated Python bindings for ggml

This folder contains:

• Scripts to generate full Python bindings from ggml headers (+ stubs for autocompletion in IDEs)
• Some barebones utils (see ggml/utils.py):
  • ggml.utils.init builds a context that's freed automatically when the pointer gets GC'd
  • ggml.utils.copy copies between same-shaped tensors (numpy or ggml), w/ automatic (de/re)quantization
  • ggml.utils.numpy returns a numpy view over a ggml tensor; if it's quantized, it returns a copy (requires allow_copy=True)
• Very basic examples (anyone wants to port llama2.c?)

Provided you set GGML_LIBRARY=.../path/to/libggml_shared.so (see instructions below), it's trivial to do some operations on quantized tensors:

# Make sure libllama.so is in your [DY]LD_LIBRARY_PATH, or set GGML_LIBRARY=.../libggml_shared.so

from ggml import lib, ffi
from ggml.utils import init, copy, numpy
import numpy as np

ctx = init(mem_size=12*1024*1024)
n = 256
n_threads = 4

a = lib.ggml_new_tensor_1d(ctx, lib.GGML_TYPE_Q5_K, n)
b = lib.ggml_new_tensor_1d(ctx, lib.GGML_TYPE_F32, n) # Can't both be quantized
sum = lib.ggml_add(ctx, a, b) # all zeroes for now. Will be quantized too!

gf = ffi.new('struct ggml_cgraph*')
lib.ggml_build_forward_expand(gf, sum)

copy(np.array([i for i in range(n)], np.float32), a)
copy(np.array([i*100 for i in range(n)], np.float32), b)

lib.ggml_graph_compute_with_ctx(ctx, gf, n_threads)

print(numpy(a, allow_copy=True))
#  0.    1.0439453   2.0878906   3.131836    4.1757812   5.2197266. ...
print(numpy(b))
#  0.  100.        200.        300.        400.        500.         ...
print(numpy(sum, allow_copy=True))
#  0.  105.4375    210.875     316.3125    421.75      527.1875     ...

Prerequisites Copy Section

You'll need a shared library of ggml to use the bindings.

Build libggml_shared.so or libllama.so

As of this writing the best is to use ggerganov/llama.cpp's generated libggml_shared.so or libllama.so, which you can build as follows:

git clone https://github.com/ggerganov/llama.cpp
# On a CUDA-enabled system add -DLLAMA_CUBLAS=1
# On a Mac add -DLLAMA_METAL=1
cmake llama.cpp \
  -B llama_build \
  -DCMAKE_C_FLAGS=-Ofast \
  -DLLAMA_NATIVE=1 \
  -DLLAMA_LTO=1 \
  -DBUILD_SHARED_LIBS=1 \
  -DLLAMA_MPI=1 \
  -DLLAMA_BUILD_TESTS=0 \
  -DLLAMA_BUILD_EXAMPLES=0
( cd llama_build && make -j )

# On Mac, this will be libggml_shared.dylib instead
export GGML_LIBRARY=$PWD/llama_build/libggml_shared.so
# Alternatively, you can just copy it to your system's lib dir, e.g /usr/local/lib

(Optional) Regenerate the bindings and stubs

If you added or changed any signatures of the C API, you'll want to regenerate the bindings (ggml/cffi.py) and stubs (ggml/init.pyi).

Luckily it's a one-liner using regenerate.py:

pip install -q cffi

python regenerate.py

By default it assumes llama.cpp was cloned in ../../../llama.cpp (alongside the ggml folder). You can override this with:

C_INCLUDE_DIR=$LLAMA_CPP_DIR python regenerate.py

You can also edit api.h to control which files should be included in the generated bindings (defaults to llama.cpp/ggml*.h)

In fact, if you wanted to only generate bindings for the current version of the ggml repo itself (instead of llama.cpp; you'd loose support for k-quants), you could run:

API=../../include/ggml/ggml.h python regenerate.py

Develop Copy Chapter

Run tests:

pytest

Alternatives Copy Section

This example's goal is to showcase cffi-generated bindings that are trivial to use and update, but there are already alternatives in the wild:

• https://github.com/abetlen/ggml-python: these bindings seem to be hand-written and use ctypes. It has high-quality API reference docs that can be used with these bindings too, but it doesn't expose Metal, CUDA, MPI or OpenCL calls, doesn't support transparent (de/re)quantization like this example does (see ggml.utils module), and won't pick up your local changes.

• https://github.com/abetlen/llama-cpp-python: these expose the C++ llama.cpp interface, which this example cannot easily be extended to support (cffi only generates bindings of C libraries)

• pybind11 and nanobind are two alternatives to cffi that support binding C++ libraries, but it doesn't seem either of them have an automatic generator (writing bindings is rather time-consuming).

Overview (`src/seamless_communication/inference/README.md`) Copy Section

Running inference

SeamlessM4T Inference Copy Section

Here’s an example of using the CLI from the root directory to run inference.

S2ST task:

m4t_predict <path_to_input_audio> --task s2st --tgt_lang <tgt_lang> --output_path <path_to_save_audio>

T2TT task:

m4t_predict <input_text> --task t2tt --tgt_lang <tgt_lang> --src_lang <src_lang>

Please refer to the inference README for detailed instruction on how to run inference and the list of supported languages on the source, target sides for speech, text modalities.

For running S2TT/ASR natively (without Python) using GGML, please refer to the unity.cpp section.

Transcription Utilities: Denoising and Segmentation

The following information shows how to use denoising and segmentation tools for noisy and long input audios.

Demucs: Audio Denoising Tool Copy Chapter

The 'Demucs' class provides functionality for denoising audio in the transcription pipeline. It supports various configuration options, allowing for fine-tuning denoising performance based on specific requirements.

Key Features:

• Denoising audio using the Demucs model.
• Configurable parameters for denoising.
• Support for both Tensor input and audio file input.
• Automatic cleanup of temporary files generated during denoising.

Installation Copy Section

Manually install demucs:

pip install git+https://github.com/facebookresearch/demucs#egg=demucs

Usage Copy Section

To utilize Demucs for denoising audio, instantiate the Transcriber class and optionally the DenoisingConfig class with desired configuration. 'denoise' parameter is False by default, and needs to be set to True to use denoising.

import torch
from seamless_communication.inference import Transcriber
from seamless_communication.denoise.demucs import DenoisingConfig

model_name = "seamlessM4T_v2_large"
vocoder_name = "vocoder_v2" if model_name == "seamlessM4T_v2_large" else "vocoder_36langs"

transcriber = Transcriber (
    model_name,
    device=torch.device("cpu"),
    dtype=torch.float32,
)

denoise_config = DenoisingConfig(float32= True)

txt = transcriber.transcribe(audio="example.wav", src_lang="eng", denoise=True, denoise_config=denoise_config)

Silero VAD Segmenter: Audio Segmentation Tool Copy Chapter

The 'SileroVADSegmenter' class offers functionality for segmenting long audio recordings into chunks in the transcription pipeline. This tool segments based on speech timestamps.

Key Features:

• Segmenting long audio recordings into chunks based on speech presence.
• Automatic segmenting of all audio longer than the chunk size.
• Configurable parameters for chunk size and pause length.
• Resampling audio to match the model's sample rate.
• Efficient speech probability computation using sliding windows.

Usage Copy Section

To utilize Silero VAD for segmenting audio, instantiate the Transcriber class. When using the transcribe method, audio will be segmented automatically if it is longer than chunk_size_sec, which has a default value of 20. Use a smaller value for better quality transcription.
pause_length_sec determines the duration of silence between segments and has a default value of 1 second. This parameter can be customized.

import torch
from seamless_communication.inference import Transcriber

model_name = "seamlessM4T_v2_large"
vocoder_name = "vocoder_v2" if model_name == "seamlessM4T_v2_large" else "vocoder_36langs"

transcriber = Transcriber (
    model_name,
    device=torch.device("cpu"),
    dtype=torch.float32,
)

input_audio = "example.wav"

txt = transcriber.transcribe(audio=input_audio, src_lang="eng", chunk_size_sec=10, pause_length_sec=.5)

Overview (`src/seamless_communication/cli/streaming/README.md`) Copy Section

Evaluating SeamlessStreaming and Seamless models

SeamlessStreaming is the streaming only model and Seamless is the expressive streaming model.

Quick start: Copy Chapter

Evaluation can be run with the streaming_evaluate CLI.

We use the seamless_streaming_unity for loading the speech encoder and T2U models, and seamless_streaming_monotonic_decoder for loading the text decoder for streaming evaluation. This is already set as defaults for the streaming_evaluate CLI, but can be overridden using the --unity-model-name and --monotonic-decoder-model-name args if required.

Note that the numbers in our paper use single precision floating point format (fp32) for evaluation by setting --dtype fp32. Also note that the results from running these evaluations might be slightly different from the results reported in our paper (which will be updated soon with the new results).

S2TT: Copy Section

Set the task to s2tt for evaluating the speech-to-text translation part of the SeamlessStreaming model.

streaming_evaluate --task s2tt --data-file <path_to_data_tsv_file> --audio-root-dir <path_to_audio_root_directory> --output <path_to_evaluation_output_directory> --tgt-lang <3_letter_lang_code>

Note: The --ref-field can be used to specify the name of the reference column in the dataset.

ASR: Copy Section

Set the task to asr for evaluating the automatic speech recognition part of the SeamlessStreaming model. Make sure to pass the source language as the --tgt-lang arg.

streaming_evaluate --task asr --data-file <path_to_data_tsv_file> --audio-root-dir <path_to_audio_root_directory> --output <path_to_evaluation_output_directory> --tgt-lang <3_letter_source_lang_code>

S2ST: Copy Section

SeamlessStreaming:

Set the task to s2st for evaluating the speech-to-speech translation part of the SeamlessStreaming model.

streaming_evaluate --task s2st --data-file <path_to_data_tsv_file> --audio-root-dir <path_to_audio_root_directory> --output <path_to_evaluation_output_directory> --tgt-lang <3_letter_lang_code>

Seamless:

The Seamless model is a unified model for streaming expressive speech-to-speech translation. Use the --expressive arg for running evaluation of this unified model.

streaming_evaluate --task s2st --data-file <path_to_data_tsv_file> --audio-root-dir <path_to_audio_root_directory> --output <path_to_evaluation_output_directory> --tgt-lang <3_letter_lang_code> --expressive --gated-model-dir <path_to_vocoder_checkpoints_dir>

The Seamless model uses vocoder_pretssel which is a 24KHz version (vocoder_pretssel) by default. In the current version of our paper, we use 16KHz version (vocoder_pretssel_16khz) for the evaluation, so in order to reproduce those results please add this arg to the above command: --vocoder-name vocoder_pretssel_16khz.

vocoder_pretssel or vocoder_pretssel_16khz checkpoints are gated, please check out this section to acquire these checkpoints. Also, make sure to add --gated-model-dir

Overview (`src/seamless_communication/cli/m4t/audio_to_units/README.md`) Copy Section

Convert raw audio into units (unit_extraction)

Raw audio needs to be converted to units to train UnitY models and vocoders. Units act as supervision for UnitY models, and are the input to the vocoders which synthesize speech from these units.

The unit extraction pipeline comprises the following steps:

• Compute features from layer 35 (determined empirically) of the pretrained XLSR v2 model (paper), which is a wav2vec2 model at the core.
• Assign features for each timestep to a collection of precomputed K-Means centroids to produce a sequence of units similar to extracting Hubert units as described in this paper.

Quick start: Copy Chapter

audio_to_units is run with the CLI, from the root directory of the repository.

m4t_audio_to_units <path_to_input_audio>

audio_to_units calls for UnitExtractor which provides a predict method to convert an audio to units.

The convenience method resynthesize_audio of UnitExtractor, can be used to resynthesize audio waveforms from units.

Overview (`src/seamless_communication/cli/m4t/evaluate/README.md`) Copy Section

Evaluating SeamlessM4T models

Refer to the SeamlessM4T README for an overview of the M4T models.

Refer to the inference README for how to run inference with SeamlessM4T models.

Quick start: Copy Chapter

We use SACREBLEU library for computing BLEU scores and JiWER library is used to compute these CER and WER scores.

Evaluation can be run with the CLI, from the root directory of the repository.

The model can be specified with --model_name: seamlessM4T_v2_large or seamlessM4T_large or seamlessM4T_medium

m4t_evaluate --data_file <path_to_data_tsv_file> --task <task_name> --tgt_lang <tgt_lang> --output_path <path_to_save_evaluation_output> --ref_field <ref_field_name> --audio_root_dir <path_to_audio_root_directory>

Note Copy Chapter

1. We use raw (unnormalized) references to compute BLEU scores for S2TT, T2TT tasks.
2. For ASR task, src_lang needs to be passed as
3. --src_lang arg needs to be specified to run evaluation for T2TT task

Overview (`src/seamless_communication/cli/m4t/finetune/README.md`) Copy Section

Finetuning scripts for M4T Copy Chapter

This section demonstrates an example of M4T finetuning on a single translation direction: English-to-Korean.

The trainer and dataloader were designed mainly for demonstration purposes. Their simplicity should facilitate the code transparency and portability.

Data preparation Copy Chapter

M4T training dataset is a multimodal parallel corpus. Each training sample has four parts: audio and text representation of the sample in the source language, and its corresponding audio and text representation in the target language.

That kind of dataset can be prepared using dataset.py script that downloads FLEURS dataset from HuggingFace datasets hub, (optionally) extracts units from the target audio samples, and prepares a manifest consumable by finetune.py. Manifest is a text file where each line represents information about a single dataset sample, serialized in JSON format.

List of input arguments for dataset.py:

--name NAME           HuggingFace name of the dataset to prepare.
  --source_lang SOURCE_LANG
                        M4T langcode of the dataset SOURCE language
  --target_lang TARGET_LANG
                        M4T langcode of the dataset TARGET language
  --split SPLIT         Dataset split/shard to download (`train`, `validation`, `test`)
  --save_dir SAVE_DIR   Directory where the datastets will be stored with HuggingFace datasets cache files
  --huggingface_token HUGGINGFACE_TOKEN
                        Your HuggingFace token, this is necessary for some datasets like GigaSpeech.

Language codes should follow the notation adopted by M4T models.

Below is an example bash script that prepares a training and evaluation dataset for the translation direction English-to-Korean:

export DATASET_DIR=~/m4t_dataset
mkdir -p $DATASET_DIR

m4t_prepare_dataset \
  --name google/fleurs \
  --source_lang eng \
  --target_lang kor \
  --split train \
  --save_dir $DATASET_DIR
m4t_prepare_dataset \
  --name google/fleurs \
  --source_lang eng \
  --target_lang kor \
  --split validation \
  --save_dir $DATASET_DIR

Output manifests will be stored in ${DATASET_DIR}/train_manifest.json and ${DATASET_DIR}/validation_manifest.json.

Finetuning Copy Chapter

finetune.py is an example finetuning script that initializes dataloaders, and launches training loop with periodic scoring against the validation dataset.
It is recommended to launch it with torchrun. Multi-gpu and multi-node training are supported out of the box.

List of input arguments for finetune.py:

--train_dataset TRAIN_DATASET
                        Path to manifest with train samples
  --eval_dataset EVAL_DATASET
                        Path to manifest with eval samples
  --model_name MODEL_NAME
                        Base model name (e.g, `seamlessM4T_medium`, `seamlessM4T_large`)
  --save_model_to SAVE_MODEL_TO
                        Path to save best finetuned model
  --seed SEED           Randomizer seed value
  --batch_size BATCH_SIZE
                        Batch size for training and evaluation
  --patience PATIENCE   Set early termination after `patience` number of evaluations without eval loss improvements
  --max_epochs MAX_EPOCHS
                        Max number of training epochs
  --learning_rate LEARNING_RATE
                        Finetuning learning rate
  --warmup_steps WARMUP_STEPS
                        Number of steps with linearly increasing learning rate
  --eval_steps EVAL_STEPS
                        Get eval loss after each `eval_steps` training steps
  --log_steps LOG_STEPS
                        Log inner loss after each `log_steps` training steps
  --mode {FinetuneMode.SPEECH_TO_SPEECH,FinetuneMode.SPEECH_TO_TEXT,FinetuneMode.TEXT_TO_SPEECH}
                        * `SPEECH_TO_SPEECH` -- finetune S2T and T2U parts of the model;
                        * `TEXT_TO_SPEECH` -- finetune only T2U;
                        * `SPEECH_TO_TEXT` -- finetune only S2T

The scripts supports three modes of finetuning:

• SPEECH_TO_SPEECH: in this case all model weights except the text encoder will be engaged;
• TEXT_TO_SPEECH: only text-to-unit part of the model will be engaged in the finetuning, other weights will be frozen;
• SPEECH_TO_TEXT: only speech-to-text part of the model will be engaged in the finetuning.

The referenced finetuning script does not support finetuning of the text encoder.

Below is an example bash script that launches finetuning of M4T-large on the dataset prepared earlier, using a single node with eight GPUs:

torchrun \
   --rdzv-backend=c10d \
   --rdzv-endpoint=localhost:0 \
   --nnodes=1 \
   --nproc-per-node=8  \
   --no-python \
  m4t_finetune \
   --mode SPEECH_TO_TEXT \
   --train_dataset $DATASET_DIR/train_manifest.json  \
   --eval_dataset $DATASET_DIR/validation_manifest.json \
   --learning_rate 1e-6 \
   --warmup_steps 100 \
   --max_epochs 10 \
   --patience 5 \
   --model_name seamlessM4T_v2_large \
   --save_model_to $DATASET_DIR/checkpoint.pt

Excerpt from an example finetuning log:

/* Detailed source-code truncated for AI context efficiency. */

Overview (`src/seamless_communication/cli/m4t/predict/README.md`) Copy Section

Inference with SeamlessM4T models

Refer to the SeamlessM4T README for an overview of the M4T models.

Inference is run with the CLI, from the root directory of the repository.

The model can be specified with --model_name seamlessM4T_v2_large, seamlessM4T_large or seamlessM4T_medium:

S2ST:

m4t_predict <path_to_input_audio> --task s2st --tgt_lang <tgt_lang> --output_path <path_to_save_audio> --model_name seamlessM4T_v2_large

S2TT:

m4t_predict <path_to_input_audio> --task s2tt --tgt_lang <tgt_lang> --model_name seamlessM4T_v2_large

T2TT:

m4t_predict <input_text> --task t2tt --tgt_lang <tgt_lang> --src_lang <src_lang> --model_name seamlessM4T_v2_large

T2ST:

m4t_predict <input_text> --task t2st --tgt_lang <tgt_lang> --src_lang <src_lang> --output_path <path_to_save_audio> --model_name seamlessM4T_v2_large

ASR:

m4t_predict <path_to_input_audio> --task asr --tgt_lang <tgt_lang> --model_name seamlessM4T_v2_large

Inference breakdown Copy Chapter

Inference calls for the Translator object instantiated with a multitask UnitY or UnitY2 model with the options:

• seamlessM4T_v2_large
• seamlessM4T_large
• seamlessM4T_medium

and a vocoder:

• vocoder_v2 for seamlessM4T_v2_large.
• vocoder_36langs for seamlessM4T_large or seamlessM4T_medium.

import torch
from seamless_communication.inference import Translator


# Initialize a Translator object with a multitask model, vocoder on the GPU.
translator = Translator("seamlessM4T_large", "vocoder_36langs", torch.device("cuda:0"), torch.float16)

Now predict() can be used to run inference as many times on any of the supported tasks.

Given an input audio with or an input text in <src_lang>,
we first set the text_generation_opts, unit_generation_opts and then translate into <tgt_lang> as follows:

S2ST and T2ST (speech output):

# S2ST
text_output, speech_output = translator.predict(
    input=<path_to_input_audio>,
    task_str="S2ST",
    tgt_lang=<tgt_lang>,
    text_generation_opts=text_generation_opts,
    unit_generation_opts=unit_generation_opts
)

# T2ST
text_output, speech_output = translator.predict(
    input=<input_text>,
    task_str="T2ST",
    tgt_lang=<tgt_lang>,
    src_lang=<src_lang>,
    text_generation_opts=text_generation_opts,
    unit_generation_opts=unit_generation_opts
)

Note that <src_lang> must be specified for T2ST.

The generated units are synthesized and the output audio file is saved with:

# Save the translated audio output:
import torchaudio
torchaudio.save(
    <path_to_save_audio>,
    speech_output.audio_wavs[0][0].cpu(),
    sample_rate=speech_output.sample_rate,
)

S2TT, T2TT and ASR (text output):

# S2TT
text_output, _ = translator.predict(
    input=<path_to_input_audio>,
    task_str="S2TT",
    tgt_lang=<tgt_lang>,
    text_generation_opts=text_generation_opts,
    unit_generation_opts=None
)

# ASR
# This is equivalent to S2TT with `<tgt_lang>=<src_lang>`.
    text_output, _ = translator.predict(
    input=<path_to_input_audio>,
    task_str="ASR",
    tgt_lang=<src_lang>,
    text_generation_opts=text_generation_opts,
    unit_generation_opts=None
)

# T2TT
text_output, _ = translator.predict(
    input=<input_text>,
    task_str="T2TT",
    tgt_lang=<tgt_lang>,
    src_lang=<src_lang>,
    text_generation_opts=text_generation_opts,
    unit_generation_opts=None
)

Note that <src_lang> must be specified for T2TT

Overview (`src/seamless_communication/cli/toxicity/etox/README.md`) Copy Section

Tool to compute toxicity in speech (ASR-ETOX) and text (ETOX)

In this tool, we combine an ASR model (M4T or whisper) + the ETOX toxicity detection tool
to compute a toxicity score for speech segments.

ETOX was developed as part of the NLLB project and provides a wordlist detection mechanism for 200 languages. By applying ASR on top of the ETOX detection, we can detect the toxicity in speech. You can find a description of the toxicity detection wordlists in the paper cited below.

ASR-ETOX Usage Copy Chapter

The script works by taking a TSV as input. The TSV needs a header with column names, it can have multiple columns. By defaut the script will look at the audio for the name of the audio file to load, this can be overriden with --audio_column.
The file path in the TSV can be absolute or relative to a root directory specified by --audio_root_dir. They can also be audiozip file formats with the appropriate byteoffset and length, e.g.: fleurs_en_us_ogg_16khz.zip:89474600:49079.

You can choose the ASR model to use, by default it will use seamlessM4T_v2_large. If you prefer to use whisper you can specify a --model_name that starts with whisper_ and finishes with the whisper model name (e.g. whisper_large).

Outputs Copy Chapter

The output of the script is a new TSV file with three columns:

• text the transcription
• toxicity the number of toxic words detected
• bad_words a list of toxic words, separated by ,

Sample Command Copy Chapter

ASR-ETOX

• using M4T:

python -m seamless_communication.cli.toxicity.asr_etox --lang deu --audio_column ref_tgt_audio s2t/en-xx/deu.tsv ~/etox.tsv

• using Whisper:

python -m seamless_communication.cli.toxicity.asr_etox --model_name whisper_large --lang fra --audio_column ref_tgt_audio s2t/en-xx/fra.tsv ~/etox.test.tsv

ETOX

If you only care about getting the toxicity of text, you can use the etox.py script, with one text per line, specifying the language as the first argument.

cut -f 4 fleurs/s2t/en-xx/deu.tsv | python -m seamless_communication.cli.toxicity.etox deu > deu.toxicity.txt

You can also specify an input and output file:

python -m seamless_communication.cli.toxicity.etox deu deu.txt deu.toxicity.txt

Citation

If you use ETOX, ASR-ETOX and SeamlessM4T in your work, please cite:

@misc{costajussà2023toxicity,
      title={Toxicity in Multilingual Machine Translation at Scale},
      author={Marta R. Costa-jussà and Eric Smith and Christophe Ropers and Daniel Licht and Jean Maillard and Javier Ferrando and Carlos Escolano},
      year={2023},
      eprint={2210.03070},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

and

@article{seamlessm4t2023,
  title={SeamlessM4T—Massively Multilingual \& Multimodal Machine Translation},
  author={{Seamless Communication}, Lo\"{i}c Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye,  Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-juss\`{a} \footnotemark[3], Onur \,{C}elebi,Maha Elbayad,Cynthia Gao, Francisco Guzm\'an, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang},
  journal={ArXiv},
  year={2023}
}

Overview (`src/seamless_communication/cli/toxicity/mutox/README.md`) Copy Section

MuTox: MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector

MuTox, the first highly multilingual audio-based dataset with toxicity labels.
The dataset consists of 20k audio utterances for English and Spanish, and 4k for
the other 19 languages. To showcase the quality of this dataset, we train the
MuTox audio-based toxicity classifier, which allows zero-shot toxicity detection
across a broad range of languages. This classifier outperforms existing
text-based trainable classifiers by more than 1% AUC, while increasing the
language coverage from 8 to 100+ languages. When compared to a wordlist-based
classifier that covers a similar number of languages, MuTox improves precision
and recall by ∼2.5 times.

License Copy Chapter

The mutox code and model are licensed under the MIT license (see MIT_LICENSE
file at the root of seamless_communication). The mutox model depends on SONAR
encoders, most are under the MIT license but a few are under CC-BY-NC license.
See the SONAR repository for
details.

Dataset Languages. Copy Chapter

• English,
• Spanish,
• Arabic,
• Bengali,
• Mandarin Chinese,
• Dutch,
• French,
• German,
• Hindi,
• Indonesian,
• Italian,
• Japanese,
• Korean,
• Portuguese,
• Russian,
• Swahili,
• Tagalog,
• Thai,
• Turkish,
• Urdu,
• Vietnamese

Classifier details. Copy Chapter

We use multi-modal and multilingual
SONAR encoders from (Duquenne et
al., 2023). For the classifier, we use variable input sizes for the 3
feedforward layers (1024, 512, and 128).

The predictions of the classifier can be interpreted as logits (i.e. after feeding them to a sigmoid transform they become probabilities).
The 0 value can be used as a threshold, as it corresponds to the 50% predicted toxicity probability.

Classifier Quick Start Copy Chapter

This introduces the MuTox speech toxicity model, this relies on computing the
sonar embedding and then classifying it through the MuTox model. The
cli/mutox/mutox.py provides an example of reading a TSV, computing the SONAR
embedding and running the classifier on the results:

python -m seamless_communication.cli.toxicity.mutox.mutox_speech --lang fra --audio_column ref_tgt_audio /checkpoint/bokai/seamless/toxity_mitigation/exps_v5/joined_etox/fleurs/s2t/en-xx/fra.tsv /tmp/tesmortt.tsv

You can also work with text:

python -m seamless_communication.cli.toxicity.mutox.mutox_text --lang fra_Latn sentences.txt

You can also check the mutox example notebook in this directory.

Dataset Copy Chapter

The dataset is available in this tsv file. The dataset is licensed under the MIT license (see MIT_LICENSE
file at the root of seamless_communication).

The columns of the dataset are:

• id: a string id of the segment;
• lang: 3-letter language code;
• partition: one of train, dev, or devtest;
• public_url_segment: a string formatted as url:start:end, where start and end are indicated in milliseconds;
• audio_file_transcript: text transctiption of the segment;
• contains_toxicity, toxicity_types, perlocutionary_effects: annotation results as strings (see the paper for their explanation);
• label: an integer label, equal to 1 if contains_toxicity equals Yes and 0 otherwise;
• etox_result: toxic word (or multiple words, separated by |) detected by the Etox matcher;
• detoxify_score: toxicity probabilities predicted by the Detoxify system (float numbers between 0 and 1);
• mutox_speech_score, mutox_text_score, mutox_zero_shot_speech_score, mutox_zero_shot_text_score: MuTox predictions as float numbers with any value (they can be interpreted as logits, i.e. probabilities before a sigmoid transformation).

Citation Copy Chapter

@misc{costajussà2023mutox,
      title={MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector},
      author={ Marta R. Costa-jussà, Mariano Coria Meglioli, Pierre Andrews, David Dale, Prangthip Hansanti, Elahe Kalbassi, Alex Mourachko, Christophe Ropers, Carleigh Wood},
      year={2023},
      eprint={},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Overview (`src/seamless_communication/cli/toxicity/mutox_group_annotations/README.md`) Copy Section

On the Role of Speech Data in Reducing Toxicity Detection Bias

This dataset comprises additional annotations for the English and Spanish samples in the
MuTox test partition. To facilitate systematic evaluations of potential biases of speech-based
toxicity detection systems, 1954 samples have been annotated for mentions of demographic groups.
Annotators also corrected automated transcripts and adjusted judgments of toxicity where appropriate.

These annotations form the basis of our paper comparing the performance and biases of speech- and text-based toxicity
detection systems, available now on ArXiv.

License Copy Chapter

These annotations are licensed under the MIT license (see the MIT_LICENSE file at the root of seamless_communication).

Annotations Copy Chapter

• Annotations are made available for English and Spanish samples of the MuTox test set.
• Annotations were produced by three annotators per language using an iterative, multi-stage process of annotation, review, and discussion.
• Annotators marked whether a category of group was mentioned, such as a gender identity, racial or ethic group, etc.
• For each group category, annotators specified which specific demographic groups were mentioned. This was an open-ended free-text annotation.
• Group and group category annotations refer only to which groups are mentioned or referred to, and do not refer to the identity of the speaker.
• Annotators also provided a new toxicity annotation, taking values Yes, No, Cannot say and No consensus.
• Finally, annotators marked whether the original ASR-produced transcript was correct, and if not, corrected it themselves.

Group categories Copy Section

Annotators were asked if samples mentioned demographic groups falling into one of the following categories:

• Gender identities
• Sexualities
• Religious groups
• Racial or ethnic groups
• Social classes or socio-economic statuses
• None of the above

Samples where annotators could not agree are marked as No consensus.

Using the annotations Copy Chapter

The annotations are available in this TSV file.
Annotations can be joined with the original MuTox samples using the id column.

The columns are:

• lang specifies the language
• transcript_is_correct is whether the ASR transcript provided in the original MuTox dataset is correct
• transcript_corrected is the annotator-corrected transcript
• contained_toxicity_corrected is the annotator-corrected toxicity judgment
• group_categories is a list of categories of demographic groups mentioned in the sample, separated by '|', e.g. "Gender identities|Racial or ethnic groups"
• groups is a list of groups mentioned, separated by '|', e.g. "female, woman or girl|transgender"

Citation Copy Chapter

@misc{bell2024,
      title={On the Role of Speech Data in Reducing Toxicity Detection Bias}
      author={Samuel J. Bell, Mariano Coria Meglioli, Megan Richards, Eduardo Sánchez, Christophe Ropers, Skyler Wang, Adina Williams, Levent Sagun, Marta R. Costa-jussà},
      year={2024},
      eprint={2411.08135},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

--- METRICS ---

• Files Extracted: 19
• Estimated Token Budget: ~34525 tokens
• Canonical Reference: https://codewiki.google/github.com/facebookresearch/seamless_communication