audiocraft Documentation

Technical Documentation: facebookresearch/audiocraft

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

AudioCraft

AudioCraft is a PyTorch library for deep learning research on audio generation. AudioCraft contains inference and training code
for two state-of-the-art AI generative models producing high-quality audio: AudioGen and MusicGen.

Installation

AudioCraft requires Python 3.9, PyTorch 2.1.0. To install AudioCraft, you can run the following:

shell

# Best to make sure you have torch installed first, in particular before installing xformers.
# Don't run this if you already have PyTorch installed.
python -m pip install 'torch==2.1.0'
# You might need the following before trying to install the packages
python -m pip install setuptools wheel
# Then proceed to one of the following
python -m pip install -U audiocraft  # stable release
python -m pip install -U git+https://[email protected]/facebookresearch/audiocraft#egg=audiocraft  # bleeding edge
python -m pip install -e .  # or if you cloned the repo locally (mandatory if you want to train).
python -m pip install -e '.[wm]'  # if you want to train a watermarking model

We also recommend having ffmpeg installed, either through your system or Anaconda:

bash

sudo apt-get install ffmpeg
# Or if you are using Anaconda or Miniconda
conda install "ffmpeg<5" -c conda-forge

Models

At the moment, AudioCraft contains the training code and inference code for:

MusicGen: A state-of-the-art controllable text-to-music model.
AudioGen: A state-of-the-art text-to-sound model.
EnCodec: A state-of-the-art high fidelity neural audio codec.
Multi Band Diffusion: An EnCodec compatible decoder using diffusion.
MAGNeT: A state-of-the-art non-autoregressive model for text-to-music and text-to-sound.
AudioSeal: A state-of-the-art audio watermarking.
MusicGen Style: A state-of-the-art text-and-style-to-music model.
JASCO: "High quality text-to-music model conditioned on chords, melodies and drum tracks"

Training code

AudioCraft contains PyTorch components for deep learning research in audio and training pipelines for the developed models.
For a general introduction of AudioCraft design principles and instructions to develop your own training pipeline, refer to
the AudioCraft training documentation.

For reproducing existing work and using the developed training pipelines, refer to the instructions for each specific model
that provides pointers to configuration, example grids and model/task-specific information and FAQ.

API documentation

We provide some API documentation for AudioCraft.

FAQ

Is the training code available?

Yes! We provide the training code for EnCodec, MusicGen,Multi Band Diffusion and JASCO.

Where are the models stored?

Hugging Face stored the model in a specific location, which can be overridden by setting the AUDIOCRAFT_CACHE_DIR environment variable for the AudioCraft models.
In order to change the cache location of the other Hugging Face models, please check out the Hugging Face Transformers documentation for the cache setup.
Finally, if you use a model that relies on Demucs (e.g. musicgen-melody) and want to change the download location for Demucs, refer to the Torch Hub documentation.

License

The code in this repository is released under the MIT license as found in the LICENSE file.
The models weights in this repository are released under the CC-BY-NC 4.0 license as found in the LICENSE_weights file.

Citation

For the general framework of AudioCraft, please cite the following.

text

@inproceedings{copet2023simple,
    title={Simple and Controllable Music Generation},
    author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
    year={2023},
}

When referring to a specific model, please cite as mentioned in the model specific README, e.g
./docs/MUSICGEN.md, ./docs/AUDIOGEN.md, etc.

AUDIOGEN (`docs/AUDIOGEN.md`)

AudioGen: Textually-guided audio generation

AudioCraft provides the code and a model re-implementing AudioGen, a textually-guided audio generation
model that performs text-to-sound generation.

The provided AudioGen reimplementation follows the LM model architecture introduced in MusicGen
and is a single stage auto-regressive Transformer model trained over a 16kHz
EnCodec tokenizer with 4 codebooks sampled at 50 Hz.
This model variant reaches similar audio quality than the original implementation introduced in the AudioGen publication
while providing faster generation speed given the smaller frame rate.

Important note: The provided models are NOT the original models used to report numbers in the
AudioGen publication. Refer to the model card to learn more about architectural changes.

Listen to samples from the original AudioGen implementation in our sample page.

Model Card

See the model card.

Installation

Please follow the AudioCraft installation instructions from the README.

AudioCraft requires a GPU with at least 16 GB of memory for running inference with the medium-sized models (~1.5B parameters).

API and usage

We provide a simple API and 1 pre-trained models for AudioGen:

facebook/audiogen-medium: 1.5B model, text to sound - 🤗 Hub

You can play with AudioGen by running the jupyter notebook at demos/audiogen_demo.ipynb locally (if you have a GPU).

See after a quick example for using the API.

python

import torchaudio
from audiocraft.models import AudioGen
from audiocraft.data.audio import audio_write

model = AudioGen.get_pretrained('facebook/audiogen-medium')
model.set_generation_params(duration=5)  # generate 5 seconds.
descriptions = ['dog barking', 'sirene of an emergency vehicle', 'footsteps in a corridor']
wav = model.generate(descriptions)  # generates 3 samples.

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)

Training

The AudioGenSolver implements the AudioGen's training pipeline
used to develop the released model. Note that this may not fully reproduce the results presented in the paper.
Similarly to MusicGen, it defines an autoregressive language modeling task over multiple streams of
discrete tokens extracted from a pre-trained EnCodec model (see EnCodec documentation
for more details on how to train such model) with dataset-specific changes for environmental sound
processing.

Note that we do NOT provide any of the datasets used for training AudioGen.

Example configurations and grids

We provide configurations to reproduce the released models and our research.
AudioGen solvers configuration are available in config/solver/audiogen.
The base training configuration used for the released models is the following:
solver=audiogen/audiogen_base_16khz

Please find some example grids to train AudioGen at
audiocraft/grids/audiogen.

shell

# text-to-sound
dora grid audiogen.audiogen_base_16khz

Sound dataset and metadata

AudioGen's underlying dataset is an AudioDataset augmented with description metadata.
The AudioGen dataset implementation expects the metadata to be available as .json files
at the same location as the audio files or through specified external folder.
Learn more in the datasets section.

Evaluation stage

By default, evaluation stage is also computing the cross-entropy and the perplexity over the
evaluation dataset. Indeed the objective metrics used for evaluation can be costly to run
or require some extra dependencies. Please refer to the metrics documentation
for more details on the requirements for each metric.

We provide an off-the-shelf configuration to enable running the objective metrics
for audio generation in
config/solver/audiogen/evaluation/objective_eval.

One can then activate evaluation the following way:

shell

# using the configuration
dora run solver=audiogen/debug solver/audiogen/evaluation=objective_eval
# specifying each of the fields, e.g. to activate KL computation
dora run solver=audiogen/debug evaluate.metrics.kld=true

See an example evaluation grid.

Generation stage

The generation stage allows to generate samples conditionally and/or unconditionally and to perform
audio continuation (from a prompt). We currently support greedy sampling (argmax), sampling
from softmax with a given temperature, top-K and top-P (nucleus) sampling. The number of samples
generated and the batch size used are controlled by the dataset.generate configuration
while the other generation parameters are defined in generate.lm.

shell

# control sampling parameters
dora run solver=audiogen/debug generate.lm.gen_duration=5 generate.lm.use_sampling=true generate.lm.top_k=15

More information

Refer to MusicGen's instructions.

Learn more

Learn more about AudioCraft training pipelines in the dedicated section.

Citation

AudioGen

text

@article{kreuk2022audiogen,
    title={Audiogen: Textually guided audio generation},
    author={Kreuk, Felix and Synnaeve, Gabriel and Polyak, Adam and Singer, Uriel and D{\'e}fossez, Alexandre and Copet, Jade and Parikh, Devi and Taigman, Yaniv and Adi, Yossi},
    journal={arXiv preprint arXiv:2209.15352},
    year={2022}
}

MusicGen

text

@article{copet2023simple,
    title={Simple and Controllable Music Generation},
    author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},
    year={2023},
    journal={arXiv preprint arXiv:2306.05284},
}

License

See license information in the model card.

CONDITIONING (`docs/CONDITIONING.md`)

AudioCraft conditioning modules

AudioCraft provides a
modular implementation of conditioning modules
that can be used with the language model to condition the generation.
The codebase was developed in order to easily extend the set of modules
currently supported to easily develop new ways of controlling the generation.

Conditioning methods

For now, we support 3 main types of conditioning within AudioCraft:

Text-based conditioning methods
Waveform-based conditioning methods
Joint embedding conditioning methods for text and audio projected in a shared latent space.

The Language Model relies on 2 core components that handle processing information:

The ConditionProvider class, that maps metadata to processed conditions, leveraging
all the defined conditioners for the given task.
The ConditionFuser class, that takes preprocessed conditions and properly fuse the
conditioning embedding to the language model inputs following a given fusing strategy.

Different conditioners (for text, waveform, joint embeddings...) are provided as torch
modules in AudioCraft and are used internally in the language model to process the
conditioning signals and feed them to the language model.

Core concepts

Conditioners

The BaseConditioner torch module is the base implementation for all conditioners in AudioCraft.

Each conditioner is expected to implement 2 methods:

The tokenize method that is used as a preprocessing method that contains all processing
that can lead to synchronization points (e.g. BPE tokenization with transfer to the GPU).
The output of the tokenize method will then be used to feed the forward method.
The forward method that takes the output of the tokenize method and contains the core computation
to obtain the conditioning embedding along with a mask indicating valid indices (e.g. padding tokens).

ConditionProvider

The ConditionProvider prepares and provides conditions given a dictionary of conditioners.

Conditioners are specified as a dictionary of attributes and the corresponding conditioner
providing the processing logic for the given attribute.

Similarly to the conditioners, the condition provider works in two steps to avoid synchronization points:

A tokenize method that takes a list of conditioning attributes for the batch,
and runs all tokenize steps for the set of conditioners.
A forward method that takes the output of the tokenize step and runs all the forward steps
for the set of conditioners.

The list of conditioning attributes is passed as a list of ConditioningAttributes
that is presented just below.

ConditionFuser

Once all conditioning signals have been extracted and processed by the ConditionProvider
as dense embeddings, they remain to be passed to the language model along with the original
language model inputs.

The ConditionFuser handles specifically the logic to combine the different conditions
to the actual model input, supporting different strategies to combine them.

One can therefore define different strategies to combine or fuse the condition to the input, in particular:

Prepending the conditioning signal to the input with the prepend strategy,
Summing the conditioning signal to the input with the sum strategy,
Combining the conditioning relying on a cross-attention mechanism with the cross strategy,
Using input interpolation with the input_interpolate strategy.

SegmentWithAttributes and ConditioningAttributes: From metadata to conditions

The ConditioningAttributes dataclass is the base class for metadata
containing all attributes used for conditioning the language model.

It currently supports the following types of attributes:

Text conditioning attributes: Dictionary of textual attributes used for text-conditioning.
Wav conditioning attributes: Dictionary of waveform attributes used for waveform-based
conditioning such as the chroma conditioning.
JointEmbed conditioning attributes: Dictionary of text and waveform attributes
that are expected to be represented in a shared latent space.

These different types of attributes are the attributes that are processed
by the different conditioners.

ConditioningAttributes are extracted from metadata loaded along the audio in the datasets,
provided that the metadata used by the dataset implements the SegmentWithAttributes abstraction.

All metadata-enabled datasets to use for conditioning in AudioCraft inherits
the audiocraft.data.info_dataset.InfoAudioDataset class
and the corresponding metadata inherits and implements the SegmentWithAttributes abstraction.
Refer to the audiocraft.data.music_dataset.MusicAudioDataset
class as an example.

Available conditioners

Text conditioners

All text conditioners are expected to inherit from the TextConditioner class.

AudioCraft currently provides two text conditioners:

The LUTConditioner that relies on look-up-table of embeddings learned at train time,
and relying on either no tokenizer or a spacy tokenizer. This conditioner is particularly
useful for simple experiments and categorical labels.
The T5Conditioner that relies on a
pre-trained T5 model
frozen or fine-tuned at train time to extract the text embeddings.

Waveform conditioners

All waveform conditioners are expected to inherit from the WaveformConditioner class and
consist of a conditioning method that takes a waveform as input. The waveform conditioner
must implement the logic to extract the embedding from the waveform and define the downsampling
factor from the waveform to the resulting embedding.

The ChromaStemConditioner conditioner is a waveform conditioner for the chroma features
conditioning used by MusicGen. It takes a given waveform, extracts relevant stems for melody
(namely all non drums and bass stems) using a
pre-trained Demucs model
and then extracts the chromagram bins from the remaining mix of stems.

Joint embeddings conditioners

We finally provide support for conditioning based on joint text and audio embeddings through
the JointEmbeddingConditioner class and the CLAPEmbeddingConditioner that implements such
a conditioning method relying on a pretrained CLAP model.

Classifier Free Guidance

We provide a Classifier Free Guidance implementation in AudioCraft. With the classifier free
guidance dropout, all attributes are dropped with the same probability.

Attribute Dropout

We further provide an attribute dropout strategy. Unlike the classifier free guidance dropout,
the attribute dropout drops given attributes with a defined probability, allowing the model
not to expect all conditioning signals to be provided at once.

Faster computation of conditions

Conditioners that require some heavy computation on the waveform can be cached, in particular
the ChromaStemConditioner or CLAPEmbeddingConditioner. You just need to provide the
cache_path parameter to them. We recommend running dummy jobs for filling up the cache quickly.
An example is provided in the musicgen.musicgen_melody_32khz grid.

DATASETS (`docs/DATASETS.md`)

AudioCraft datasets

Our dataset manifest files consist in 1-json-per-line files, potentially gzipped,
as data.jsons or data.jsons.gz files. This JSON contains the path to the audio
file and associated metadata. The manifest files are then provided in the configuration,
as datasource sub-configuration. A datasource contains the pointers to the paths of
the manifest files for each AudioCraft stage (or split) along with additional information
(eg. maximum sample rate to use against this dataset). All the datasources are under the
dset group config, with a dedicated configuration file for each dataset.

Getting started

Example

See the provided example in the directory that provides a manifest to use the example dataset
provided under the dataset folder.

The manifest files are stored in the egs folder.

shell

egs/
  example/data.json.gz

A datasource is defined in the configuration folder, in the dset group config for this dataset
at config/dset/audio/example:

shell

# @package __global__

datasource:
  max_sample_rate: 44100
  max_channels: 2

  train: egs/example
  valid: egs/example
  evaluate: egs/example
  generate: egs/example

For proper dataset, one should create manifest for each of the splits and specify the correct path
to the given manifest in the datasource for each split.

Then, using a dataset through the configuration can be done pointing to the
corresponding dataset configuration:

shell

dset=<dataset_name> # <dataset_name> should match the yaml file name

# for example
dset=audio/example

Creating manifest files

Assuming you want to create manifest files to load with AudioCraft's AudioDataset, you can use
the following command to create new manifest files from a given folder containing audio files:

shell

python -m audiocraft.data.audio_dataset <path_to_dataset_folder> egs/my_dataset/my_dataset_split/data.jsonl.gz

# For example to generate the manifest for dset=audio/example
# note: we don't use any split and we don't compress the jsonl file for this dummy example
python -m audiocraft.data.audio_dataset dataset/example egs/example/data.jsonl

# More info with: python -m audiocraft.data.audio_dataset --help

Additional information

MusicDataset and metadata

The MusicDataset is an AudioDataset with additional metadata. The MusicDataset expects
the additional metadata to be stored in a JSON file that has the same path as the corresponding
audio file, but with a .json extension.

SoundDataset and metadata

The SoundDataset is an AudioDataset with descriptions metadata. Similarly to the MusicDataset,
the SoundDataset expects the additional metadata to be stored in a JSON file that has the same
path as the corresponding audio file, but with a .json extension. Additionally, the SoundDataset
supports an additional parameter pointing to an extra folder external_metadata_source containing
all the JSON metadata files given they have the same filename as the audio file.

ENCODEC (`docs/ENCODEC.md`)

EnCodec: High Fidelity Neural Audio Compression

AudioCraft provides the training code for EnCodec, a state-of-the-art deep learning
based audio codec supporting both mono and stereo audio, presented in the
High Fidelity Neural Audio Compression paper.
Check out our sample page.

Original EnCodec models

The EnCodec models presented in High Fidelity Neural Audio Compression can be accessed
and used with the EnCodec repository.

Note: We do not guarantee compatibility between the AudioCraft and EnCodec codebases
and released checkpoints at this stage.

Installation

Please follow the AudioCraft installation instructions from the README.

Training

The CompressionSolver implements the audio reconstruction
task to train an EnCodec model. Specifically, it trains an encoder-decoder with a quantization
bottleneck - a SEANet encoder-decoder with Residual Vector Quantization bottleneck for EnCodec -
using a combination of objective and perceptual losses in the forms of discriminators.

The default configuration matches a causal EnCodec training at a single bandwidth.

Example configuration and grids

We provide sample configuration and grids for training EnCodec models.

The compression configuration are defined in
config/solver/compression.

The example grids are available at
audiocraft/grids/compression.

shell

# base causal encodec on monophonic audio sampled at 24 khz
dora grid compression.encodec_base_24khz
# encodec model used for MusicGen on monophonic audio sampled at 32 khz
dora grid compression.encodec_musicgen_32khz

Training and validation stages

The model is trained using a combination of objective and perceptual losses.
More specifically, EnCodec is trained with the MS-STFT discriminator along with
objective losses through the use of a loss balancer to effectively weight
the different losses, in an intuitive manner.

Evaluation stage

Evaluation metrics for audio generation:

SI-SNR: Scale-Invariant Signal-to-Noise Ratio.
ViSQOL: Virtual Speech Quality Objective Listener.

Note: Path to the ViSQOL binary (compiled with bazel) needs to be provided in
order to run the ViSQOL metric on the reference and degraded signals.
The metric is disabled by default.
Please refer to the metrics documentation to learn more.

Generation stage

The generation stage consists in generating the reconstructed audio from samples
with the current model. The number of samples generated and the batch size used are
controlled by the dataset.generate configuration. The output path and audio formats
are defined in the generate stage configuration.

shell

# generate samples every 5 epoch
dora run solver=compression/encodec_base_24khz generate.every=5
# run with a different dset
dora run solver=compression/encodec_base_24khz generate.path=<PATH_IN_DORA_XP_FOLDER>
# limit the number of samples or use a different batch size
dora grid solver=compression/encodec_base_24khz dataset.generate.num_samples=10 dataset.generate.batch_size=4

Playing with the model

Once you have a model trained, it is possible to get the entire solver, or just
the trained model with the following functions:

text

/* Detailed source-code truncated for AI context efficiency. */

Importing / Exporting models

At the moment we do not have a definitive workflow for exporting EnCodec models, for
instance to Hugging Face (HF). We are working on supporting automatic conversion between
AudioCraft and Hugging Face implementations.

We still have some support for fine-tuning an EnCodec model coming from HF in AudioCraft,
using for instance continue_from=//pretrained/facebook/encodec_32k.

An AudioCraft checkpoint can be exported in a more compact format (excluding the optimizer etc.)
using audiocraft.utils.export.export_encodec. For instance, you could run

python

from audiocraft.utils import export
from audiocraft import train
xp = train.main.get_xp_from_sig('SIG')
export.export_encodec(
    xp.folder / 'checkpoint.th',
    '/checkpoints/my_audio_lm/compression_state_dict.bin')


from audiocraft.models import CompressionModel
model = CompressionModel.get_pretrained('/checkpoints/my_audio_lm/compression_state_dict.bin')

from audiocraft.solvers import CompressionSolver
# The two are strictly equivalent, but this function supports also loading from non-already exported models.
model = CompressionSolver.model_from_checkpoint('//pretrained//checkpoints/my_audio_lm/compression_state_dict.bin')

We will see then how to use this model as a tokenizer for MusicGen/AudioGen in the
MusicGen documentation.

Learn more

Learn more about AudioCraft training pipelines in the dedicated section.

Citation

text

@article{defossez2022highfi,
  title={High Fidelity Neural Audio Compression},
  author={Défossez, Alexandre and Copet, Jade and Synnaeve, Gabriel and Adi, Yossi},
  journal={arXiv preprint arXiv:2210.13438},
  year={2022}
}

License

See license information in the README.

JASCO (`docs/JASCO.md`)

JASCO: Joint Audio And Symbolic Conditioning for Temporally Controlled Text-To-Music Generation

AudioCraft provides the code and models for JASCO, Joint Audio And Symbolic Conditioning for Temporally Controlled Text-To-Music Generation.

We present JASCO, a temporally controlled text-to-music generation model utilizing both symbolic and audio-based conditions.
JASCO can generate high-quality music samples conditioned on global text descriptions along with fine-grained local controls.
JASCO is based on the Flow Matching modeling paradigm together with a novel conditioning method, allowing for music generation controlled both locally (e.g., chords) and globally (text description).

Check out our sample page or test the available demo!

We use ~16K hours of licensed music to train JASCO.

Model Card

See the model card.

Installation

First, Please follow the AudioCraft installation instructions from the README.

Then, download and install chord_extractor from source

See further required installation under Data Preprocessing section

Usage

We currently offer two ways to interact with JASCO:

You can use the gradio demo locally by running python -m demos.jasco_app, you can add --share to deploy a sharable space mounted on your device.
You can play with JASCO by running the jupyter notebook at demos/jasco_demo.ipynb locally.

API

We provide a simple API and pre-trained models:

facebook/jasco-chords-drums-400M: 400M model, text to music with chords and drums support, generates 10-second samples - 🤗 Hub
facebook/jasco-chords-drums-1B: 1B model, text to music with chords and drums support, generates 10-second samples - 🤗 Hub
facebook/jasco-chords-drums-melody-400M: 400M model, text to music with chords, drums and melody support, generates 10-second samples - 🤗 Hub
facebook/jasco-chords-drums-melody-1B: 1B model, text to music with chords, drums and melody support, generates 10-second samples - 🤗 Hub

See after a quick example for using the API.

python

from audiocraft.models import JASCO

model = JASCO.get_pretrained('facebook/jasco-chords-drums-400M', chords_mapping_path='../assets/chord_to_index_mapping.pkl')

model.set_generation_params(
    cfg_coef_all=5.0,
    cfg_coef_txt=0.0
)

# set textual prompt
text = "Strings, woodwind, orchestral, symphony."

# define chord progression
chords = [('C', 0.0), ('D', 2.0), ('F', 4.0), ('Ab', 6.0), ('Bb', 7.0), ('C', 8.0)]

# run inference
output = model.generate_music(descriptions=[text], chords=chords, progress=True)

audio_write('output', output.cpu().squeeze(0), model.sample_rate, strategy="loudness", loudness_compressor=True)

For more examples check out demos/jasco_demo.ipynb

🤗 Transformers Usage

Coming soon...

Data Preprocessing

In order to to use the JascoDataset with chords / melody conditioning, please follow the instructions below:

Chords conditioning

To extract chords from your desired data follow the following steps:

Prepare a *.jsonl containing list of absolute file paths in your dataset, should simply be absolute paths seperated by newlines.
Download and install chord_extractor from source
For training purposes run: python scripts/chords/extract_chords.py --src_jsonl_file=<abs path to .jsonl file containing list of absolute file paths seperated by new line> --target_output_dir=<target directory to save parsed chord files to, individual files will be saved inside>

and then run: python scripts/chords/build_chord_map.py --chords_folder=<abs path to directory containing parsed chords files> --output_directory=

For evaluation of our released models run: python scripts/chords/extract_chords.py --src_jsonl_file=<abs path to .jsonl file containing list of absolute file paths seperated by new line> --target_output_dir=<target directory to save parsed chord files to, individual files will be saved inside> --path_to_pre_defined_map=<abs path to pre-defined mapping file>

and then run: python scripts/chords/build_chord_map.py --chords_folder=<abs path to directory containing parsed chords files> --output_directory= --path_to_pre_defined_map=<for evaluation purpose, use pre-defined chord-to-index map absolute path>

NOTE: current scripts assume that all audio files are of .wav format, some changes may be required if your data consists of other formats.

NOTE: predefined chord mapping file is available in assets directory.

Melody conditioning

This section relies on Deepsalience repo with slight custom scripts written.

Clone repo and create virtual environment

git clone [email protected]:lonzi/ismir2017-deepsalience.git forked_deepsalience_repo
cd forked_deepsalience_repo
conda create --name deep_salience python=3.7
conda activate deep_salience
pip install -r requirements.txt

Salience map dumps (of entire directory, using slurm job)

From src dir

create job array: python predict/create_predict_saliency_cmds.py --src_dir= --out_dir= --n_shards=<desired number of shards> --multithread
run job array: sbatch predict_saliency.sh

From track list

create job array: python predict/create_predict_saliency_cmds.py --tracks_list=tracks_train.txt --out_dir= --n_shards=2 --multithread --sbatch_script_name=predict_saliency_train.sh --saliency_threshold=
run job array: sbatch predict_saliency_train.sh

tracks_train.txt: a list of track paths to process seperated by new lines

Training

The JascoSolver implements JASCO's training pipeline.
conditional flow matching objective over the continuous extracted latents from a pre-trained EnCodec model (see EnCodec documentation
for more details on how to train such model).

Note that we do NOT provide any of the datasets used for training JASCO.
We provide a dummy dataset containing just a few examples for illustrative purposes.

Please read first the TRAINING documentation, in particular the Environment Setup section.

Fine tuning existing models

You can initialize your model to one of the pretrained models by using the continue_from argument, in particular

bash

# Using pretrained JASCO model.
dora run solver=jasco/chords_drums model/lm/model_scale=small continue_from=//pretrained/facebook/jasco-chords-drums-400M conditioner=jasco_chords_drums

# Using another model you already trained with a Dora signature SIG.
dora run solver=jasco/chords_drums model/lm/model_scale=small continue_from=//sig/SIG conditioner=jasco_chords_drums

# Or providing manually a path
dora run solver=jasco/chords_drums model/lm/model_scale=small conditioner=jasco_chords_drums continue_from=/checkpoints/my_other_xp/checkpoint.th

Warning: You are responsible for selecting the other parameters accordingly, in a way that make it compatible
with the model you are fine tuning. Configuration is NOT automatically inherited from the model you continue from. In particular make sure to select the proper conditioner and model/lm/model_scale.

Warning: We currently do not support fine tuning a model with slightly different layers. If you decide
to change some parts, like the conditioning or some other parts of the model, you are responsible for manually crafting a checkpoint file from which we can safely run load_state_dict.
If you decide to do so, make sure your checkpoint is saved with torch.save and contains a dict
{'best_state': {'model': model_state_dict_here}}. Directly give the path to continue_from without a //pretrained/ prefix.

Evaluation & Generation stage

See MusicGen

Playing with the model

Once you have launched some experiments, you can easily get access
to the Solver with the latest trained model using the following snippet.

python

from audiocraft.solvers.jasco import JascoSolver

solver = JascoSolver.get_eval_solver_from_sig('SIG', device='cpu', batch_size=8)
solver.model
solver.dataloaders

Importing / Exporting models

We do not support currently loading a model from the Hugging Face implementation or exporting to it.
If you want to export your model in a way that is compatible with audiocraft.models.JASCO
API, you can run:

python

from audiocraft.utils import export
from audiocraft import train
xp = train.main.get_xp_from_sig('SIG_OF_LM')
export.export_lm(xp.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/state_dict.bin')
# You also need to bundle the EnCodec model you used !!
## Case 1) you trained your own
xp_encodec = train.main.get_xp_from_sig('SIG_OF_ENCODEC')
export.export_encodec(xp_encodec.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/compression_state_dict.bin')
## Case 2) you used a pretrained model. Give the name you used without the //pretrained/ prefix.
## This will actually not dump the actual model, simply a pointer to the right model to download.
export.export_pretrained_compression_model('facebook/encodec_32khz', '/checkpoints/my_audio_lm/compression_state_dict.bin')

Now you can load your custom model with:

python

import audiocraft.models
jasco = audiocraft.models.JASCO.get_pretrained('/checkpoints/my_audio_lm/')

Learn more

Learn more about AudioCraft training pipelines in the dedicated section.

Citation

text

@misc{tal2024joint,
    title={Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation}, 
    author={Or Tal and Alon Ziv and Itai Gat and Felix Kreuk and Yossi Adi},
    year={2024},
    eprint={2406.10970},
    archivePrefix={arXiv},
    primaryClass={cs.SD}
}

License

See license information in the model card.

MAGNET (`docs/MAGNET.md`)

MAGNeT: Masked Audio Generation using a Single Non-Autoregressive Transformer

AudioCraft provides the code and models for MAGNeT, Masked Audio Generation using a Single Non-Autoregressive Transformer.

MAGNeT is a text-to-music and text-to-sound model capable of generating high-quality audio samples conditioned on text descriptions.
It is a masked generative non-autoregressive Transformer trained over a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50 Hz.
Unlike prior work on masked generative audio Transformers, such as SoundStorm and VampNet,
MAGNeT doesn't require semantic token conditioning, model cascading or audio prompting, and employs a full text-to-audio using a single non-autoregressive Transformer.

Check out our sample page or test the available demo!

We use 16K hours of licensed music to train MAGNeT. Specifically, we rely on an internal dataset
of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data.

Model Card

See the model card.

Installation

Please follow the AudioCraft installation instructions from the README.

AudioCraft requires a GPU with at least 16 GB of memory for running inference with the medium-sized models (~1.5B parameters).

Usage

We currently offer two ways to interact with MAGNeT:

You can use the gradio demo locally by running python -m demos.magnet_app --share.
You can play with MAGNeT by running the jupyter notebook at demos/magnet_demo.ipynb locally (if you have a GPU).

API

We provide a simple API and 6 pre-trained models. The pre trained models are:

facebook/magnet-small-10secs: 300M model, text to music, generates 10-second samples - 🤗 Hub
facebook/magnet-medium-10secs: 1.5B model, text to music, generates 10-second samples - 🤗 Hub
facebook/magnet-small-30secs: 300M model, text to music, generates 30-second samples - 🤗 Hub
facebook/magnet-medium-30secs: 1.5B model, text to music, generates 30-second samples - 🤗 Hub
facebook/audio-magnet-small: 300M model, text to sound-effect - 🤗 Hub
facebook/audio-magnet-medium: 1.5B model, text to sound-effect - 🤗 Hub

In order to use MAGNeT locally you must have a GPU. We recommend 16GB of memory, especially for
the medium size models.

See after a quick example for using the API.

python

import torchaudio
from audiocraft.models import MAGNeT
from audiocraft.data.audio import audio_write

model = MAGNeT.get_pretrained('facebook/magnet-small-10secs')
descriptions = ['disco beat', 'energetic EDM', 'funky groove']
wav = model.generate(descriptions)  # generates 3 samples.

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)

🤗 Transformers Usage

Coming soon...

Training

The MagnetSolver implements MAGNeT's training pipeline.
It defines a masked generation task over multiple streams of discrete tokens
extracted from a pre-trained EnCodec model (see EnCodec documentation
for more details on how to train such model).

Note that we do NOT provide any of the datasets used for training MAGNeT.
We provide a dummy dataset containing just a few examples for illustrative purposes.

Please read first the TRAINING documentation, in particular the Environment Setup section.

Example configurations and grids

We provide configurations to reproduce the released models and our research.
MAGNeT solvers configuration are available in config/solver/magnet,
in particular:

MAGNeT model for text-to-music:
solver=magnet/magnet_32khz
MAGNeT model for text-to-sound:
solver=magnet/audio_magnet_16khz

We provide 3 different scales, e.g. model/lm/model_scale=small (300M), or medium (1.5B), and large (3.3B).

Please find some example grids to train MAGNeT at
audiocraft/grids/magnet.

shell

# text-to-music
dora grid magnet.magnet_32khz --dry_run --init

# text-to-sound
dora grid magnet.audio_magnet_16khz --dry_run --init

# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.

dataset and metadata

Learn more in the datasets section.

Music Models

MAGNeT's underlying dataset is an AudioDataset augmented with music-specific metadata.
The MAGNeT dataset implementation expects the metadata to be available as .json files
at the same location as the audio files.

Sound Models

Audio-MAGNeT's underlying dataset is an AudioDataset augmented with description metadata.
The Audio-MAGNeT dataset implementation expects the metadata to be available as .json files
at the same location as the audio files or through specified external folder.

Audio tokenizers

See MusicGen

Fine tuning existing models

You can initialize your model to one of the pretrained models by using the continue_from argument, in particular

bash

# Using pretrained MAGNeT model.
dora run solver=magnet/magnet_32khz model/lm/model_scale=medium continue_from=//pretrained/facebook/magnet-medium-10secs conditioner=text2music

# Using another model you already trained with a Dora signature SIG.
dora run solver=magnet/magnet_32khz model/lm/model_scale=medium continue_from=//sig/SIG conditioner=text2music

# Or providing manually a path
dora run solver=magnet/magnet_32khz model/lm/model_scale=medium continue_from=/checkpoints/my_other_xp/checkpoint.th

Evaluation stage

For the 6 pretrained MAGNeT models, objective metrics could be reproduced using the following grids:

shell

# text-to-music
REGEN=1 dora grid magnet.magnet_pretrained_32khz_eval --dry_run --init

# text-to-sound
REGEN=1 dora grid magnet.audio_magnet_pretrained_16khz_eval --dry_run --init

# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.

See MusicGen for more details.

Generation stage

See MusicGen

Playing with the model

Once you have launched some experiments, you can easily get access
to the Solver with the latest trained model using the following snippet.

python

from audiocraft.solvers.magnet import MagnetSolver

solver = MagnetSolver.get_eval_solver_from_sig('SIG', device='cpu', batch_size=8)
solver.model
solver.dataloaders

Importing / Exporting models

python

from audiocraft.utils import export
from audiocraft import train
xp = train.main.get_xp_from_sig('SIG_OF_LM')
export.export_lm(xp.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/state_dict.bin')
# You also need to bundle the EnCodec model you used !!
## Case 1) you trained your own
xp_encodec = train.main.get_xp_from_sig('SIG_OF_ENCODEC')
export.export_encodec(xp_encodec.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/compression_state_dict.bin')
## Case 2) you used a pretrained model. Give the name you used without the //pretrained/ prefix.
## This will actually not dump the actual model, simply a pointer to the right model to download.
export.export_pretrained_compression_model('facebook/encodec_32khz', '/checkpoints/my_audio_lm/compression_state_dict.bin')

Now you can load your custom model with:

python

import audiocraft.models
magnet = audiocraft.models.MAGNeT.get_pretrained('/checkpoints/my_audio_lm/')

Learn more

Learn more about AudioCraft training pipelines in the dedicated section.

FAQ

What are top-k, top-p, temperature and classifier-free guidance?

Check out @FurkanGozukara tutorial.

Should I use FSDP or autocast ?

The two are mutually exclusive (because FSDP does autocast on its own).
You can use autocast up to 1.5B (medium), if you have enough RAM on your GPU.
FSDP makes everything more complex but will free up some memory for the actual
activations by sharding the optimizer state.

Citation

text

@misc{ziv2024masked,
      title={Masked Audio Generation using a Single Non-Autoregressive Transformer}, 
      author={Alon Ziv and Itai Gat and Gael Le Lan and Tal Remez and Felix Kreuk and Alexandre Défossez and Jade Copet and Gabriel Synnaeve and Yossi Adi},
      year={2024},
      eprint={2401.04577},
      archivePrefix={arXiv},
      primaryClass={cs.SD}
}

License

See license information in the model card.

MBD (`docs/MBD.md`)

MultiBand Diffusion

AudioCraft provides the code and models for MultiBand Diffusion, From Discrete Tokens to High Fidelity Audio using MultiBand Diffusion.
MultiBand diffusion is a collection of 4 models that can decode tokens from
EnCodec tokenizer into waveform audio. You can listen to some examples on the sample page.

Installation

Please follow the AudioCraft installation instructions from the README.

Usage

We offer a number of way to use MultiBand Diffusion:

The MusicGen demo includes a toggle to try diffusion decoder. You can use the demo locally by running python -m demos.musicgen_app --share, or through the MusicGen Colab.
You can play with MusicGen by running the jupyter notebook at demos/musicgen_demo.ipynb locally (if you have a GPU).

API

We provide a simple API and pre-trained models for MusicGen and for EnCodec at 24 khz for 3 bitrates (1.5 kbps, 3 kbps and 6 kbps).

See after a quick example for using MultiBandDiffusion with the MusicGen API:

python

import torchaudio
from audiocraft.models import MusicGen, MultiBandDiffusion
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('facebook/musicgen-melody')
mbd = MultiBandDiffusion.get_mbd_musicgen()
model.set_generation_params(duration=8)  # generate 8 seconds.
wav, tokens = model.generate_unconditional(4, return_tokens=True)    # generates 4 unconditional audio samples and keep the tokens for MBD generation
descriptions = ['happy rock', 'energetic EDM', 'sad jazz']
wav_diffusion = mbd.tokens_to_wav(tokens)
wav, tokens = model.generate(descriptions, return_tokens=True)  # generates 3 samples and keep the tokens.
wav_diffusion = mbd.tokens_to_wav(tokens)
melody, sr = torchaudio.load('./assets/bach.mp3')
# Generates using the melody from the given audio and the provided descriptions, returns audio and audio tokens.
wav, tokens = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr, return_tokens=True)
wav_diffusion = mbd.tokens_to_wav(tokens)

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav and {idx}_diffusion.wav, with loudness normalization at -14 db LUFS for comparing the methods.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
    audio_write(f'{idx}_diffusion', wav_diffusion[idx].cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)

For the compression task (and to compare with EnCodec):

python

import torch
from audiocraft.models import MultiBandDiffusion
from encodec import EncodecModel
from audiocraft.data.audio import audio_read, audio_write

bandwidth = 3.0  # 1.5, 3.0, 6.0
mbd = MultiBandDiffusion.get_mbd_24khz(bw=bandwidth)
encodec = EncodecModel.encodec_model_24khz()

somepath = ''
wav, sr = audio_read(somepath)
with torch.no_grad():
    compressed_encodec = encodec(wav)
    compressed_diffusion = mbd.regenerate(wav, sample_rate=sr)

audio_write('sample_encodec', compressed_encodec.squeeze(0).cpu(), mbd.sample_rate, strategy="loudness", loudness_compressor=True)
audio_write('sample_diffusion', compressed_diffusion.squeeze(0).cpu(), mbd.sample_rate, strategy="loudness", loudness_compressor=True)

Training

The DiffusionSolver implements our diffusion training pipeline.
It generates waveform audio conditioned on the embeddings extracted from a pre-trained EnCodec model
(see EnCodec documentation for more details on how to train such model).

Note that we do NOT provide any of the datasets used for training our diffusion models.
We provide a dummy dataset containing just a few examples for illustrative purposes.

Example configurations and grids

One can train diffusion models as described in the paper by using this dora grid.

shell

# 4 bands MBD trainning
dora grid diffusion.4_bands_base_32khz

Learn more

Learn more about AudioCraft training pipelines in the dedicated section.

Citation

text

@article{sanroman2023fromdi,
  title={From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion},
  author={San Roman, Robin and Adi, Yossi and Deleforge, Antoine and Serizel, Romain and Synnaeve, Gabriel and Défossez, Alexandre},
  journal={arXiv preprint arXiv:},
  year={2023}
}

License

See license information in the README.

METRICS (`docs/METRICS.md`)

AudioCraft objective metrics

In addition to training losses, AudioCraft provides a set of objective metrics
for audio synthesis and audio generation. As these metrics may require
extra dependencies and can be costly to train, they are often disabled by default.
This section provides guidance for setting up and using these metrics in
the AudioCraft training pipelines.

Available metrics

Audio synthesis quality metrics

SI-SNR

We provide an implementation of the Scale-Invariant Signal-to-Noise Ratio in PyTorch.
No specific requirement is needed for this metric. Please activate the metric at the
evaluation stage with the appropriate flag:

Warning: We report the opposite of the SI-SNR, e.g. multiplied by -1. This is due to internal
details where the SI-SNR score can also be used as a training loss function, where lower
values should indicate better reconstruction. Negative values are such expected and a good sign! Those should be again multiplied by -1 before publication :)

shell

dora run <...> evaluate.metrics.sisnr=true

ViSQOL

We provide a Python wrapper around the ViSQOL official implementation
to conveniently run ViSQOL within the training pipelines.

One must specify the path to the ViSQOL installation through the configuration in order
to enable ViSQOL computations in AudioCraft:

shell

# the first parameter is used to activate visqol computation while the second specify
# the path to visqol's library to be used by our python wrapper
dora run <...> evaluate.metrics.visqol=true metrics.visqol.bin=<path_to_visqol>

See an example grid: Compression with ViSQOL

To learn more about ViSQOL and how to build ViSQOL binary using bazel, please refer to the
instructions available in the open source repository.

Audio generation metrics

Frechet Audio Distance

Similarly to ViSQOL, we use a Python wrapper around the Frechet Audio Distance
official implementation
in TensorFlow.

Note that we had to make several changes to the actual code in order to make it work.
Please refer to the FrechetAudioDistanceMetric class documentation
for more details. We do not plan to provide further support in obtaining a working setup for the
Frechet Audio Distance at this stage.

shell

# the first parameter is used to activate FAD metric computation while the second specify
# the path to FAD library to be used by our python wrapper
dora run <...> evaluate.metrics.fad=true metrics.fad.bin=<path_to_google_research_repository>

See an example grid: Evaluation with FAD

Kullback-Leibler Divergence

We provide a PyTorch implementation of the Kullback-Leibler Divergence computed over the probabilities
of the labels obtained by a state-of-the-art audio classifier. We provide our implementation of the KLD
using the PaSST classifier.

In order to use the KLD metric over PaSST, you must install the PaSST library as an extra dependency:

shell

pip install 'git+https://github.com/kkoutini/[email protected]#egg=hear21passt'

Then similarly, you can use the metric activating the corresponding flag:

shell

# one could extend the kld metric with additional audio classifier models that can then be picked through the configuration
dora run <...> evaluate.metrics.kld=true metrics.kld.model=passt

Text consistency

We provide a text-consistency metric, similarly to the MuLan Cycle Consistency from
MusicLM or the CLAP score used in
Make-An-Audio.
More specifically, we provide a PyTorch implementation of a Text consistency metric
relying on a pre-trained Contrastive Language-Audio Pretraining (CLAP).

Please install the CLAP library as an extra dependency prior to using the metric:

shell

pip install laion_clap

Then similarly, you can use the metric activating the corresponding flag:

shell

# one could extend the text consistency metric with additional audio classifier models that can then be picked through the configuration
dora run ... evaluate.metrics.text_consistency=true metrics.text_consistency.model=clap

Note that the text consistency metric based on CLAP will require the CLAP checkpoint to be
provided in the configuration.

Chroma cosine similarity

Finally, as introduced in MusicGen, we provide a Chroma Cosine Similarity metric in PyTorch.
No specific requirement is needed for this metric. Please activate the metric at the
evaluation stage with the appropriate flag:

shell

dora run ... evaluate.metrics.chroma_cosine=true

Comparing against reconstructed audio

For all the above audio generation metrics, we offer the option to compute the metric on the reconstructed audio
fed in EnCodec instead of the generated sample using the flag <metric>.use_gt=true.

Example usage

You will find example of configuration for the different metrics introduced above in:

The musicgen's default solver for all audio generation metrics
The compression's default solver for all audio synthesis metrics

Similarly, we provide different examples in our grids:

MUSICGEN (`docs/MUSICGEN.md`)

MusicGen: Simple and Controllable Music Generation

AudioCraft provides the code and models for MusicGen, a simple and controllable model for music generation.
MusicGen is a single stage auto-regressive Transformer model trained over a 32kHz
EnCodec tokenizer with 4 codebooks sampled at 50 Hz.
Unlike existing methods like MusicLM, MusicGen doesn't require
a self-supervised semantic representation, and it generates all 4 codebooks in one pass. By introducing
a small delay between the codebooks, we show we can predict them in parallel, thus having only 50 auto-regressive
steps per second of audio.
Check out our sample page or test the available demo!

We use 20K hours of licensed music to train MusicGen. Specifically, we rely on an internal dataset
of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data.

Model Card

See the model card.

Installation

Please follow the AudioCraft installation instructions from the README.

AudioCraft requires a GPU with at least 16 GB of memory for running inference with the medium-sized models (~1.5B parameters).

Usage

We offer a number of way to interact with MusicGen:

A demo is also available on the facebook/MusicGen Hugging Face Space
(huge thanks to all the HF team for their support).
You can run the extended demo on a Colab:
colab notebook
You can use the gradio demo locally by running python -m demos.musicgen_app --share.
You can play with MusicGen by running the jupyter notebook at demos/musicgen_demo.ipynb locally (if you have a GPU).
Finally, checkout @camenduru Colab page
which is regularly updated with contributions from @camenduru and the community.

API

We provide a simple API and 10 pre-trained models. The pre trained models are:

facebook/musicgen-small: 300M model, text to music only - 🤗 Hub
facebook/musicgen-medium: 1.5B model, text to music only - 🤗 Hub
facebook/musicgen-melody: 1.5B model, text to music and text+melody to music - 🤗 Hub
facebook/musicgen-large: 3.3B model, text to music only - 🤗 Hub
facebook/musicgen-melody-large: 3.3B model, text to music and text+melody to music - 🤗 Hub
facebook/musicgen-stereo-*: All the previous models fine tuned for stereo generation -
small,
medium,
large,
melody,
melody large.

We observe the best trade-off between quality and compute with the facebook/musicgen-medium or facebook/musicgen-melody model.
In order to use MusicGen locally you must have a GPU. We recommend 16GB of memory, but smaller
GPUs will be able to generate short sequences, or longer sequences with the facebook/musicgen-small model.

See after a quick example for using the API.

python

import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('facebook/musicgen-melody')
model.set_generation_params(duration=8)  # generate 8 seconds.
wav = model.generate_unconditional(4)    # generates 4 unconditional audio samples
descriptions = ['happy rock', 'energetic EDM', 'sad jazz']
wav = model.generate(descriptions)  # generates 3 samples.

melody, sr = torchaudio.load('./assets/bach.mp3')
# generates using the melody from the given audio and the provided descriptions.
wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr)

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)

🤗 Transformers Usage

MusicGen is available in the 🤗 Transformers library from version 4.31.0 onwards, requiring minimal dependencies
and additional packages. Steps to get started:

First install the 🤗 Transformers library from main:

shell

pip install git+https://github.com/huggingface/transformers.git

Run the following Python code to generate text-conditional audio samples:

from transformers import AutoProcessor, MusicgenForConditionalGeneration


processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")

inputs = processor(
    text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
    padding=True,
    return_tensors="pt",
)

audio_values = model.generate(**inputs, max_new_tokens=256)

Listen to the audio samples either in an ipynb notebook:

from IPython.display import Audio

sampling_rate = model.config.audio_encoder.sampling_rate
Audio(audio_values[0].numpy(), rate=sampling_rate)

Or save them as a .wav file using a third-party library, e.g. scipy:

import scipy

sampling_rate = model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write("musicgen_out.wav", rate=sampling_rate, data=audio_values[0, 0].numpy())

For more details on using the MusicGen model for inference using the 🤗 Transformers library, refer to the
MusicGen docs or the hands-on
Google Colab.

Training

The MusicGenSolver implements MusicGen's training pipeline.
It defines an autoregressive language modeling task over multiple streams of discrete tokens
extracted from a pre-trained EnCodec model (see EnCodec documentation
for more details on how to train such model).

Note that we do NOT provide any of the datasets used for training MusicGen.
We provide a dummy dataset containing just a few examples for illustrative purposes.

Please read first the TRAINING documentation, in particular the Environment Setup section.

Warning: As of version 1.1.0, a few breaking changes were introduced. Check the CHANGELOG.md
file for more information. You might need to retrain some of your models.

Example configurations and grids

We provide configurations to reproduce the released models and our research.
MusicGen solvers configuration are available in config/solver/musicgen,
in particular:

MusicGen base model for text-to-music:
solver=musicgen/musicgen_base_32khz
MusicGen model with chromagram-conditioning support:
solver=musicgen/musicgen_melody_32khz

We provide 3 different scales, e.g. model/lm/model_scale=small (300M), or medium (1.5B), and large (3.3B).

Please find some example grids to train MusicGen at
audiocraft/grids/musicgen.

shell

# text-to-music
dora grid musicgen.musicgen_base_32khz --dry_run --init
# melody-guided music generation
dora grid musicgen.musicgen_melody_base_32khz --dry_run --init
# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.

Music dataset and metadata

MusicGen's underlying dataset is an AudioDataset augmented with music-specific metadata.
The MusicGen dataset implementation expects the metadata to be available as .json files
at the same location as the audio files. Learn more in the datasets section.

Audio tokenizers

We support a number of audio tokenizers: either pretrained EnCodec models, DAC, or your own models.
The tokenizer is controlled with the setting compression_model_checkpoint.
For instance,

bash

# Using the 32kHz EnCodec trained on music
dora run solver=musicgen/debug \
    compression_model_checkpoint=//pretrained/facebook/encodec_32khz \
    transformer_lm.n_q=4 transformer_lm.card=2048

# Using DAC
dora run solver=musicgen/debug \
    compression_model_checkpoint=//pretrained/dac_44khz \
    transformer_lm.n_q=9 transformer_lm.card=1024 \
    'codebooks_pattern.delay.delays=[0,1,2,3,4,5,6,7,8]'

# Using your own model after export (see ENCODEC.md)
dora run solver=musicgen/debug \
    compression_model_checkpoint=//pretrained//checkpoints/my_audio_lm/compression_state_dict.bin \
    transformer_lm.n_q=... transformer_lm.card=...

# Using your own model from its training checkpoint.
dora run solver=musicgen/debug \
    compression_model_checkpoint=//sig/SIG \ # where SIG is the Dora signature of the EnCodec XP.
    transformer_lm.n_q=... transformer_lm.card=...

Warning: you are responsible for setting the proper value for transformer_lm.n_q and transformer_lm.card (cardinality of the codebooks). You also have to update the codebook_pattern to match n_q as shown in the example for using DAC. .

Training stereo models

Use the option interleave_stereo_codebooks.use set to True to activate stereo training along with channels=2. Left and right channels will be
encoded separately by the compression model, then their codebook will be interleaved, e.g. order of codebook is
[1_L, 1_R, 2_L, 2_R, ...]. You will also need to update the delays for the codebook patterns to match the number of codebooks, and the n_q value passed to the transformer LM:

text

dora run solver=musicgen/debug \
    compression_model_checkpoint=//pretrained/facebook/encodec_32khz \
    channels=2 interleave_stereo_codebooks.use=True \
    transformer_lm.n_q=8 transformer_lm.card=2048 \
    codebooks_pattern.delay.delays='[0, 0, 1, 1, 2, 2, 3, 3]'

Fine tuning existing models

You can initialize your model to one of the pretrained models by using the continue_from argument, in particular

bash

# Using pretrained MusicGen model.
dora run solver=musicgen/musicgen_base_32khz model/lm/model_scale=medium continue_from=//pretrained/facebook/musicgen-medium conditioner=text2music

# Using another model you already trained with a Dora signature SIG.
dora run solver=musicgen/musicgen_base_32khz model/lm/model_scale=medium continue_from=//sig/SIG conditioner=text2music

# Or providing manually a path
dora run solver=musicgen/musicgen_base_32khz model/lm/model_scale=medium continue_from=/checkpoints/my_other_xp/checkpoint.th

Fine tuning mono model to stereo

You will not be able to continue_from a mono model with stereo training, as the shape of the embeddings and output linears
would not match. You can use the following snippet to prepare a proper finetuning checkpoint.

python

from pathlib import Path
import torch

# Download the pretrained model, e.g. from
# https://huggingface.co/facebook/musicgen-melody/blob/main/state_dict.bin

model_name = 'musicgen-melody'
root = Path.home() / 'checkpoints'
# You are responsible for downloading the following checkpoint in the proper location
input_state_dict_path = root / model_name / 'state_dict.bin'
state = torch.load(input_state_dict_path, 'cpu')
bs = state['best_state']
# there is a slight different in format between training checkpoints and exported public checkpoints.
# If you want to use your own mono models from one of your training checkpont, following the instructions
# for exporting a model explained later on this page.
assert 'model' not in bs, 'The following code is for using an exported pretrained model'
nbs = dict(bs)
for k in range(8):
    # We will just copy mono embeddings and linears twice, once for left and right channels.
    nbs[f'linears.{k}.weight'] = bs[f'linears.{k//2}.weight']
    nbs[f'emb.{k}.weight'] = bs[f'emb.{k//2}.weight']
torch.save({'best_state': {'model': nbs}}, root / f'stereo_finetune_{model_name}.th')

Now, you can use $HOME/checkpoints/stereo_finetune_musicgen-melody.th as a continue_from target (without a //pretrained prefix!).

Caching of EnCodec tokens

It is possible to precompute the EnCodec tokens and other metadata.
An example of generating and using this cache provided in the musicgen.musicgen_base_cached_32khz grid.

Evaluation stage

We provide an off-the-shelf configuration to enable running the objective metrics
for audio generation in
config/solver/musicgen/evaluation/objective_eval.

One can then activate evaluation the following way:

shell

# using the configuration
dora run solver=musicgen/debug solver/musicgen/evaluation=objective_eval
# specifying each of the fields, e.g. to activate KL computation
dora run solver=musicgen/debug evaluate.metrics.kld=true

See an example evaluation grid.

Generation stage

shell

# control sampling parameters
dora run solver=musicgen/debug generate.lm.gen_duration=10 generate.lm.use_sampling=true generate.lm.top_k=15

Listening to samples

Note that generation happens automatically every 25 epochs. You can easily access and
compare samples between models (as long as they are trained) on the same dataset using the
MOS tool. For that first pip install Flask gunicorn. Then

text

gunicorn -w 4 -b 127.0.0.1:8895 -t 120 'scripts.mos:app'  --access-logfile -

And access the tool at https://127.0.0.1:8895.

Playing with the model

Once you have launched some experiments, you can easily get access
to the Solver with the latest trained model using the following snippet.

python

from audiocraft.solvers.musicgen import MusicGenSolver

solver = MusicGenSolver.get_eval_solver_from_sig('SIG', device='cpu', batch_size=8)
solver.model
solver.dataloaders

Importing / Exporting models

python

from audiocraft.utils import export
from audiocraft import train
xp = train.main.get_xp_from_sig('SIG_OF_LM')
export.export_lm(xp.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/state_dict.bin')
# You also need to bundle the EnCodec model you used !!
## Case 1) you trained your own
xp_encodec = train.main.get_xp_from_sig('SIG_OF_ENCODEC')
export.export_encodec(xp_encodec.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/compression_state_dict.bin')
## Case 2) you used a pretrained model. Give the name you used without the //pretrained/ prefix.
## This will actually not dump the actual model, simply a pointer to the right model to download.
export.export_pretrained_compression_model('facebook/encodec_32khz', '/checkpoints/my_audio_lm/compression_state_dict.bin')

Now you can load your custom model with:

python

import audiocraft.models
musicgen = audiocraft.models.MusicGen.get_pretrained('/checkpoints/my_audio_lm/')

Learn more

Learn more about AudioCraft training pipelines in the dedicated section.

FAQ

I need help on Windows

@FurkanGozukara made a complete tutorial for AudioCraft/MusicGen on Windows

I need help for running the demo on Colab

Check @camenduru tutorial on YouTube.

What are top-k, top-p, temperature and classifier-free guidance?

Check out @FurkanGozukara tutorial.

Should I use FSDP or autocast ?

Citation

text

@inproceedings{copet2023simple,
    title={Simple and Controllable Music Generation},
    author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
    year={2023},
}

License

See license information in the model card.

MUSICGEN STYLE (`docs/MUSICGEN_STYLE.md`)

MusicGen-Style: Audio Conditioning for Music Generation via Discrete Bottleneck Features

AudioCraft provides the code and models for MusicGen-Style, Audio Conditioning for Music Generation via Discrete Bottleneck Features.

MusicGen-Style is a text-and-audio-to-music model that can be conditioned on textual and audio data (thanks to a style conditioner).
The style conditioner takes as input a music excerpt of a few seconds (between 1.5 and 4.5) extracts some features that are used by the model to generate music in the same style.
This style conditioning can be mixed with textual description.

Check out our [sample page][musicgen_style_samples] or test the available demo!

We use 16K hours of licensed music to train MusicGen-Style. Specifically, we rely on an internal dataset
of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data.

Model Card

See the model card.

Installation

Please follow the AudioCraft installation instructions from the README.

MusicGen-Stem requires a GPU with at least 16 GB of memory for running inference with the medium-sized models (~1.5B parameters).

Usage

You can play with MusicGen-Style by running the jupyter notebook at demos/musicgen_style_demo.ipynb locally (if you have a GPU).
You can use the gradio demo locally by running python -m demos.musicgen_style_app --share.
You can play with MusicGen by running the jupyter notebook at demos/musicgen_style_demo.ipynb locally (if you have a GPU).

API

We provide a simple API 1 pre-trained model with MERT used as a feature extractor for the style conditioner:

facebook/musicgen-style: medium (1.5B) MusicGen model, text and style to music, generates 30-second samples - 🤗 Hub

In order to use MusicGen-Style locally you must have a GPU. We recommend 16GB of memory.

See after a quick example for using the API.

To perform text-to-music:

python

import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('facebook/musicgen-style')


model.set_generation_params(
    duration=8, # generate 8 seconds, can go up to 30
    use_sampling=True, 
    top_k=250,
    cfg_coef=3., # Classifier Free Guidance coefficient 
    cfg_coef_beta=None, # double CFG is only useful for text-and-style conditioning
)  

descriptions = ['disco beat', 'energetic EDM', 'funky groove']
wav = model.generate(descriptions)  # generates 3 samples.

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)

To perform style-to-music:

python

import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('facebook/musicgen-style')


model.set_generation_params(
    duration=8, # generate 8 seconds, can go up to 30
    use_sampling=True, 
    top_k=250,
    cfg_coef=3., # Classifier Free Guidance coefficient 
    cfg_coef_beta=None, # double CFG is only useful for text-and-style conditioning
)

model.set_style_conditioner_params(
    eval_q=1, # integer between 1 and 6
              # eval_q is the level of quantization that passes
              # through the conditioner. When low, the models adheres less to the 
              # audio conditioning
    excerpt_length=3., # the length in seconds that is taken by the model in the provided excerpt
    )

melody, sr = torchaudio.load('./assets/electronic.mp3')


wav = model.generate_with_chroma(descriptions=[None, None, None], 
                melody[None].expand(3, -1, -1), sr)  # generates 3 samples.

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)

To perform style-and-text-to-music:

text

/* Detailed source-code truncated for AI context efficiency. */

Training

To train MusicGen-Style, we use the MusicGenSolver.

Note that we do NOT provide any of the datasets used for training MusicGen-Style.
We provide a dummy dataset containing just a few examples for illustrative purposes.

Please read first the TRAINING documentation, in particular the Environment Setup section.

Example configurations and grids

We provide the configuration to reproduce the training of MusicGen-Style in config/solver/musicgen/musicgen_style_32khz.yaml,

In particular, the conditioner configuration is provided in /config/conditioner/style2music.yaml.

The grid to train the model is
audiocraft/grids/musicgen/musicgen_style_32khz.py.

shell

# text-and-style-to-music
dora grid musicgen.musicgen_style_32khz --dry_run --init

# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.

dataset and metadata

Learn more in the datasets section.

Audio tokenizers

See MusicGen

Fine tuning existing models

You can initialize your model to one of the pretrained models by using the continue_from argument, in particular

bash

# Using pretrained MusicGen-Style model.
dora run solver=musicgen/musicgen_style_32khz model/lm/model_scale=medium continue_from=//pretrained/facebook/musicgen-style conditioner=style2music

# Using another model you already trained with a Dora signature SIG.
dora run solver=musicgen/musicgen_style_32khz model/lm/model_scale=medium continue_from=//sig/SIG conditioner=style2music

# Or providing manually a path
dora run solver=musicgen/musicgen_style_32khz model/lm/model_scale=medium continue_from=/checkpoints/my_other_xp/checkpoint.th

TRAINING (`docs/TRAINING.md`)

AudioCraft training pipelines

AudioCraft training pipelines are built on top of PyTorch as our core deep learning library
and Flashy as our training pipeline design library,
and Dora as our experiment manager.
AudioCraft training pipelines are designed to be research and experiment-friendly.

Environment setup

For the base installation, follow the instructions from the README.md.
Below are some additional instructions for setting up the environment to train new models.

Team and cluster configuration

In order to support multiple teams and clusters, AudioCraft uses an environment configuration.
The team configuration allows to specify cluster-specific configurations (e.g. SLURM configuration),
or convenient mapping of paths between the supported environments.

Each team can have a yaml file under the configuration folder. To select a team set the
AUDIOCRAFT_TEAM environment variable to a valid team name (e.g. labs or default):

shell

conda env config vars set AUDIOCRAFT_TEAM=default

Alternatively, you can add it to your .bashrc:

shell

export AUDIOCRAFT_TEAM=default

If not defined, the environment will default to the default team.

The cluster is automatically detected, but it is also possible to override it by setting
the AUDIOCRAFT_CLUSTER environment variable.

Based on this team and cluster, the environment is then configured with:

The dora experiment outputs directory.
The available slurm partitions: categorized by global and team.
A shared reference directory: In order to facilitate sharing research models while remaining
agnostic to the used compute cluster, we created the //reference symbol that can be used in
YAML config to point to a defined reference folder containing shared checkpoints
(e.g. baselines, models for evaluation...).

Important: The default output dir for trained models and checkpoints is under /tmp/. This is suitable
only for quick testing. If you are doing anything serious you MUST edit the file default.yaml and
properly set the dora_dir entries.

Overriding environment configurations

You can set the following environment variables to bypass the team's environment configuration:

AUDIOCRAFT_CONFIG: absolute path to a team config yaml file.
AUDIOCRAFT_DORA_DIR: absolute path to a custom dora directory.
AUDIOCRAFT_REFERENCE_DIR: absolute path to the shared reference directory.

Training pipelines

Each task supported in AudioCraft has its own training pipeline and dedicated solver.
Learn more about solvers and key designs around AudioCraft training pipeline below.
Please refer to the documentation of each task and model for specific information on a given task.

Solvers

The core training component in AudioCraft is the solver. A solver holds the definition
of how to solve a given task: It implements the training pipeline logic, combining the datasets,
model, optimization criterion and components and the full training loop. We refer the reader
to Flashy for core principles around solvers.

AudioCraft proposes an initial solver, the StandardSolver that is used as the base implementation
for downstream solvers. This standard solver provides a nice base management of logging,
checkpoints loading/saving, xp restoration, etc. on top of the base Flashy implementation.
In AudioCraft, we made the assumption that all tasks are following the same set of stages:
train, valid, evaluate and generation, each relying on a dedicated dataset.

Each solver is responsible for defining the task to solve and the associated stages
of the training loop in order to leave the full ownership of the training pipeline
to the researchers. This includes loading the datasets, building the model and
optimisation components, registering them and defining the execution of each stage.
To create a new solver for a given task, one should extend the StandardSolver
and define each stage of the training loop. One can further customise its own solver
starting from scratch instead of inheriting from the standard solver.

text

/* Detailed source-code truncated for AI context efficiency. */

About Epochs

AudioCraft Solvers uses the concept of Epoch. One epoch doesn't necessarily mean one pass over the entire
dataset, but instead represent the smallest amount of computation that we want to work with before checkpointing.
Typically, we find that having an Epoch time around 30min is ideal both in terms of safety (checkpointing often enough)
and getting updates often enough. One Epoch is at least a train stage that lasts for optim.updates_per_epoch (2000 by default),
and a valid stage. You can control how long the valid stage takes with dataset.valid.num_samples.
Other stages (evaluate, generate) will only happen every X epochs, as given by evaluate.every and generate.every).

Models

In AudioCraft, a model is a container object that wraps one or more torch modules together
with potential processing logic to use in a solver. For example, a model would wrap an encoder module,
a quantisation bottleneck module, a decoder and some tensor processing logic. Each of the previous components
can be considered as a small « model unit » on its own but the container model is a practical component
to manipulate and train a set of modules together.

Datasets

See the dedicated documentation on datasets.

Metrics

See the dedicated documentation on metrics.

Conditioners

AudioCraft language models can be conditioned in various ways and the codebase offers a modular implementation
of different conditioners that can be potentially combined together.
Learn more in the dedicated documentation on conditioning.

Configuration

AudioCraft's configuration is defined in yaml files and the framework relies on
hydra and omegaconf to parse
and manipulate the configuration through Dora.

:warning: Important considerations around configurations

Our configuration management relies on Hydra and the concept of group configs to structure
and compose configurations. Updating the root default configuration files will then have
an impact on all solvers and tasks.
One should never change the default configuration files. Instead they should use Hydra config groups in order to store custom configuration.
Once this configuration is created and used for running experiments, you should not edit it anymore.

Note that as we are using Dora as our experiment manager, all our experiment tracking is based on
signatures computed from delta between configurations.
One must therefore ensure backward compatibility of the configuration at all time.
See Dora's README and the
section below introduction Dora.

Configuration structure

The configuration is organized in config groups:

conditioner: default values for conditioning modules.
dset: contains all data source related information (paths to manifest files
and metadata for a given dataset).
model: contains configuration for each model defined in AudioCraft and configurations
for different variants of models.
solver: contains the default configuration for each solver as well as configuration
for each solver task, combining all the above components.
teams: contains the cluster configuration per teams. See environment setup for more details.

The config.yaml file is the main configuration that composes the above groups
and contains default configuration for AudioCraft.

Solver's core configuration structure

The core configuration structure shared across solver is available in solvers/default.yaml.

Other configuration modules

AudioCraft configuration contains the different setups we used for our research and publications.

Running experiments with Dora

Launching jobs

Try launching jobs for different tasks locally with dora run:

shell

# run compression task with lightweight encodec
dora run solver=compression/debug

Most of the time, the jobs are launched through dora grids, for example:

shell

# run compression task through debug grid
dora grid compression.debug

Learn more about running experiments with Dora below.

A small introduction to Dora

Dora is the experiment manager tool used in AudioCraft.
Check out the README to learn how Dora works. Here is a quick summary of what to know:

An XP is a unique set of hyper-parameters with a given signature. The signature is a hash
of those hyper-parameters. We always refer to an XP with its signature, e.g. 9357e12e. We will see
after that one can retrieve the hyper-params and re-rerun it in a single command.
In fact, the hash is defined as a delta between the base config and the one obtained
with the config overrides you passed from the command line. This means you must never change
the conf/**.yaml files directly, except for editing things like paths. Changing the default values
in the config files means the XP signature won't reflect that change, and wrong checkpoints might be reused.
I know, this is annoying, but the reason is that otherwise, any change to the config file would mean
that all XPs ran so far would see their signature change.

Dora commands

shell

dora info -f 81de367c  # this will show the hyper-parameter used by a specific XP.
                       # Be careful some overrides might present twice, and the right most one
                       # will give you the right value for it.

dora run -d -f 81de367c   # run an XP with the hyper-parameters from XP 81de367c.
                          # `-d` is for distributed, it will use all available GPUs.

dora run -d -f 81de367c dataset.batch_size=32  # start from the config of XP 81de367c but change some hyper-params.
                                               # This will give you a new XP with a new signature (e.g. 3fe9c332).

dora info -f SIG -t    # will tail the log (if the XP has scheduled).
# if you need to access the logs of the process for rank > 0, in particular because a crash didn't happen in the main
# process, then use `dora info -f SIG` to get the main log name (finished into something like `/5037674_0_0_log.out`)
# and worker K can be accessed as `/5037674_0_{K}_log.out`.
# This is only for scheduled jobs, for local distributed runs with `-d`, then you should go into the XP folder,
# and look for `worker_{K}.log` logs.

An XP runs from a specific folder based on its signature, under the
<cluster_specific_path>//experiments/audiocraft/outputs/ folder.
You can safely interrupt a training and resume it, it will reuse any existing checkpoint,
as it will reuse the same folder. If you made some change to the code and need to ignore
a previous checkpoint you can use dora run --clear [RUN ARGS].

If you have a Slurm cluster, you can also use the dora grid command, e.g.

shell

# Run a dummy grid located at `audiocraft/grids/my_grid_folder/my_grid_name.py`
dora grid my_grid_folder.my_grid_name
# The following will simply display the grid and also initialize the Dora experiments database.
# You can then simply refer to a config using its signature (e.g. as `dora run -f SIG`).
dora grid my_grid_folder.my_grid_name --dry_run --init

Please refer to the Dora documentation for more information.

Clearing up past experiments

shell

# This will cancel all the XPs and delete their folder and checkpoints.
# It will then reschedule them starting from scratch.
dora grid my_grid_folder.my_grid_name --clear
# The following will delete the folder and checkpoint for a single XP,
# and then run it afresh.
dora run [-f BASE_SIG] [ARGS] --clear

WATERMARKING (`docs/WATERMARKING.md`)

AudioSeal: Proactive Localized Watermarking

AudioCraft provides the training code and models for AudioSeal, a method for speech localized watermarking Proactive Detection of Voice Cloning with Localized Watermarking, with state-of-the-art robustness and detector speed. It jointly trains a generator that embeds a watermark in the audio, and a detector that detects the watermarked fragments in longer audios, even in the presence of editing.

Installation and setup

Make sure to install audiocraft version 1.4.0a1 or later, and with the [wm] extra (see README).
Alternatively, you can just install audioseal yourself. To install AudioSeal, follow Installation guidelines in the AudioSeal repo.

NOTE: Since we use AAC augmentation in our training loop, you need to install ffmpeg, or it will not work (See Section "Installation" in README).

Make sure you follow steps for basic training setup before starting.

API

Check the Github repository for more details.

Training

The WatermarkSolver implements the AudioSeal's training pipeline. It joins the generator and detector that wrap
audioseal.AudioSealWM and audioseal.AudioSealDetector respectively. For the training recipe, see config/solver/watermark/robustness.yaml.

For illustration, we use the three example audios in datasets, with datasourc definition in dset/audio/example.yaml (Please read DATASET to understand AudioCraft's dataset structure.)

To run the Watermarking training pipeline locally:

bash

dora run solver=watermark/robustness dset=audio/example

you can override model / experiment parameters here directly like:

bash

dora run solver=watermark/robustness dset=audio/example sample_rate=24000

If you want to run in debug mode:

bash

python3 -m pdb -c c -m dora run solver=watermark/robustness dset=audio/example

--- METRICS ---

Files Extracted: 13
Estimated Token Budget: ~25005 tokens
Canonical Reference: https://codewiki.google/github.com/facebookresearch/audiocraft

Technical Documentation: facebookresearch/audiocraft

Overview (`README.md`) Copy Section

AudioCraft

Installation Copy Chapter

Models Copy Chapter

Training code Copy Chapter

API documentation Copy Chapter

FAQ Copy Chapter

Is the training code available?

Where are the models stored?

License Copy Chapter

Citation Copy Chapter

AUDIOGEN (`docs/AUDIOGEN.md`) Copy Section

AudioGen: Textually-guided audio generation

Model Card Copy Chapter

Installation Copy Chapter

API and usage Copy Chapter

Training Copy Chapter

Example configurations and grids Copy Section

Sound dataset and metadata Copy Section

Evaluation stage Copy Section

Generation stage Copy Section

More information Copy Chapter

Learn more Copy Section

Citation Copy Chapter

License Copy Chapter

CONDITIONING (`docs/CONDITIONING.md`) Copy Section

AudioCraft conditioning modules

Conditioning methods Copy Chapter

Core concepts Copy Chapter

Conditioners Copy Section

ConditionProvider Copy Section

ConditionFuser Copy Section

SegmentWithAttributes and ConditioningAttributes: From metadata to conditions Copy Section

Available conditioners Copy Chapter

Text conditioners Copy Section

Waveform conditioners Copy Section

Joint embeddings conditioners Copy Section

Classifier Free Guidance Copy Chapter

Attribute Dropout Copy Chapter

Faster computation of conditions Copy Chapter

DATASETS (`docs/DATASETS.md`) Copy Section

AudioCraft datasets

Getting started Copy Chapter

Example Copy Section

Creating manifest files Copy Section

Additional information Copy Chapter

MusicDataset and metadata Copy Section

SoundDataset and metadata Copy Section

ENCODEC (`docs/ENCODEC.md`) Copy Section

EnCodec: High Fidelity Neural Audio Compression

Original EnCodec models Copy Chapter

Installation Copy Chapter

Training Copy Chapter

Example configuration and grids Copy Section

Training and validation stages Copy Section

Evaluation stage Copy Section

Generation stage Copy Section

Playing with the model Copy Section

Importing / Exporting models Copy Section

Learn more Copy Section

Citation Copy Chapter

License Copy Chapter

JASCO (`docs/JASCO.md`) Copy Section

JASCO: Joint Audio And Symbolic Conditioning for Temporally Controlled Text-To-Music Generation

Model Card Copy Chapter

Installation Copy Chapter

Usage Copy Chapter

API Copy Chapter

🤗 Transformers Usage Copy Chapter

Data Preprocessing Copy Chapter

Chords conditioning Copy Section

Melody conditioning Copy Section

Clone repo and create virtual environment

Salience map dumps (of entire directory, using slurm job)

From src dir

From track list

Training Copy Chapter

Fine tuning existing models Copy Section

Evaluation & Generation stage Copy Section

Overview (`README.md`)

Installation

Models

Training code

API documentation

FAQ

License

Citation

AUDIOGEN (`docs/AUDIOGEN.md`)

Model Card

Installation

API and usage

Training

Example configurations and grids

Sound dataset and metadata

Evaluation stage

Generation stage

More information

Learn more

Citation

License

CONDITIONING (`docs/CONDITIONING.md`)

Conditioning methods

Core concepts

Conditioners

ConditionProvider

ConditionFuser

SegmentWithAttributes and ConditioningAttributes: From metadata to conditions

Available conditioners

Text conditioners

Waveform conditioners

Joint embeddings conditioners

Classifier Free Guidance

Attribute Dropout

Faster computation of conditions

DATASETS (`docs/DATASETS.md`)

Getting started

Example

Creating manifest files

Additional information

MusicDataset and metadata

SoundDataset and metadata

ENCODEC (`docs/ENCODEC.md`)

Original EnCodec models

Installation

Training

Example configuration and grids

Training and validation stages

Evaluation stage

Generation stage

Playing with the model

Importing / Exporting models

Learn more

Citation

License

JASCO (`docs/JASCO.md`)

Model Card

Installation

Usage

API

🤗 Transformers Usage

Data Preprocessing

Chords conditioning

Melody conditioning

Training

Fine tuning existing models

Evaluation & Generation stage

Playing with the model

Importing / Exporting models

Learn more

Citation

License

MAGNET (`docs/MAGNET.md`)

Model Card

Installation

Usage

API

🤗 Transformers Usage

Training

Example configurations and grids