# Technical Documentation: facebookresearch/audiocraft
> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/facebookresearch/audiocraft)
### Overview (`README.md`)
# AudioCraft
AudioCraft is a PyTorch library for deep learning research on audio generation. AudioCraft contains inference and training code
for two state-of-the-art AI generative models producing high-quality audio: AudioGen and MusicGen.
## Installation
AudioCraft requires Python 3.9, PyTorch 2.1.0. To install AudioCraft, you can run the following:
```shell
# Best to make sure you have torch installed first, in particular before installing xformers.
# Don't run this if you already have PyTorch installed.
python -m pip install 'torch==2.1.0'
# You might need the following before trying to install the packages
python -m pip install setuptools wheel
# Then proceed to one of the following
python -m pip install -U audiocraft # stable release
python -m pip install -U git+https://git@github.com/facebookresearch/audiocraft#egg=audiocraft # bleeding edge
python -m pip install -e . # or if you cloned the repo locally (mandatory if you want to train).
python -m pip install -e '.[wm]' # if you want to train a watermarking model
```
We also recommend having `ffmpeg` installed, either through your system or Anaconda:
```bash
sudo apt-get install ffmpeg
# Or if you are using Anaconda or Miniconda
conda install "ffmpeg<5" -c conda-forge
```
## Models
At the moment, AudioCraft contains the training code and inference code for:
* [MusicGen](./docs/MUSICGEN.md): A state-of-the-art controllable text-to-music model.
* [AudioGen](./docs/AUDIOGEN.md): A state-of-the-art text-to-sound model.
* [EnCodec](./docs/ENCODEC.md): A state-of-the-art high fidelity neural audio codec.
* [Multi Band Diffusion](./docs/MBD.md): An EnCodec compatible decoder using diffusion.
* [MAGNeT](./docs/MAGNET.md): A state-of-the-art non-autoregressive model for text-to-music and text-to-sound.
* [AudioSeal](./docs/WATERMARKING.md): A state-of-the-art audio watermarking.
* [MusicGen Style](./docs/MUSICGEN_STYLE.md): A state-of-the-art text-and-style-to-music model.
* [JASCO](./docs/JASCO.md): "High quality text-to-music model conditioned on chords, melodies and drum tracks"
## Training code
AudioCraft contains PyTorch components for deep learning research in audio and training pipelines for the developed models.
For a general introduction of AudioCraft design principles and instructions to develop your own training pipeline, refer to
the [AudioCraft training documentation](./docs/TRAINING.md).
For reproducing existing work and using the developed training pipelines, refer to the instructions for each specific model
that provides pointers to configuration, example grids and model/task-specific information and FAQ.
## API documentation
We provide some [API documentation](https://facebookresearch.github.io/audiocraft/api_docs/audiocraft/index.html) for AudioCraft.
## FAQ
#### Is the training code available?
Yes! We provide the training code for [EnCodec](./docs/ENCODEC.md), [MusicGen](./docs/MUSICGEN.md),[Multi Band Diffusion](./docs/MBD.md) and [JASCO](./docs/JASCO.md).
#### Where are the models stored?
Hugging Face stored the model in a specific location, which can be overridden by setting the `AUDIOCRAFT_CACHE_DIR` environment variable for the AudioCraft models.
In order to change the cache location of the other Hugging Face models, please check out the [Hugging Face Transformers documentation for the cache setup](https://huggingface.co/docs/transformers/installation#cache-setup).
Finally, if you use a model that relies on Demucs (e.g. `musicgen-melody`) and want to change the download location for Demucs, refer to the [Torch Hub documentation](https://pytorch.org/docs/stable/hub.html#where-are-my-downloaded-models-saved).
## License
* The code in this repository is released under the MIT license as found in the [LICENSE file](LICENSE).
* The models weights in this repository are released under the CC-BY-NC 4.0 license as found in the [LICENSE_weights file](LICENSE_weights).
## Citation
For the general framework of AudioCraft, please cite the following.
```
@inproceedings{copet2023simple,
title={Simple and Controllable Music Generation},
author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
year={2023},
}
```
When referring to a specific model, please cite as mentioned in the model specific README, e.g
[./docs/MUSICGEN.md](./docs/MUSICGEN.md), [./docs/AUDIOGEN.md](./docs/AUDIOGEN.md), etc.
---
### AUDIOGEN (`docs/AUDIOGEN.md`)
# AudioGen: Textually-guided audio generation
AudioCraft provides the code and a model re-implementing AudioGen, a [textually-guided audio generation][audiogen_arxiv]
model that performs text-to-sound generation.
The provided AudioGen reimplementation follows the LM model architecture introduced in [MusicGen][musicgen_arxiv]
and is a single stage auto-regressive Transformer model trained over a 16kHz
EnCodec tokenizer with 4 codebooks sampled at 50 Hz.
This model variant reaches similar audio quality than the original implementation introduced in the AudioGen publication
while providing faster generation speed given the smaller frame rate.
**Important note:** The provided models are NOT the original models used to report numbers in the
[AudioGen publication][audiogen_arxiv]. Refer to the model card to learn more about architectural changes.
Listen to samples from the **original AudioGen implementation** in our [sample page][audiogen_samples].
## Model Card
See [the model card](../model_cards/AUDIOGEN_MODEL_CARD.md).
## Installation
Please follow the AudioCraft installation instructions from the [README](../README.md).
AudioCraft requires a GPU with at least 16 GB of memory for running inference with the medium-sized models (~1.5B parameters).
## API and usage
We provide a simple API and 1 pre-trained models for AudioGen:
`facebook/audiogen-medium`: 1.5B model, text to sound - [🤗 Hub](https://huggingface.co/facebook/audiogen-medium)
You can play with AudioGen by running the jupyter notebook at [`demos/audiogen_demo.ipynb`](../demos/audiogen_demo.ipynb) locally (if you have a GPU).
See after a quick example for using the API.
```python
import torchaudio
from audiocraft.models import AudioGen
from audiocraft.data.audio import audio_write
model = AudioGen.get_pretrained('facebook/audiogen-medium')
model.set_generation_params(duration=5) # generate 5 seconds.
descriptions = ['dog barking', 'sirene of an emergency vehicle', 'footsteps in a corridor']
wav = model.generate(descriptions) # generates 3 samples.
for idx, one_wav in enumerate(wav):
# Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
```
## Training
The [AudioGenSolver](../audiocraft/solvers/audiogen.py) implements the AudioGen's training pipeline
used to develop the released model. Note that this may not fully reproduce the results presented in the paper.
Similarly to MusicGen, it defines an autoregressive language modeling task over multiple streams of
discrete tokens extracted from a pre-trained EnCodec model (see [EnCodec documentation](./ENCODEC.md)
for more details on how to train such model) with dataset-specific changes for environmental sound
processing.
Note that **we do NOT provide any of the datasets** used for training AudioGen.
### Example configurations and grids
We provide configurations to reproduce the released models and our research.
AudioGen solvers configuration are available in [config/solver/audiogen](../config/solver/audiogen).
The base training configuration used for the released models is the following:
[`solver=audiogen/audiogen_base_16khz`](../config/solver/audiogen/audiogen_base_16khz.yaml)
Please find some example grids to train AudioGen at
[audiocraft/grids/audiogen](../audiocraft/grids/audiogen/).
```shell
# text-to-sound
dora grid audiogen.audiogen_base_16khz
```
### Sound dataset and metadata
AudioGen's underlying dataset is an AudioDataset augmented with description metadata.
The AudioGen dataset implementation expects the metadata to be available as `.json` files
at the same location as the audio files or through specified external folder.
Learn more in the [datasets section](./DATASETS.md).
### Evaluation stage
By default, evaluation stage is also computing the cross-entropy and the perplexity over the
evaluation dataset. Indeed the objective metrics used for evaluation can be costly to run
or require some extra dependencies. Please refer to the [metrics documentation](./METRICS.md)
for more details on the requirements for each metric.
We provide an off-the-shelf configuration to enable running the objective metrics
for audio generation in
[config/solver/audiogen/evaluation/objective_eval](../config/solver/audiogen/evaluation/objective_eval.yaml).
One can then activate evaluation the following way:
```shell
# using the configuration
dora run solver=audiogen/debug solver/audiogen/evaluation=objective_eval
# specifying each of the fields, e.g. to activate KL computation
dora run solver=audiogen/debug evaluate.metrics.kld=true
```
See [an example evaluation grid](../audiocraft/grids/audiogen/audiogen_pretrained_16khz_eval.py).
### Generation stage
The generation stage allows to generate samples conditionally and/or unconditionally and to perform
audio continuation (from a prompt). We currently support greedy sampling (argmax), sampling
from softmax with a given temperature, top-K and top-P (nucleus) sampling. The number of samples
generated and the batch size used are controlled by the `dataset.generate` configuration
while the other generation parameters are defined in `generate.lm`.
```shell
# control sampling parameters
dora run solver=audiogen/debug generate.lm.gen_duration=5 generate.lm.use_sampling=true generate.lm.top_k=15
```
## More information
Refer to [MusicGen's instructions](./MUSICGEN.md).
### Learn more
Learn more about AudioCraft training pipelines in the [dedicated section](./TRAINING.md).
## Citation
AudioGen
```
@article{kreuk2022audiogen,
title={Audiogen: Textually guided audio generation},
author={Kreuk, Felix and Synnaeve, Gabriel and Polyak, Adam and Singer, Uriel and D{\'e}fossez, Alexandre and Copet, Jade and Parikh, Devi and Taigman, Yaniv and Adi, Yossi},
journal={arXiv preprint arXiv:2209.15352},
year={2022}
}
```
MusicGen
```
@article{copet2023simple,
title={Simple and Controllable Music Generation},
author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},
year={2023},
journal={arXiv preprint arXiv:2306.05284},
}
```
## License
See license information in the [model card](../model_cards/AUDIOGEN_MODEL_CARD.md).
[audiogen_arxiv]: https://arxiv.org/abs/2209.15352
[musicgen_arxiv]: https://arxiv.org/abs/2306.05284
[audiogen_samples]: https://felixkreuk.github.io/audiogen/
---
### CONDITIONING (`docs/CONDITIONING.md`)
# AudioCraft conditioning modules
AudioCraft provides a
[modular implementation of conditioning modules](../audiocraft/modules/conditioners.py)
that can be used with the language model to condition the generation.
The codebase was developed in order to easily extend the set of modules
currently supported to easily develop new ways of controlling the generation.
## Conditioning methods
For now, we support 3 main types of conditioning within AudioCraft:
* Text-based conditioning methods
* Waveform-based conditioning methods
* Joint embedding conditioning methods for text and audio projected in a shared latent space.
The Language Model relies on 2 core components that handle processing information:
* The `ConditionProvider` class, that maps metadata to processed conditions, leveraging
all the defined conditioners for the given task.
* The `ConditionFuser` class, that takes preprocessed conditions and properly fuse the
conditioning embedding to the language model inputs following a given fusing strategy.
Different conditioners (for text, waveform, joint embeddings...) are provided as torch
modules in AudioCraft and are used internally in the language model to process the
conditioning signals and feed them to the language model.
## Core concepts
### Conditioners
The `BaseConditioner` torch module is the base implementation for all conditioners in AudioCraft.
Each conditioner is expected to implement 2 methods:
* The `tokenize` method that is used as a preprocessing method that contains all processing
that can lead to synchronization points (e.g. BPE tokenization with transfer to the GPU).
The output of the tokenize method will then be used to feed the forward method.
* The `forward` method that takes the output of the tokenize method and contains the core computation
to obtain the conditioning embedding along with a mask indicating valid indices (e.g. padding tokens).
### ConditionProvider
The ConditionProvider prepares and provides conditions given a dictionary of conditioners.
Conditioners are specified as a dictionary of attributes and the corresponding conditioner
providing the processing logic for the given attribute.
Similarly to the conditioners, the condition provider works in two steps to avoid synchronization points:
* A `tokenize` method that takes a list of conditioning attributes for the batch,
and runs all tokenize steps for the set of conditioners.
* A `forward` method that takes the output of the tokenize step and runs all the forward steps
for the set of conditioners.
The list of conditioning attributes is passed as a list of `ConditioningAttributes`
that is presented just below.
### ConditionFuser
Once all conditioning signals have been extracted and processed by the `ConditionProvider`
as dense embeddings, they remain to be passed to the language model along with the original
language model inputs.
The `ConditionFuser` handles specifically the logic to combine the different conditions
to the actual model input, supporting different strategies to combine them.
One can therefore define different strategies to combine or fuse the condition to the input, in particular:
* Prepending the conditioning signal to the input with the `prepend` strategy,
* Summing the conditioning signal to the input with the `sum` strategy,
* Combining the conditioning relying on a cross-attention mechanism with the `cross` strategy,
* Using input interpolation with the `input_interpolate` strategy.
### SegmentWithAttributes and ConditioningAttributes: From metadata to conditions
The `ConditioningAttributes` dataclass is the base class for metadata
containing all attributes used for conditioning the language model.
It currently supports the following types of attributes:
* Text conditioning attributes: Dictionary of textual attributes used for text-conditioning.
* Wav conditioning attributes: Dictionary of waveform attributes used for waveform-based
conditioning such as the chroma conditioning.
* JointEmbed conditioning attributes: Dictionary of text and waveform attributes
that are expected to be represented in a shared latent space.
These different types of attributes are the attributes that are processed
by the different conditioners.
`ConditioningAttributes` are extracted from metadata loaded along the audio in the datasets,
provided that the metadata used by the dataset implements the `SegmentWithAttributes` abstraction.
All metadata-enabled datasets to use for conditioning in AudioCraft inherits
the [`audiocraft.data.info_dataset.InfoAudioDataset`](../audiocraft/data/info_audio_dataset.py) class
and the corresponding metadata inherits and implements the `SegmentWithAttributes` abstraction.
Refer to the [`audiocraft.data.music_dataset.MusicAudioDataset`](../audiocraft/data/music_dataset.py)
class as an example.
## Available conditioners
### Text conditioners
All text conditioners are expected to inherit from the `TextConditioner` class.
AudioCraft currently provides two text conditioners:
* The `LUTConditioner` that relies on look-up-table of embeddings learned at train time,
and relying on either no tokenizer or a spacy tokenizer. This conditioner is particularly
useful for simple experiments and categorical labels.
* The `T5Conditioner` that relies on a
[pre-trained T5 model](https://huggingface.co/docs/transformers/model_doc/t5)
frozen or fine-tuned at train time to extract the text embeddings.
### Waveform conditioners
All waveform conditioners are expected to inherit from the `WaveformConditioner` class and
consist of a conditioning method that takes a waveform as input. The waveform conditioner
must implement the logic to extract the embedding from the waveform and define the downsampling
factor from the waveform to the resulting embedding.
The `ChromaStemConditioner` conditioner is a waveform conditioner for the chroma features
conditioning used by MusicGen. It takes a given waveform, extracts relevant stems for melody
(namely all non drums and bass stems) using a
[pre-trained Demucs model](https://github.com/facebookresearch/demucs)
and then extracts the chromagram bins from the remaining mix of stems.
### Joint embeddings conditioners
We finally provide support for conditioning based on joint text and audio embeddings through
the `JointEmbeddingConditioner` class and the `CLAPEmbeddingConditioner` that implements such
a conditioning method relying on a [pretrained CLAP model](https://github.com/LAION-AI/CLAP).
## Classifier Free Guidance
We provide a Classifier Free Guidance implementation in AudioCraft. With the classifier free
guidance dropout, all attributes are dropped with the same probability.
## Attribute Dropout
We further provide an attribute dropout strategy. Unlike the classifier free guidance dropout,
the attribute dropout drops given attributes with a defined probability, allowing the model
not to expect all conditioning signals to be provided at once.
## Faster computation of conditions
Conditioners that require some heavy computation on the waveform can be cached, in particular
the `ChromaStemConditioner` or `CLAPEmbeddingConditioner`. You just need to provide the
`cache_path` parameter to them. We recommend running dummy jobs for filling up the cache quickly.
An example is provided in the [musicgen.musicgen_melody_32khz grid](../audiocraft/grids/musicgen/musicgen_melody_32khz.py).
---
### DATASETS (`docs/DATASETS.md`)
# AudioCraft datasets
Our dataset manifest files consist in 1-json-per-line files, potentially gzipped,
as `data.jsons` or `data.jsons.gz` files. This JSON contains the path to the audio
file and associated metadata. The manifest files are then provided in the configuration,
as `datasource` sub-configuration. A datasource contains the pointers to the paths of
the manifest files for each AudioCraft stage (or split) along with additional information
(eg. maximum sample rate to use against this dataset). All the datasources are under the
`dset` group config, with a dedicated configuration file for each dataset.
## Getting started
### Example
See the provided example in the directory that provides a manifest to use the example dataset
provided under the [dataset folder](../dataset/example).
The manifest files are stored in the [egs folder](../egs/example).
```shell
egs/
example/data.json.gz
```
A datasource is defined in the configuration folder, in the dset group config for this dataset
at [config/dset/audio/example](../config/dset/audio/example.yaml):
```shell
# @package __global__
datasource:
max_sample_rate: 44100
max_channels: 2
train: egs/example
valid: egs/example
evaluate: egs/example
generate: egs/example
```
For proper dataset, one should create manifest for each of the splits and specify the correct path
to the given manifest in the datasource for each split.
Then, using a dataset through the configuration can be done pointing to the
corresponding dataset configuration:
```shell
dset= # should match the yaml file name
# for example
dset=audio/example
```
### Creating manifest files
Assuming you want to create manifest files to load with AudioCraft's AudioDataset, you can use
the following command to create new manifest files from a given folder containing audio files:
```shell
python -m audiocraft.data.audio_dataset egs/my_dataset/my_dataset_split/data.jsonl.gz
# For example to generate the manifest for dset=audio/example
# note: we don't use any split and we don't compress the jsonl file for this dummy example
python -m audiocraft.data.audio_dataset dataset/example egs/example/data.jsonl
# More info with: python -m audiocraft.data.audio_dataset --help
```
## Additional information
### MusicDataset and metadata
The MusicDataset is an AudioDataset with additional metadata. The MusicDataset expects
the additional metadata to be stored in a JSON file that has the same path as the corresponding
audio file, but with a `.json` extension.
### SoundDataset and metadata
The SoundDataset is an AudioDataset with descriptions metadata. Similarly to the MusicDataset,
the SoundDataset expects the additional metadata to be stored in a JSON file that has the same
path as the corresponding audio file, but with a `.json` extension. Additionally, the SoundDataset
supports an additional parameter pointing to an extra folder `external_metadata_source` containing
all the JSON metadata files given they have the same filename as the audio file.
---
### ENCODEC (`docs/ENCODEC.md`)
# EnCodec: High Fidelity Neural Audio Compression
AudioCraft provides the training code for EnCodec, a state-of-the-art deep learning
based audio codec supporting both mono and stereo audio, presented in the
[High Fidelity Neural Audio Compression][arxiv] paper.
Check out our [sample page][encodec_samples].
## Original EnCodec models
The EnCodec models presented in High Fidelity Neural Audio Compression can be accessed
and used with the [EnCodec repository](https://github.com/facebookresearch/encodec).
**Note**: We do not guarantee compatibility between the AudioCraft and EnCodec codebases
and released checkpoints at this stage.
## Installation
Please follow the AudioCraft installation instructions from the [README](../README.md).
## Training
The [CompressionSolver](../audiocraft/solvers/compression.py) implements the audio reconstruction
task to train an EnCodec model. Specifically, it trains an encoder-decoder with a quantization
bottleneck - a SEANet encoder-decoder with Residual Vector Quantization bottleneck for EnCodec -
using a combination of objective and perceptual losses in the forms of discriminators.
The default configuration matches a causal EnCodec training at a single bandwidth.
### Example configuration and grids
We provide sample configuration and grids for training EnCodec models.
The compression configuration are defined in
[config/solver/compression](../config/solver/compression).
The example grids are available at
[audiocraft/grids/compression](../audiocraft/grids/compression).
```shell
# base causal encodec on monophonic audio sampled at 24 khz
dora grid compression.encodec_base_24khz
# encodec model used for MusicGen on monophonic audio sampled at 32 khz
dora grid compression.encodec_musicgen_32khz
```
### Training and validation stages
The model is trained using a combination of objective and perceptual losses.
More specifically, EnCodec is trained with the MS-STFT discriminator along with
objective losses through the use of a loss balancer to effectively weight
the different losses, in an intuitive manner.
### Evaluation stage
Evaluation metrics for audio generation:
* SI-SNR: Scale-Invariant Signal-to-Noise Ratio.
* ViSQOL: Virtual Speech Quality Objective Listener.
Note: Path to the ViSQOL binary (compiled with bazel) needs to be provided in
order to run the ViSQOL metric on the reference and degraded signals.
The metric is disabled by default.
Please refer to the [metrics documentation](../METRICS.md) to learn more.
### Generation stage
The generation stage consists in generating the reconstructed audio from samples
with the current model. The number of samples generated and the batch size used are
controlled by the `dataset.generate` configuration. The output path and audio formats
are defined in the generate stage configuration.
```shell
# generate samples every 5 epoch
dora run solver=compression/encodec_base_24khz generate.every=5
# run with a different dset
dora run solver=compression/encodec_base_24khz generate.path=
# limit the number of samples or use a different batch size
dora grid solver=compression/encodec_base_24khz dataset.generate.num_samples=10 dataset.generate.batch_size=4
```
### Playing with the model
Once you have a model trained, it is possible to get the entire solver, or just
the trained model with the following functions:
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Importing / Exporting models
At the moment we do not have a definitive workflow for exporting EnCodec models, for
instance to Hugging Face (HF). We are working on supporting automatic conversion between
AudioCraft and Hugging Face implementations.
We still have some support for fine-tuning an EnCodec model coming from HF in AudioCraft,
using for instance `continue_from=//pretrained/facebook/encodec_32k`.
An AudioCraft checkpoint can be exported in a more compact format (excluding the optimizer etc.)
using `audiocraft.utils.export.export_encodec`. For instance, you could run
```python
from audiocraft.utils import export
from audiocraft import train
xp = train.main.get_xp_from_sig('SIG')
export.export_encodec(
xp.folder / 'checkpoint.th',
'/checkpoints/my_audio_lm/compression_state_dict.bin')
from audiocraft.models import CompressionModel
model = CompressionModel.get_pretrained('/checkpoints/my_audio_lm/compression_state_dict.bin')
from audiocraft.solvers import CompressionSolver
# The two are strictly equivalent, but this function supports also loading from non-already exported models.
model = CompressionSolver.model_from_checkpoint('//pretrained//checkpoints/my_audio_lm/compression_state_dict.bin')
```
We will see then how to use this model as a tokenizer for MusicGen/AudioGen in the
[MusicGen documentation](./MUSICGEN.md).
### Learn more
Learn more about AudioCraft training pipelines in the [dedicated section](./TRAINING.md).
## Citation
```
@article{defossez2022highfi,
title={High Fidelity Neural Audio Compression},
author={Défossez, Alexandre and Copet, Jade and Synnaeve, Gabriel and Adi, Yossi},
journal={arXiv preprint arXiv:2210.13438},
year={2022}
}
```
## License
See license information in the [README](../README.md).
[arxiv]: https://arxiv.org/abs/2210.13438
[encodec_samples]: https://ai.honu.io/papers/encodec/samples.html
---
### JASCO (`docs/JASCO.md`)
# JASCO: Joint Audio And Symbolic Conditioning for Temporally Controlled Text-To-Music Generation
AudioCraft provides the code and models for JASCO, [Joint Audio And Symbolic Conditioning for Temporally Controlled Text-To-Music Generation][arxiv].
We present JASCO, a temporally controlled text-to-music generation model utilizing both symbolic and audio-based conditions.
JASCO can generate high-quality music samples conditioned on global text descriptions along with fine-grained local controls.
JASCO is based on the Flow Matching modeling paradigm together with a novel conditioning method, allowing for music generation controlled both locally (e.g., chords) and globally (text description).
Check out our [sample page][sample_page] or test the available demo!
We use ~16K hours of licensed music to train JASCO.
## Model Card
See [the model card](../model_cards/JASCO_MODEL_CARD.md).
## Installation
First, Please follow the AudioCraft installation instructions from the [README](../README.md).
Then, download and install chord_extractor from [source](http://www.isophonics.net/nnls-chroma)
See further required installation under **Data Preprocessing** section
## Usage
We currently offer two ways to interact with JASCO:
1. You can use the gradio demo locally by running [`python -m demos.jasco_app`](../demos/jasco_app.py), you can add `--share` to deploy a sharable space mounted on your device.
2. You can play with JASCO by running the jupyter notebook at [`demos/jasco_demo.ipynb`](../demos/jasco_demo.ipynb) locally.
## API
We provide a simple API and pre-trained models:
- `facebook/jasco-chords-drums-400M`: 400M model, text to music with chords and drums support, generates 10-second samples - [🤗 Hub](https://huggingface.co/facebook/jasco-chords-drums-400M)
- `facebook/jasco-chords-drums-1B`: 1B model, text to music with chords and drums support, generates 10-second samples - [🤗 Hub](https://huggingface.co/facebook/jasco-chords-drums-1B)
- `facebook/jasco-chords-drums-melody-400M`: 400M model, text to music with chords, drums and melody support, generates 10-second samples - [🤗 Hub](https://huggingface.co/facebook/jasco-chords-drums-melody-400M)
- `facebook/jasco-chords-drums-melody-1B`: 1B model, text to music with chords, drums and melody support, generates 10-second samples - [🤗 Hub](https://huggingface.co/facebook/jasco-chords-drums-melody-1B)
See after a quick example for using the API.
```python
from audiocraft.models import JASCO
model = JASCO.get_pretrained('facebook/jasco-chords-drums-400M', chords_mapping_path='../assets/chord_to_index_mapping.pkl')
model.set_generation_params(
cfg_coef_all=5.0,
cfg_coef_txt=0.0
)
# set textual prompt
text = "Strings, woodwind, orchestral, symphony."
# define chord progression
chords = [('C', 0.0), ('D', 2.0), ('F', 4.0), ('Ab', 6.0), ('Bb', 7.0), ('C', 8.0)]
# run inference
output = model.generate_music(descriptions=[text], chords=chords, progress=True)
audio_write('output', output.cpu().squeeze(0), model.sample_rate, strategy="loudness", loudness_compressor=True)
```
For more examples check out `demos/jasco_demo.ipynb`
## 🤗 Transformers Usage
Coming soon...
## Data Preprocessing
In order to to use the JascoDataset with chords / melody conditioning, please follow the instructions below:
### Chords conditioning
To extract chords from your desired data follow the following steps:
1. Prepare a `*.jsonl` containing list of absolute file paths in your dataset, should simply be absolute paths seperated by newlines.
2. Download and install chord_extractor from [source](http://www.isophonics.net/nnls-chroma)
3. For training purposes run: `python scripts/chords/extract_chords.py --src_jsonl_file= --target_output_dir=`
and then run: `python scripts/chords/build_chord_map.py --chords_folder= --output_directory=`
4. For evaluation of our released models run: `python scripts/chords/extract_chords.py --src_jsonl_file= --target_output_dir= --path_to_pre_defined_map=`
and then run: `python scripts/chords/build_chord_map.py --chords_folder= --output_directory= --path_to_pre_defined_map=`
NOTE: current scripts assume that all audio files are of `.wav` format, some changes may be required if your data consists of other formats.
NOTE: predefined chord mapping file is available in `assets` directory.
### Melody conditioning
This section relies on [Deepsalience repo](https://github.com/rabitt/ismir2017-deepsalience) with slight custom scripts written.
#### Clone repo and create virtual environment
1. `git clone git@github.com:lonzi/ismir2017-deepsalience.git forked_deepsalience_repo`
2. `cd forked_deepsalience_repo`
3. `conda create --name deep_salience python=3.7`
4. `conda activate deep_salience`
5. `pip install -r requirements.txt`
#### Salience map dumps (of entire directory, using slurm job)
##### From src dir
1. create job array: `python predict/create_predict_saliency_cmds.py --src_dir= --out_dir= --n_shards= --multithread`
2. run job array: `sbatch predict_saliency.sh`
##### From track list
1. create job array: `python predict/create_predict_saliency_cmds.py --tracks_list=tracks_train.txt --out_dir= --n_shards=2 --multithread --sbatch_script_name=predict_saliency_train.sh --saliency_threshold=`
2. run job array: `sbatch predict_saliency_train.sh`
tracks_train.txt: a list of track paths to process seperated by new lines
## Training
The [JascoSolver](../audiocraft/solvers/jasco.py) implements JASCO's training pipeline.
conditional flow matching objective over the continuous extracted latents from a pre-trained EnCodec model (see [EnCodec documentation](./ENCODEC.md)
for more details on how to train such model).
Note that **we do NOT provide any of the datasets** used for training JASCO.
We provide a dummy dataset containing just a few examples for illustrative purposes.
Please read first the [TRAINING documentation](./TRAINING.md), in particular the Environment Setup section.
### Fine tuning existing models
You can initialize your model to one of the pretrained models by using the `continue_from` argument, in particular
```bash
# Using pretrained JASCO model.
dora run solver=jasco/chords_drums model/lm/model_scale=small continue_from=//pretrained/facebook/jasco-chords-drums-400M conditioner=jasco_chords_drums
# Using another model you already trained with a Dora signature SIG.
dora run solver=jasco/chords_drums model/lm/model_scale=small continue_from=//sig/SIG conditioner=jasco_chords_drums
# Or providing manually a path
dora run solver=jasco/chords_drums model/lm/model_scale=small conditioner=jasco_chords_drums continue_from=/checkpoints/my_other_xp/checkpoint.th
```
**Warning:** You are responsible for selecting the other parameters accordingly, in a way that make it compatible
with the model you are fine tuning. Configuration is NOT automatically inherited from the model you continue from. In particular make sure to select the proper `conditioner` and `model/lm/model_scale`.
**Warning:** We currently do not support fine tuning a model with slightly different layers. If you decide
to change some parts, like the conditioning or some other parts of the model, you are responsible for manually crafting a checkpoint file from which we can safely run `load_state_dict`.
If you decide to do so, make sure your checkpoint is saved with `torch.save` and contains a dict
`{'best_state': {'model': model_state_dict_here}}`. Directly give the path to `continue_from` without a `//pretrained/` prefix.
### Evaluation & Generation stage
See [MusicGen](./MUSICGEN.md)
### Playing with the model
Once you have launched some experiments, you can easily get access
to the Solver with the latest trained model using the following snippet.
```python
from audiocraft.solvers.jasco import JascoSolver
solver = JascoSolver.get_eval_solver_from_sig('SIG', device='cpu', batch_size=8)
solver.model
solver.dataloaders
```
### Importing / Exporting models
We do not support currently loading a model from the Hugging Face implementation or exporting to it.
If you want to export your model in a way that is compatible with `audiocraft.models.JASCO`
API, you can run:
```python
from audiocraft.utils import export
from audiocraft import train
xp = train.main.get_xp_from_sig('SIG_OF_LM')
export.export_lm(xp.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/state_dict.bin')
# You also need to bundle the EnCodec model you used !!
## Case 1) you trained your own
xp_encodec = train.main.get_xp_from_sig('SIG_OF_ENCODEC')
export.export_encodec(xp_encodec.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/compression_state_dict.bin')
## Case 2) you used a pretrained model. Give the name you used without the //pretrained/ prefix.
## This will actually not dump the actual model, simply a pointer to the right model to download.
export.export_pretrained_compression_model('facebook/encodec_32khz', '/checkpoints/my_audio_lm/compression_state_dict.bin')
```
Now you can load your custom model with:
```python
import audiocraft.models
jasco = audiocraft.models.JASCO.get_pretrained('/checkpoints/my_audio_lm/')
```
### Learn more
Learn more about AudioCraft training pipelines in the [dedicated section](./TRAINING.md).
## Citation
```
@misc{tal2024joint,
title={Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation},
author={Or Tal and Alon Ziv and Itai Gat and Felix Kreuk and Yossi Adi},
year={2024},
eprint={2406.10970},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
```
## License
See license information in the [model card](../model_cards/JASCO_MODEL_CARD.md).
[arxiv]: https://arxiv.org/pdf/2406.10970
[sample_page]: https://pages.cs.huji.ac.il/adiyoss-lab/JASCO/
---
### MAGNET (`docs/MAGNET.md`)
# MAGNeT: Masked Audio Generation using a Single Non-Autoregressive Transformer
AudioCraft provides the code and models for MAGNeT, [Masked Audio Generation using a Single Non-Autoregressive Transformer][arxiv].
MAGNeT is a text-to-music and text-to-sound model capable of generating high-quality audio samples conditioned on text descriptions.
It is a masked generative non-autoregressive Transformer trained over a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50 Hz.
Unlike prior work on masked generative audio Transformers, such as [SoundStorm](https://arxiv.org/abs/2305.09636) and [VampNet](https://arxiv.org/abs/2307.04686),
MAGNeT doesn't require semantic token conditioning, model cascading or audio prompting, and employs a full text-to-audio using a single non-autoregressive Transformer.
Check out our [sample page][magnet_samples] or test the available demo!
We use 16K hours of licensed music to train MAGNeT. Specifically, we rely on an internal dataset
of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data.
## Model Card
See [the model card](../model_cards/MAGNET_MODEL_CARD.md).
## Installation
Please follow the AudioCraft installation instructions from the [README](../README.md).
AudioCraft requires a GPU with at least 16 GB of memory for running inference with the medium-sized models (~1.5B parameters).
## Usage
We currently offer two ways to interact with MAGNeT:
1. You can use the gradio demo locally by running [`python -m demos.magnet_app --share`](../demos/magnet_app.py).
2. You can play with MAGNeT by running the jupyter notebook at [`demos/magnet_demo.ipynb`](../demos/magnet_demo.ipynb) locally (if you have a GPU).
## API
We provide a simple API and 6 pre-trained models. The pre trained models are:
- `facebook/magnet-small-10secs`: 300M model, text to music, generates 10-second samples - [🤗 Hub](https://huggingface.co/facebook/magnet-small-10secs)
- `facebook/magnet-medium-10secs`: 1.5B model, text to music, generates 10-second samples - [🤗 Hub](https://huggingface.co/facebook/magnet-medium-10secs)
- `facebook/magnet-small-30secs`: 300M model, text to music, generates 30-second samples - [🤗 Hub](https://huggingface.co/facebook/magnet-small-30secs)
- `facebook/magnet-medium-30secs`: 1.5B model, text to music, generates 30-second samples - [🤗 Hub](https://huggingface.co/facebook/magnet-medium-30secs)
- `facebook/audio-magnet-small`: 300M model, text to sound-effect - [🤗 Hub](https://huggingface.co/facebook/audio-magnet-small)
- `facebook/audio-magnet-medium`: 1.5B model, text to sound-effect - [🤗 Hub](https://huggingface.co/facebook/audio-magnet-medium)
In order to use MAGNeT locally **you must have a GPU**. We recommend 16GB of memory, especially for
the medium size models.
See after a quick example for using the API.
```python
import torchaudio
from audiocraft.models import MAGNeT
from audiocraft.data.audio import audio_write
model = MAGNeT.get_pretrained('facebook/magnet-small-10secs')
descriptions = ['disco beat', 'energetic EDM', 'funky groove']
wav = model.generate(descriptions) # generates 3 samples.
for idx, one_wav in enumerate(wav):
# Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
```
## 🤗 Transformers Usage
Coming soon...
## Training
The [MagnetSolver](../audiocraft/solvers/magnet.py) implements MAGNeT's training pipeline.
It defines a masked generation task over multiple streams of discrete tokens
extracted from a pre-trained EnCodec model (see [EnCodec documentation](./ENCODEC.md)
for more details on how to train such model).
Note that **we do NOT provide any of the datasets** used for training MAGNeT.
We provide a dummy dataset containing just a few examples for illustrative purposes.
Please read first the [TRAINING documentation](./TRAINING.md), in particular the Environment Setup section.
### Example configurations and grids
We provide configurations to reproduce the released models and our research.
MAGNeT solvers configuration are available in [config/solver/magnet](../config/solver/magnet),
in particular:
* MAGNeT model for text-to-music:
[`solver=magnet/magnet_32khz`](../config/solver/magnet/magnet_32khz.yaml)
* MAGNeT model for text-to-sound:
[`solver=magnet/audio_magnet_16khz`](../config/solver/magnet/audio_magnet_16khz.yaml)
We provide 3 different scales, e.g. `model/lm/model_scale=small` (300M), or `medium` (1.5B), and `large` (3.3B).
Please find some example grids to train MAGNeT at
[audiocraft/grids/magnet](../audiocraft/grids/magnet/).
```shell
# text-to-music
dora grid magnet.magnet_32khz --dry_run --init
# text-to-sound
dora grid magnet.audio_magnet_16khz --dry_run --init
# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.
```
### dataset and metadata
Learn more in the [datasets section](./DATASETS.md).
#### Music Models
MAGNeT's underlying dataset is an AudioDataset augmented with music-specific metadata.
The MAGNeT dataset implementation expects the metadata to be available as `.json` files
at the same location as the audio files.
#### Sound Models
Audio-MAGNeT's underlying dataset is an AudioDataset augmented with description metadata.
The Audio-MAGNeT dataset implementation expects the metadata to be available as `.json` files
at the same location as the audio files or through specified external folder.
### Audio tokenizers
See [MusicGen](./MUSICGEN.md)
### Fine tuning existing models
You can initialize your model to one of the pretrained models by using the `continue_from` argument, in particular
```bash
# Using pretrained MAGNeT model.
dora run solver=magnet/magnet_32khz model/lm/model_scale=medium continue_from=//pretrained/facebook/magnet-medium-10secs conditioner=text2music
# Using another model you already trained with a Dora signature SIG.
dora run solver=magnet/magnet_32khz model/lm/model_scale=medium continue_from=//sig/SIG conditioner=text2music
# Or providing manually a path
dora run solver=magnet/magnet_32khz model/lm/model_scale=medium continue_from=/checkpoints/my_other_xp/checkpoint.th
```
**Warning:** You are responsible for selecting the other parameters accordingly, in a way that make it compatible
with the model you are fine tuning. Configuration is NOT automatically inherited from the model you continue from. In particular make sure to select the proper `conditioner` and `model/lm/model_scale`.
**Warning:** We currently do not support fine tuning a model with slightly different layers. If you decide
to change some parts, like the conditioning or some other parts of the model, you are responsible for manually crafting a checkpoint file from which we can safely run `load_state_dict`.
If you decide to do so, make sure your checkpoint is saved with `torch.save` and contains a dict
`{'best_state': {'model': model_state_dict_here}}`. Directly give the path to `continue_from` without a `//pretrained/` prefix.
### Evaluation stage
For the 6 pretrained MAGNeT models, objective metrics could be reproduced using the following grids:
```shell
# text-to-music
REGEN=1 dora grid magnet.magnet_pretrained_32khz_eval --dry_run --init
# text-to-sound
REGEN=1 dora grid magnet.audio_magnet_pretrained_16khz_eval --dry_run --init
# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.
```
See [MusicGen](./MUSICGEN.md) for more details.
### Generation stage
See [MusicGen](./MUSICGEN.md)
### Playing with the model
Once you have launched some experiments, you can easily get access
to the Solver with the latest trained model using the following snippet.
```python
from audiocraft.solvers.magnet import MagnetSolver
solver = MagnetSolver.get_eval_solver_from_sig('SIG', device='cpu', batch_size=8)
solver.model
solver.dataloaders
```
### Importing / Exporting models
We do not support currently loading a model from the Hugging Face implementation or exporting to it.
If you want to export your model in a way that is compatible with `audiocraft.models.MAGNeT`
API, you can run:
```python
from audiocraft.utils import export
from audiocraft import train
xp = train.main.get_xp_from_sig('SIG_OF_LM')
export.export_lm(xp.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/state_dict.bin')
# You also need to bundle the EnCodec model you used !!
## Case 1) you trained your own
xp_encodec = train.main.get_xp_from_sig('SIG_OF_ENCODEC')
export.export_encodec(xp_encodec.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/compression_state_dict.bin')
## Case 2) you used a pretrained model. Give the name you used without the //pretrained/ prefix.
## This will actually not dump the actual model, simply a pointer to the right model to download.
export.export_pretrained_compression_model('facebook/encodec_32khz', '/checkpoints/my_audio_lm/compression_state_dict.bin')
```
Now you can load your custom model with:
```python
import audiocraft.models
magnet = audiocraft.models.MAGNeT.get_pretrained('/checkpoints/my_audio_lm/')
```
### Learn more
Learn more about AudioCraft training pipelines in the [dedicated section](./TRAINING.md).
## FAQ
#### What are top-k, top-p, temperature and classifier-free guidance?
Check out [@FurkanGozukara tutorial](https://github.com/FurkanGozukara/Stable-Diffusion/blob/main/Tutorials/AI-Music-Generation-Audiocraft-Tutorial.md#more-info-about-top-k-top-p-temperature-and-classifier-free-guidance-from-chatgpt).
#### Should I use FSDP or autocast ?
The two are mutually exclusive (because FSDP does autocast on its own).
You can use autocast up to 1.5B (medium), if you have enough RAM on your GPU.
FSDP makes everything more complex but will free up some memory for the actual
activations by sharding the optimizer state.
## Citation
```
@misc{ziv2024masked,
title={Masked Audio Generation using a Single Non-Autoregressive Transformer},
author={Alon Ziv and Itai Gat and Gael Le Lan and Tal Remez and Felix Kreuk and Alexandre Défossez and Jade Copet and Gabriel Synnaeve and Yossi Adi},
year={2024},
eprint={2401.04577},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
```
## License
See license information in the [model card](../model_cards/MAGNET_MODEL_CARD.md).
[arxiv]: https://arxiv.org/abs/2401.04577
[magnet_samples]: https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT/
---
### MBD (`docs/MBD.md`)
# MultiBand Diffusion
AudioCraft provides the code and models for MultiBand Diffusion, [From Discrete Tokens to High Fidelity Audio using MultiBand Diffusion][arxiv].
MultiBand diffusion is a collection of 4 models that can decode tokens from
EnCodec tokenizer into waveform audio. You can listen to some examples on the sample page.
## Installation
Please follow the AudioCraft installation instructions from the [README](../README.md).
## Usage
We offer a number of way to use MultiBand Diffusion:
1. The MusicGen demo includes a toggle to try diffusion decoder. You can use the demo locally by running [`python -m demos.musicgen_app --share`](../demos/musicgen_app.py), or through the [MusicGen Colab](https://colab.research.google.com/drive/1JlTOjB-G0A2Hz3h8PK63vLZk4xdCI5QB?usp=sharing).
2. You can play with MusicGen by running the jupyter notebook at [`demos/musicgen_demo.ipynb`](../demos/musicgen_demo.ipynb) locally (if you have a GPU).
## API
We provide a simple API and pre-trained models for MusicGen and for EnCodec at 24 khz for 3 bitrates (1.5 kbps, 3 kbps and 6 kbps).
See after a quick example for using MultiBandDiffusion with the MusicGen API:
```python
import torchaudio
from audiocraft.models import MusicGen, MultiBandDiffusion
from audiocraft.data.audio import audio_write
model = MusicGen.get_pretrained('facebook/musicgen-melody')
mbd = MultiBandDiffusion.get_mbd_musicgen()
model.set_generation_params(duration=8) # generate 8 seconds.
wav, tokens = model.generate_unconditional(4, return_tokens=True) # generates 4 unconditional audio samples and keep the tokens for MBD generation
descriptions = ['happy rock', 'energetic EDM', 'sad jazz']
wav_diffusion = mbd.tokens_to_wav(tokens)
wav, tokens = model.generate(descriptions, return_tokens=True) # generates 3 samples and keep the tokens.
wav_diffusion = mbd.tokens_to_wav(tokens)
melody, sr = torchaudio.load('./assets/bach.mp3')
# Generates using the melody from the given audio and the provided descriptions, returns audio and audio tokens.
wav, tokens = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr, return_tokens=True)
wav_diffusion = mbd.tokens_to_wav(tokens)
for idx, one_wav in enumerate(wav):
# Will save under {idx}.wav and {idx}_diffusion.wav, with loudness normalization at -14 db LUFS for comparing the methods.
audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
audio_write(f'{idx}_diffusion', wav_diffusion[idx].cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
```
For the compression task (and to compare with [EnCodec](https://github.com/facebookresearch/encodec)):
```python
import torch
from audiocraft.models import MultiBandDiffusion
from encodec import EncodecModel
from audiocraft.data.audio import audio_read, audio_write
bandwidth = 3.0 # 1.5, 3.0, 6.0
mbd = MultiBandDiffusion.get_mbd_24khz(bw=bandwidth)
encodec = EncodecModel.encodec_model_24khz()
somepath = ''
wav, sr = audio_read(somepath)
with torch.no_grad():
compressed_encodec = encodec(wav)
compressed_diffusion = mbd.regenerate(wav, sample_rate=sr)
audio_write('sample_encodec', compressed_encodec.squeeze(0).cpu(), mbd.sample_rate, strategy="loudness", loudness_compressor=True)
audio_write('sample_diffusion', compressed_diffusion.squeeze(0).cpu(), mbd.sample_rate, strategy="loudness", loudness_compressor=True)
```
## Training
The [DiffusionSolver](../audiocraft/solvers/diffusion.py) implements our diffusion training pipeline.
It generates waveform audio conditioned on the embeddings extracted from a pre-trained EnCodec model
(see [EnCodec documentation](./ENCODEC.md) for more details on how to train such model).
Note that **we do NOT provide any of the datasets** used for training our diffusion models.
We provide a dummy dataset containing just a few examples for illustrative purposes.
### Example configurations and grids
One can train diffusion models as described in the paper by using this [dora grid](../audiocraft/grids/diffusion/4_bands_base_32khz.py).
```shell
# 4 bands MBD trainning
dora grid diffusion.4_bands_base_32khz
```
### Learn more
Learn more about AudioCraft training pipelines in the [dedicated section](./TRAINING.md).
## Citation
```
@article{sanroman2023fromdi,
title={From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion},
author={San Roman, Robin and Adi, Yossi and Deleforge, Antoine and Serizel, Romain and Synnaeve, Gabriel and Défossez, Alexandre},
journal={arXiv preprint arXiv:},
year={2023}
}
```
## License
See license information in the [README](../README.md).
[arxiv]: https://arxiv.org/abs/2308.02560
[mbd_samples]: https://ai.honu.io/papers/mbd/
---
### METRICS (`docs/METRICS.md`)
# AudioCraft objective metrics
In addition to training losses, AudioCraft provides a set of objective metrics
for audio synthesis and audio generation. As these metrics may require
extra dependencies and can be costly to train, they are often disabled by default.
This section provides guidance for setting up and using these metrics in
the AudioCraft training pipelines.
## Available metrics
### Audio synthesis quality metrics
#### SI-SNR
We provide an implementation of the Scale-Invariant Signal-to-Noise Ratio in PyTorch.
No specific requirement is needed for this metric. Please activate the metric at the
evaluation stage with the appropriate flag:
**Warning:** We report the opposite of the SI-SNR, e.g. multiplied by -1. This is due to internal
details where the SI-SNR score can also be used as a training loss function, where lower
values should indicate better reconstruction. Negative values are such expected and a good sign! Those should be again multiplied by `-1` before publication :)
```shell
dora run <...> evaluate.metrics.sisnr=true
```
#### ViSQOL
We provide a Python wrapper around the ViSQOL [official implementation](https://github.com/google/visqol)
to conveniently run ViSQOL within the training pipelines.
One must specify the path to the ViSQOL installation through the configuration in order
to enable ViSQOL computations in AudioCraft:
```shell
# the first parameter is used to activate visqol computation while the second specify
# the path to visqol's library to be used by our python wrapper
dora run <...> evaluate.metrics.visqol=true metrics.visqol.bin=
```
See an example grid: [Compression with ViSQOL](../audiocraft/grids/compression/encodec_musicgen_32khz.py)
To learn more about ViSQOL and how to build ViSQOL binary using bazel, please refer to the
instructions available in the [open source repository](https://github.com/google/visqol).
### Audio generation metrics
#### Frechet Audio Distance
Similarly to ViSQOL, we use a Python wrapper around the Frechet Audio Distance
[official implementation](https://github.com/google-research/google-research/tree/master/frechet_audio_distance)
in TensorFlow.
Note that we had to make several changes to the actual code in order to make it work.
Please refer to the [FrechetAudioDistanceMetric](../audiocraft/metrics/fad.py) class documentation
for more details. We do not plan to provide further support in obtaining a working setup for the
Frechet Audio Distance at this stage.
```shell
# the first parameter is used to activate FAD metric computation while the second specify
# the path to FAD library to be used by our python wrapper
dora run <...> evaluate.metrics.fad=true metrics.fad.bin=
```
See an example grid: [Evaluation with FAD](../audiocraft/grids/musicgen/musicgen_pretrained_32khz_eval.py)
#### Kullback-Leibler Divergence
We provide a PyTorch implementation of the Kullback-Leibler Divergence computed over the probabilities
of the labels obtained by a state-of-the-art audio classifier. We provide our implementation of the KLD
using the [PaSST classifier](https://github.com/kkoutini/PaSST).
In order to use the KLD metric over PaSST, you must install the PaSST library as an extra dependency:
```shell
pip install 'git+https://github.com/kkoutini/passt_hear21@0.0.19#egg=hear21passt'
```
Then similarly, you can use the metric activating the corresponding flag:
```shell
# one could extend the kld metric with additional audio classifier models that can then be picked through the configuration
dora run <...> evaluate.metrics.kld=true metrics.kld.model=passt
```
#### Text consistency
We provide a text-consistency metric, similarly to the MuLan Cycle Consistency from
[MusicLM](https://arxiv.org/pdf/2301.11325.pdf) or the CLAP score used in
[Make-An-Audio](https://arxiv.org/pdf/2301.12661v1.pdf).
More specifically, we provide a PyTorch implementation of a Text consistency metric
relying on a pre-trained [Contrastive Language-Audio Pretraining (CLAP)](https://github.com/LAION-AI/CLAP).
Please install the CLAP library as an extra dependency prior to using the metric:
```shell
pip install laion_clap
```
Then similarly, you can use the metric activating the corresponding flag:
```shell
# one could extend the text consistency metric with additional audio classifier models that can then be picked through the configuration
dora run ... evaluate.metrics.text_consistency=true metrics.text_consistency.model=clap
```
Note that the text consistency metric based on CLAP will require the CLAP checkpoint to be
provided in the configuration.
#### Chroma cosine similarity
Finally, as introduced in MusicGen, we provide a Chroma Cosine Similarity metric in PyTorch.
No specific requirement is needed for this metric. Please activate the metric at the
evaluation stage with the appropriate flag:
```shell
dora run ... evaluate.metrics.chroma_cosine=true
```
#### Comparing against reconstructed audio
For all the above audio generation metrics, we offer the option to compute the metric on the reconstructed audio
fed in EnCodec instead of the generated sample using the flag `.use_gt=true`.
## Example usage
You will find example of configuration for the different metrics introduced above in:
* The [musicgen's default solver](../config/solver/musicgen/default.yaml) for all audio generation metrics
* The [compression's default solver](../config/solver/compression/default.yaml) for all audio synthesis metrics
Similarly, we provide different examples in our grids:
* [Evaluation with ViSQOL](../audiocraft/grids/compression/encodec_musicgen_32khz.py)
* [Evaluation with FAD and others](../audiocraft/grids/musicgen/musicgen_pretrained_32khz_eval.py)
---
### MUSICGEN (`docs/MUSICGEN.md`)
# MusicGen: Simple and Controllable Music Generation
AudioCraft provides the code and models for MusicGen, [a simple and controllable model for music generation][arxiv].
MusicGen is a single stage auto-regressive Transformer model trained over a 32kHz
EnCodec tokenizer with 4 codebooks sampled at 50 Hz.
Unlike existing methods like [MusicLM](https://arxiv.org/abs/2301.11325), MusicGen doesn't require
a self-supervised semantic representation, and it generates all 4 codebooks in one pass. By introducing
a small delay between the codebooks, we show we can predict them in parallel, thus having only 50 auto-regressive
steps per second of audio.
Check out our [sample page][musicgen_samples] or test the available demo!
We use 20K hours of licensed music to train MusicGen. Specifically, we rely on an internal dataset
of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data.
## Model Card
See [the model card](../model_cards/MUSICGEN_MODEL_CARD.md).
## Installation
Please follow the AudioCraft installation instructions from the [README](../README.md).
AudioCraft requires a GPU with at least 16 GB of memory for running inference with the medium-sized models (~1.5B parameters).
## Usage
We offer a number of way to interact with MusicGen:
1. A demo is also available on the [`facebook/MusicGen` Hugging Face Space](https://huggingface.co/spaces/facebook/MusicGen)
(huge thanks to all the HF team for their support).
2. You can run the extended demo on a Colab:
[colab notebook](https://ai.honu.io/red/musicgen-colab)
3. You can use the gradio demo locally by running [`python -m demos.musicgen_app --share`](../demos/musicgen_app.py).
4. You can play with MusicGen by running the jupyter notebook at [`demos/musicgen_demo.ipynb`](../demos/musicgen_demo.ipynb) locally (if you have a GPU).
5. Finally, checkout [@camenduru Colab page](https://github.com/camenduru/MusicGen-colab)
which is regularly updated with contributions from @camenduru and the community.
## API
We provide a simple API and 10 pre-trained models. The pre trained models are:
- `facebook/musicgen-small`: 300M model, text to music only - [🤗 Hub](https://huggingface.co/facebook/musicgen-small)
- `facebook/musicgen-medium`: 1.5B model, text to music only - [🤗 Hub](https://huggingface.co/facebook/musicgen-medium)
- `facebook/musicgen-melody`: 1.5B model, text to music and text+melody to music - [🤗 Hub](https://huggingface.co/facebook/musicgen-melody)
- `facebook/musicgen-large`: 3.3B model, text to music only - [🤗 Hub](https://huggingface.co/facebook/musicgen-large)
- `facebook/musicgen-melody-large`: 3.3B model, text to music and text+melody to music - [🤗 Hub](https://huggingface.co/facebook/musicgen-melody-large)
- `facebook/musicgen-stereo-*`: All the previous models fine tuned for stereo generation -
[small](https://huggingface.co/facebook/musicgen-stereo-small),
[medium](https://huggingface.co/facebook/musicgen-stereo-medium),
[large](https://huggingface.co/facebook/musicgen-stereo-large),
[melody](https://huggingface.co/facebook/musicgen-stereo-melody),
[melody large](https://huggingface.co/facebook/musicgen-stereo-melody-large).
We observe the best trade-off between quality and compute with the `facebook/musicgen-medium` or `facebook/musicgen-melody` model.
In order to use MusicGen locally **you must have a GPU**. We recommend 16GB of memory, but smaller
GPUs will be able to generate short sequences, or longer sequences with the `facebook/musicgen-small` model.
See after a quick example for using the API.
```python
import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write
model = MusicGen.get_pretrained('facebook/musicgen-melody')
model.set_generation_params(duration=8) # generate 8 seconds.
wav = model.generate_unconditional(4) # generates 4 unconditional audio samples
descriptions = ['happy rock', 'energetic EDM', 'sad jazz']
wav = model.generate(descriptions) # generates 3 samples.
melody, sr = torchaudio.load('./assets/bach.mp3')
# generates using the melody from the given audio and the provided descriptions.
wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr)
for idx, one_wav in enumerate(wav):
# Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
```
## 🤗 Transformers Usage
MusicGen is available in the 🤗 Transformers library from version 4.31.0 onwards, requiring minimal dependencies
and additional packages. Steps to get started:
1. First install the 🤗 [Transformers library](https://github.com/huggingface/transformers) from main:
```shell
pip install git+https://github.com/huggingface/transformers.git
```
2. Run the following Python code to generate text-conditional audio samples:
```py
from transformers import AutoProcessor, MusicgenForConditionalGeneration
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
inputs = processor(
text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
padding=True,
return_tensors="pt",
)
audio_values = model.generate(**inputs, max_new_tokens=256)
```
3. Listen to the audio samples either in an ipynb notebook:
```py
from IPython.display import Audio
sampling_rate = model.config.audio_encoder.sampling_rate
Audio(audio_values[0].numpy(), rate=sampling_rate)
```
Or save them as a `.wav` file using a third-party library, e.g. `scipy`:
```py
import scipy
sampling_rate = model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write("musicgen_out.wav", rate=sampling_rate, data=audio_values[0, 0].numpy())
```
For more details on using the MusicGen model for inference using the 🤗 Transformers library, refer to the
[MusicGen docs](https://huggingface.co/docs/transformers/main/en/model_doc/musicgen) or the hands-on
[Google Colab](https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/MusicGen.ipynb).
## Training
The [MusicGenSolver](../audiocraft/solvers/musicgen.py) implements MusicGen's training pipeline.
It defines an autoregressive language modeling task over multiple streams of discrete tokens
extracted from a pre-trained EnCodec model (see [EnCodec documentation](./ENCODEC.md)
for more details on how to train such model).
Note that **we do NOT provide any of the datasets** used for training MusicGen.
We provide a dummy dataset containing just a few examples for illustrative purposes.
Please read first the [TRAINING documentation](./TRAINING.md), in particular the Environment Setup section.
**Warning:** As of version 1.1.0, a few breaking changes were introduced. Check the [CHANGELOG.md](../CHANGELOG.md)
file for more information. You might need to retrain some of your models.
### Example configurations and grids
We provide configurations to reproduce the released models and our research.
MusicGen solvers configuration are available in [config/solver/musicgen](../config/solver/musicgen),
in particular:
* MusicGen base model for text-to-music:
[`solver=musicgen/musicgen_base_32khz`](../config/solver/musicgen/musicgen_base_32khz.yaml)
* MusicGen model with chromagram-conditioning support:
[`solver=musicgen/musicgen_melody_32khz`](../config/solver/musicgen/musicgen_melody_32khz.yaml)
We provide 3 different scales, e.g. `model/lm/model_scale=small` (300M), or `medium` (1.5B), and `large` (3.3B).
Please find some example grids to train MusicGen at
[audiocraft/grids/musicgen](../audiocraft/grids/musicgen/).
```shell
# text-to-music
dora grid musicgen.musicgen_base_32khz --dry_run --init
# melody-guided music generation
dora grid musicgen.musicgen_melody_base_32khz --dry_run --init
# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.
```
### Music dataset and metadata
MusicGen's underlying dataset is an AudioDataset augmented with music-specific metadata.
The MusicGen dataset implementation expects the metadata to be available as `.json` files
at the same location as the audio files. Learn more in the [datasets section](./DATASETS.md).
### Audio tokenizers
We support a number of audio tokenizers: either pretrained EnCodec models, [DAC](https://github.com/descriptinc/descript-audio-codec), or your own models.
The tokenizer is controlled with the setting `compression_model_checkpoint`.
For instance,
```bash
# Using the 32kHz EnCodec trained on music
dora run solver=musicgen/debug \
compression_model_checkpoint=//pretrained/facebook/encodec_32khz \
transformer_lm.n_q=4 transformer_lm.card=2048
# Using DAC
dora run solver=musicgen/debug \
compression_model_checkpoint=//pretrained/dac_44khz \
transformer_lm.n_q=9 transformer_lm.card=1024 \
'codebooks_pattern.delay.delays=[0,1,2,3,4,5,6,7,8]'
# Using your own model after export (see ENCODEC.md)
dora run solver=musicgen/debug \
compression_model_checkpoint=//pretrained//checkpoints/my_audio_lm/compression_state_dict.bin \
transformer_lm.n_q=... transformer_lm.card=...
# Using your own model from its training checkpoint.
dora run solver=musicgen/debug \
compression_model_checkpoint=//sig/SIG \ # where SIG is the Dora signature of the EnCodec XP.
transformer_lm.n_q=... transformer_lm.card=...
```
**Warning:** you are responsible for setting the proper value for `transformer_lm.n_q` and `transformer_lm.card` (cardinality of the codebooks). You also have to update the codebook_pattern to match `n_q` as shown in the example for using DAC. .
### Training stereo models
Use the option `interleave_stereo_codebooks.use` set to `True` to activate stereo training along with `channels=2`. Left and right channels will be
encoded separately by the compression model, then their codebook will be interleaved, e.g. order of codebook is
`[1_L, 1_R, 2_L, 2_R, ...]`. You will also need to update the delays for the codebook patterns to match the number of codebooks, and the `n_q` value passed to the transformer LM:
```
dora run solver=musicgen/debug \
compression_model_checkpoint=//pretrained/facebook/encodec_32khz \
channels=2 interleave_stereo_codebooks.use=True \
transformer_lm.n_q=8 transformer_lm.card=2048 \
codebooks_pattern.delay.delays='[0, 0, 1, 1, 2, 2, 3, 3]'
```
### Fine tuning existing models
You can initialize your model to one of the pretrained models by using the `continue_from` argument, in particular
```bash
# Using pretrained MusicGen model.
dora run solver=musicgen/musicgen_base_32khz model/lm/model_scale=medium continue_from=//pretrained/facebook/musicgen-medium conditioner=text2music
# Using another model you already trained with a Dora signature SIG.
dora run solver=musicgen/musicgen_base_32khz model/lm/model_scale=medium continue_from=//sig/SIG conditioner=text2music
# Or providing manually a path
dora run solver=musicgen/musicgen_base_32khz model/lm/model_scale=medium continue_from=/checkpoints/my_other_xp/checkpoint.th
```
**Warning:** You are responsible for selecting the other parameters accordingly, in a way that make it compatible
with the model you are fine tuning. Configuration is NOT automatically inherited from the model you continue from. In particular make sure to select the proper `conditioner` and `model/lm/model_scale`.
**Warning:** We currently do not support fine tuning a model with slightly different layers. If you decide
to change some parts, like the conditioning or some other parts of the model, you are responsible for manually crafting a checkpoint file from which we can safely run `load_state_dict`.
If you decide to do so, make sure your checkpoint is saved with `torch.save` and contains a dict
`{'best_state': {'model': model_state_dict_here}}`. Directly give the path to `continue_from` without a `//pretrained/` prefix.
#### Fine tuning mono model to stereo
You will not be able to `continue_from` a mono model with stereo training, as the shape of the embeddings and output linears
would not match. You can use the following snippet to prepare a proper finetuning checkpoint.
```python
from pathlib import Path
import torch
# Download the pretrained model, e.g. from
# https://huggingface.co/facebook/musicgen-melody/blob/main/state_dict.bin
model_name = 'musicgen-melody'
root = Path.home() / 'checkpoints'
# You are responsible for downloading the following checkpoint in the proper location
input_state_dict_path = root / model_name / 'state_dict.bin'
state = torch.load(input_state_dict_path, 'cpu')
bs = state['best_state']
# there is a slight different in format between training checkpoints and exported public checkpoints.
# If you want to use your own mono models from one of your training checkpont, following the instructions
# for exporting a model explained later on this page.
assert 'model' not in bs, 'The following code is for using an exported pretrained model'
nbs = dict(bs)
for k in range(8):
# We will just copy mono embeddings and linears twice, once for left and right channels.
nbs[f'linears.{k}.weight'] = bs[f'linears.{k//2}.weight']
nbs[f'emb.{k}.weight'] = bs[f'emb.{k//2}.weight']
torch.save({'best_state': {'model': nbs}}, root / f'stereo_finetune_{model_name}.th')
```
Now, you can use `$HOME/checkpoints/stereo_finetune_musicgen-melody.th` as a `continue_from` target (without a `//pretrained` prefix!).
### Caching of EnCodec tokens
It is possible to precompute the EnCodec tokens and other metadata.
An example of generating and using this cache provided in the [musicgen.musicgen_base_cached_32khz grid](../audiocraft/grids/musicgen/musicgen_base_cached_32khz.py).
### Evaluation stage
By default, evaluation stage is also computing the cross-entropy and the perplexity over the
evaluation dataset. Indeed the objective metrics used for evaluation can be costly to run
or require some extra dependencies. Please refer to the [metrics documentation](./METRICS.md)
for more details on the requirements for each metric.
We provide an off-the-shelf configuration to enable running the objective metrics
for audio generation in
[config/solver/musicgen/evaluation/objective_eval](../config/solver/musicgen/evaluation/objective_eval.yaml).
One can then activate evaluation the following way:
```shell
# using the configuration
dora run solver=musicgen/debug solver/musicgen/evaluation=objective_eval
# specifying each of the fields, e.g. to activate KL computation
dora run solver=musicgen/debug evaluate.metrics.kld=true
```
See [an example evaluation grid](../audiocraft/grids/musicgen/musicgen_pretrained_32khz_eval.py).
### Generation stage
The generation stage allows to generate samples conditionally and/or unconditionally and to perform
audio continuation (from a prompt). We currently support greedy sampling (argmax), sampling
from softmax with a given temperature, top-K and top-P (nucleus) sampling. The number of samples
generated and the batch size used are controlled by the `dataset.generate` configuration
while the other generation parameters are defined in `generate.lm`.
```shell
# control sampling parameters
dora run solver=musicgen/debug generate.lm.gen_duration=10 generate.lm.use_sampling=true generate.lm.top_k=15
```
#### Listening to samples
Note that generation happens automatically every 25 epochs. You can easily access and
compare samples between models (as long as they are trained) on the same dataset using the
MOS tool. For that first `pip install Flask gunicorn`. Then
```
gunicorn -w 4 -b 127.0.0.1:8895 -t 120 'scripts.mos:app' --access-logfile -
```
And access the tool at [https://127.0.0.1:8895](https://127.0.0.1:8895).
### Playing with the model
Once you have launched some experiments, you can easily get access
to the Solver with the latest trained model using the following snippet.
```python
from audiocraft.solvers.musicgen import MusicGenSolver
solver = MusicGenSolver.get_eval_solver_from_sig('SIG', device='cpu', batch_size=8)
solver.model
solver.dataloaders
```
### Importing / Exporting models
We do not support currently loading a model from the Hugging Face implementation or exporting to it.
If you want to export your model in a way that is compatible with `audiocraft.models.MusicGen`
API, you can run:
```python
from audiocraft.utils import export
from audiocraft import train
xp = train.main.get_xp_from_sig('SIG_OF_LM')
export.export_lm(xp.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/state_dict.bin')
# You also need to bundle the EnCodec model you used !!
## Case 1) you trained your own
xp_encodec = train.main.get_xp_from_sig('SIG_OF_ENCODEC')
export.export_encodec(xp_encodec.folder / 'checkpoint.th', '/checkpoints/my_audio_lm/compression_state_dict.bin')
## Case 2) you used a pretrained model. Give the name you used without the //pretrained/ prefix.
## This will actually not dump the actual model, simply a pointer to the right model to download.
export.export_pretrained_compression_model('facebook/encodec_32khz', '/checkpoints/my_audio_lm/compression_state_dict.bin')
```
Now you can load your custom model with:
```python
import audiocraft.models
musicgen = audiocraft.models.MusicGen.get_pretrained('/checkpoints/my_audio_lm/')
```
### Learn more
Learn more about AudioCraft training pipelines in the [dedicated section](./TRAINING.md).
## FAQ
#### I need help on Windows
@FurkanGozukara made a complete tutorial for [AudioCraft/MusicGen on Windows](https://youtu.be/v-YpvPkhdO4)
#### I need help for running the demo on Colab
Check [@camenduru tutorial on YouTube](https://www.youtube.com/watch?v=EGfxuTy9Eeo).
#### What are top-k, top-p, temperature and classifier-free guidance?
Check out [@FurkanGozukara tutorial](https://github.com/FurkanGozukara/Stable-Diffusion/blob/main/Tutorials/AI-Music-Generation-Audiocraft-Tutorial.md#more-info-about-top-k-top-p-temperature-and-classifier-free-guidance-from-chatgpt).
#### Should I use FSDP or autocast ?
The two are mutually exclusive (because FSDP does autocast on its own).
You can use autocast up to 1.5B (medium), if you have enough RAM on your GPU.
FSDP makes everything more complex but will free up some memory for the actual
activations by sharding the optimizer state.
## Citation
```
@inproceedings{copet2023simple,
title={Simple and Controllable Music Generation},
author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
year={2023},
}
```
## License
See license information in the [model card](../model_cards/MUSICGEN_MODEL_CARD.md).
[arxiv]: https://arxiv.org/abs/2306.05284
[musicgen_samples]: https://ai.honu.io/papers/musicgen/
---
### MUSICGEN STYLE (`docs/MUSICGEN_STYLE.md`)
# MusicGen-Style: Audio Conditioning for Music Generation via Discrete Bottleneck Features
AudioCraft provides the code and models for MusicGen-Style, [Audio Conditioning for Music Generation via Discrete Bottleneck Features][arxiv].
MusicGen-Style is a text-and-audio-to-music model that can be conditioned on textual and audio data (thanks to a style conditioner).
The style conditioner takes as input a music excerpt of a few seconds (between 1.5 and 4.5) extracts some features that are used by the model to generate music in the same style.
This style conditioning can be mixed with textual description.
Check out our [sample page][musicgen_style_samples] or test the available demo!
We use 16K hours of licensed music to train MusicGen-Style. Specifically, we rely on an internal dataset
of 10K high-quality music tracks, and on the ShutterStock and Pond5 music data.
## Model Card
See [the model card](../model_cards/MUSICGEN_STYLE_MODEL_CARD.md).
## Installation
Please follow the AudioCraft installation instructions from the [README](../README.md).
MusicGen-Stem requires a GPU with at least 16 GB of memory for running inference with the medium-sized models (~1.5B parameters).
## Usage
1. You can play with MusicGen-Style by running the jupyter notebook at [`demos/musicgen_style_demo.ipynb`](../demos/musicgen_style_demo.ipynb) locally (if you have a GPU).
2. You can use the gradio demo locally by running python -m demos.musicgen_style_app --share.
3. You can play with MusicGen by running the jupyter notebook at demos/musicgen_style_demo.ipynb locally (if you have a GPU).
## API
We provide a simple API 1 pre-trained model with MERT used as a feature extractor for the style conditioner:
- `facebook/musicgen-style`: medium (1.5B) MusicGen model, text and style to music, generates 30-second samples - [🤗 Hub](https://huggingface.co/facebook/musicgen-style)
In order to use MusicGen-Style locally **you must have a GPU**. We recommend 16GB of memory.
See after a quick example for using the API.
To perform text-to-music:
```python
import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write
model = MusicGen.get_pretrained('facebook/musicgen-style')
model.set_generation_params(
duration=8, # generate 8 seconds, can go up to 30
use_sampling=True,
top_k=250,
cfg_coef=3., # Classifier Free Guidance coefficient
cfg_coef_beta=None, # double CFG is only useful for text-and-style conditioning
)
descriptions = ['disco beat', 'energetic EDM', 'funky groove']
wav = model.generate(descriptions) # generates 3 samples.
for idx, one_wav in enumerate(wav):
# Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
```
To perform style-to-music:
```python
import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write
model = MusicGen.get_pretrained('facebook/musicgen-style')
model.set_generation_params(
duration=8, # generate 8 seconds, can go up to 30
use_sampling=True,
top_k=250,
cfg_coef=3., # Classifier Free Guidance coefficient
cfg_coef_beta=None, # double CFG is only useful for text-and-style conditioning
)
model.set_style_conditioner_params(
eval_q=1, # integer between 1 and 6
# eval_q is the level of quantization that passes
# through the conditioner. When low, the models adheres less to the
# audio conditioning
excerpt_length=3., # the length in seconds that is taken by the model in the provided excerpt
)
melody, sr = torchaudio.load('./assets/electronic.mp3')
wav = model.generate_with_chroma(descriptions=[None, None, None],
melody[None].expand(3, -1, -1), sr) # generates 3 samples.
for idx, one_wav in enumerate(wav):
# Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness", loudness_compressor=True)
```
To perform style-and-text-to-music:
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Training
To train MusicGen-Style, we use the [MusicGenSolver](../audiocraft/solvers/musicgen.py).
Note that **we do NOT provide any of the datasets** used for training MusicGen-Style.
We provide a dummy dataset containing just a few examples for illustrative purposes.
Please read first the [TRAINING documentation](./TRAINING.md), in particular the Environment Setup section.
### Example configurations and grids
We provide the configuration to reproduce the training of MusicGen-Style in [config/solver/musicgen/musicgen_style_32khz.yaml](../config/solver/musicgen/musicgen_style_32khz.yaml),
In particular, the conditioner configuration is provided in [/config/conditioner/style2music.yaml](../config/conditioner/style2music.yaml).
The grid to train the model is
[audiocraft/grids/musicgen/musicgen_style_32khz.py](../audiocraft/grids/musicgen/musicgen_style_32khz.py).
```shell
# text-and-style-to-music
dora grid musicgen.musicgen_style_32khz --dry_run --init
# Remove the `--dry_run --init` flags to actually schedule the jobs once everything is setup.
```
### dataset and metadata
Learn more in the [datasets section](./DATASETS.md).
### Audio tokenizers
See [MusicGen](./MUSICGEN.md)
### Fine tuning existing models
You can initialize your model to one of the pretrained models by using the `continue_from` argument, in particular
```bash
# Using pretrained MusicGen-Style model.
dora run solver=musicgen/musicgen_style_32khz model/lm/model_scale=medium continue_from=//pretrained/facebook/musicgen-style conditioner=style2music
# Using another model you already trained with a Dora signature SIG.
dora run solver=musicgen/musicgen_style_32khz model/lm/model_scale=medium continue_from=//sig/SIG conditioner=style2music
# Or providing manually a path
dora run solver=musicgen/musicgen_style_32khz model/lm/model_scale=medium continue_from=/checkpoints/my_other_xp/checkpoint.th
```
**Warning:** You are responsible for selecting the other parameters accordingly, in a way that make it compatible
with the model you are fine tuning. Configuration is NOT automatically inherited from the model you continue from. In particular make sure to select the proper `conditioner` and `model/lm/model_scale`.
**Warning:** We currently do not support fine tuning a model with slightly different layers. If you decide
to change some parts, like the conditioning or some other parts of the model, you are responsible for manually crafting a checkpoint file from which we can safely run `load_state_dict`.
If you decide to do so, make sure your checkpoint is saved with `torch.save` and contains a dict
`{'best_state': {'model': model_state_dict_here}}`. Directly give the path to `continue_from` without a `//pretrained/` prefix.
[arxiv]: https://arxiv.org/abs/2407.12563
[musicgen_samples]: https://musicgenstyle.github.io/
---
### TRAINING (`docs/TRAINING.md`)
# AudioCraft training pipelines
AudioCraft training pipelines are built on top of PyTorch as our core deep learning library
and [Flashy](https://github.com/facebookresearch/flashy) as our training pipeline design library,
and [Dora](https://github.com/facebookresearch/dora) as our experiment manager.
AudioCraft training pipelines are designed to be research and experiment-friendly.
## Environment setup
For the base installation, follow the instructions from the [README.md](../README.md).
Below are some additional instructions for setting up the environment to train new models.
### Team and cluster configuration
In order to support multiple teams and clusters, AudioCraft uses an environment configuration.
The team configuration allows to specify cluster-specific configurations (e.g. SLURM configuration),
or convenient mapping of paths between the supported environments.
Each team can have a yaml file under the [configuration folder](../config). To select a team set the
`AUDIOCRAFT_TEAM` environment variable to a valid team name (e.g. `labs` or `default`):
```shell
conda env config vars set AUDIOCRAFT_TEAM=default
```
Alternatively, you can add it to your `.bashrc`:
```shell
export AUDIOCRAFT_TEAM=default
```
If not defined, the environment will default to the `default` team.
The cluster is automatically detected, but it is also possible to override it by setting
the `AUDIOCRAFT_CLUSTER` environment variable.
Based on this team and cluster, the environment is then configured with:
* The dora experiment outputs directory.
* The available slurm partitions: categorized by global and team.
* A shared reference directory: In order to facilitate sharing research models while remaining
agnostic to the used compute cluster, we created the `//reference` symbol that can be used in
YAML config to point to a defined reference folder containing shared checkpoints
(e.g. baselines, models for evaluation...).
**Important:** The default output dir for trained models and checkpoints is under `/tmp/`. This is suitable
only for quick testing. If you are doing anything serious you MUST edit the file `default.yaml` and
properly set the `dora_dir` entries.
#### Overriding environment configurations
You can set the following environment variables to bypass the team's environment configuration:
* `AUDIOCRAFT_CONFIG`: absolute path to a team config yaml file.
* `AUDIOCRAFT_DORA_DIR`: absolute path to a custom dora directory.
* `AUDIOCRAFT_REFERENCE_DIR`: absolute path to the shared reference directory.
## Training pipelines
Each task supported in AudioCraft has its own training pipeline and dedicated solver.
Learn more about solvers and key designs around AudioCraft training pipeline below.
Please refer to the documentation of each task and model for specific information on a given task.
### Solvers
The core training component in AudioCraft is the solver. A solver holds the definition
of how to solve a given task: It implements the training pipeline logic, combining the datasets,
model, optimization criterion and components and the full training loop. We refer the reader
to [Flashy](https://github.com/facebookresearch/flashy) for core principles around solvers.
AudioCraft proposes an initial solver, the `StandardSolver` that is used as the base implementation
for downstream solvers. This standard solver provides a nice base management of logging,
checkpoints loading/saving, xp restoration, etc. on top of the base Flashy implementation.
In AudioCraft, we made the assumption that all tasks are following the same set of stages:
train, valid, evaluate and generation, each relying on a dedicated dataset.
Each solver is responsible for defining the task to solve and the associated stages
of the training loop in order to leave the full ownership of the training pipeline
to the researchers. This includes loading the datasets, building the model and
optimisation components, registering them and defining the execution of each stage.
To create a new solver for a given task, one should extend the StandardSolver
and define each stage of the training loop. One can further customise its own solver
starting from scratch instead of inheriting from the standard solver.
```
/* Detailed source-code truncated for AI context efficiency. */
```
### About Epochs
AudioCraft Solvers uses the concept of Epoch. One epoch doesn't necessarily mean one pass over the entire
dataset, but instead represent the smallest amount of computation that we want to work with before checkpointing.
Typically, we find that having an Epoch time around 30min is ideal both in terms of safety (checkpointing often enough)
and getting updates often enough. One Epoch is at least a `train` stage that lasts for `optim.updates_per_epoch` (2000 by default),
and a `valid` stage. You can control how long the valid stage takes with `dataset.valid.num_samples`.
Other stages (`evaluate`, `generate`) will only happen every X epochs, as given by `evaluate.every` and `generate.every`).
### Models
In AudioCraft, a model is a container object that wraps one or more torch modules together
with potential processing logic to use in a solver. For example, a model would wrap an encoder module,
a quantisation bottleneck module, a decoder and some tensor processing logic. Each of the previous components
can be considered as a small « model unit » on its own but the container model is a practical component
to manipulate and train a set of modules together.
### Datasets
See the [dedicated documentation on datasets](./DATASETS.md).
### Metrics
See the [dedicated documentation on metrics](./METRICS.md).
### Conditioners
AudioCraft language models can be conditioned in various ways and the codebase offers a modular implementation
of different conditioners that can be potentially combined together.
Learn more in the [dedicated documentation on conditioning](./CONDITIONING.md).
### Configuration
AudioCraft's configuration is defined in yaml files and the framework relies on
[hydra](https://hydra.cc/docs/intro/) and [omegaconf](https://omegaconf.readthedocs.io/) to parse
and manipulate the configuration through Dora.
##### :warning: Important considerations around configurations
Our configuration management relies on Hydra and the concept of group configs to structure
and compose configurations. Updating the root default configuration files will then have
an impact on all solvers and tasks.
**One should never change the default configuration files. Instead they should use Hydra config groups in order to store custom configuration.**
Once this configuration is created and used for running experiments, you should not edit it anymore.
Note that as we are using Dora as our experiment manager, all our experiment tracking is based on
signatures computed from delta between configurations.
**One must therefore ensure backward compatibility of the configuration at all time.**
See [Dora's README](https://github.com/facebookresearch/dora) and the
[section below introduction Dora](#running-experiments-with-dora).
##### Configuration structure
The configuration is organized in config groups:
* `conditioner`: default values for conditioning modules.
* `dset`: contains all data source related information (paths to manifest files
and metadata for a given dataset).
* `model`: contains configuration for each model defined in AudioCraft and configurations
for different variants of models.
* `solver`: contains the default configuration for each solver as well as configuration
for each solver task, combining all the above components.
* `teams`: contains the cluster configuration per teams. See environment setup for more details.
The `config.yaml` file is the main configuration that composes the above groups
and contains default configuration for AudioCraft.
##### Solver's core configuration structure
The core configuration structure shared across solver is available in `solvers/default.yaml`.
##### Other configuration modules
AudioCraft configuration contains the different setups we used for our research and publications.
## Running experiments with Dora
### Launching jobs
Try launching jobs for different tasks locally with dora run:
```shell
# run compression task with lightweight encodec
dora run solver=compression/debug
```
Most of the time, the jobs are launched through dora grids, for example:
```shell
# run compression task through debug grid
dora grid compression.debug
```
Learn more about running experiments with Dora below.
### A small introduction to Dora
[Dora](https://github.com/facebookresearch/dora) is the experiment manager tool used in AudioCraft.
Check out the README to learn how Dora works. Here is a quick summary of what to know:
* An XP is a unique set of hyper-parameters with a given signature. The signature is a hash
of those hyper-parameters. We always refer to an XP with its signature, e.g. 9357e12e. We will see
after that one can retrieve the hyper-params and re-rerun it in a single command.
* In fact, the hash is defined as a delta between the base config and the one obtained
with the config overrides you passed from the command line. This means you must never change
the `conf/**.yaml` files directly, except for editing things like paths. Changing the default values
in the config files means the XP signature won't reflect that change, and wrong checkpoints might be reused.
I know, this is annoying, but the reason is that otherwise, any change to the config file would mean
that all XPs ran so far would see their signature change.
#### Dora commands
```shell
dora info -f 81de367c # this will show the hyper-parameter used by a specific XP.
# Be careful some overrides might present twice, and the right most one
# will give you the right value for it.
dora run -d -f 81de367c # run an XP with the hyper-parameters from XP 81de367c.
# `-d` is for distributed, it will use all available GPUs.
dora run -d -f 81de367c dataset.batch_size=32 # start from the config of XP 81de367c but change some hyper-params.
# This will give you a new XP with a new signature (e.g. 3fe9c332).
dora info -f SIG -t # will tail the log (if the XP has scheduled).
# if you need to access the logs of the process for rank > 0, in particular because a crash didn't happen in the main
# process, then use `dora info -f SIG` to get the main log name (finished into something like `/5037674_0_0_log.out`)
# and worker K can be accessed as `/5037674_0_{K}_log.out`.
# This is only for scheduled jobs, for local distributed runs with `-d`, then you should go into the XP folder,
# and look for `worker_{K}.log` logs.
```
An XP runs from a specific folder based on its signature, under the
`//experiments/audiocraft/outputs/` folder.
You can safely interrupt a training and resume it, it will reuse any existing checkpoint,
as it will reuse the same folder. If you made some change to the code and need to ignore
a previous checkpoint you can use `dora run --clear [RUN ARGS]`.
If you have a Slurm cluster, you can also use the dora grid command, e.g.
```shell
# Run a dummy grid located at `audiocraft/grids/my_grid_folder/my_grid_name.py`
dora grid my_grid_folder.my_grid_name
# The following will simply display the grid and also initialize the Dora experiments database.
# You can then simply refer to a config using its signature (e.g. as `dora run -f SIG`).
dora grid my_grid_folder.my_grid_name --dry_run --init
```
Please refer to the [Dora documentation](https://github.com/facebookresearch/dora) for more information.
#### Clearing up past experiments
```shell
# This will cancel all the XPs and delete their folder and checkpoints.
# It will then reschedule them starting from scratch.
dora grid my_grid_folder.my_grid_name --clear
# The following will delete the folder and checkpoint for a single XP,
# and then run it afresh.
dora run [-f BASE_SIG] [ARGS] --clear
```
---
### WATERMARKING (`docs/WATERMARKING.md`)
# AudioSeal: Proactive Localized Watermarking
AudioCraft provides the training code and models for AudioSeal, a method for speech localized watermarking [Proactive Detection of Voice Cloning with Localized Watermarking][arxiv], with state-of-the-art robustness and detector speed. It jointly trains a generator that embeds a watermark in the audio, and a detector that detects the watermarked fragments in longer audios, even in the presence of editing.
## Installation and setup
Make sure to install audiocraft version `1.4.0a1` or later, and with the `[wm]` extra (see [README](../README.md)).
Alternatively, you can just install audioseal yourself. To install AudioSeal, follow [Installation](https://github.com/facebookresearch/audioseal) guidelines in the AudioSeal repo.
_NOTE_: Since we use AAC augmentation in our training loop, you need to install ffmpeg, or it will not work (See Section "Installation" in [README](../README.md)).
Make sure you follow [steps for basic training setup](TRAINING.md) before starting.
## API
Check the [Github repository](https://github.com/facebookresearch/audioseal) for more details.
## Training
The [WatermarkSolver](../audiocraft/solvers/watermark.py) implements the AudioSeal's training pipeline. It joins the generator and detector that wrap
`audioseal.AudioSealWM` and `audioseal.AudioSealDetector` respectively. For the training recipe, see [config/solver/watermark/robustness.yaml](../config/solver/watermark/robustness.yaml).
For illustration, we use the three example audios in `datasets`, with datasourc definition in [dset/audio/example.yaml](../config/dset/audio/example.yaml) (Please read [DATASET](./DATASETS.md) to understand AudioCraft's dataset structure.)
To run the Watermarking training pipeline locally:
```bash
dora run solver=watermark/robustness dset=audio/example
```
you can override model / experiment parameters here directly like:
```bash
dora run solver=watermark/robustness dset=audio/example sample_rate=24000
```
If you want to run in debug mode:
```bash
python3 -m pdb -c c -m dora run solver=watermark/robustness dset=audio/example
```
---
--- METRICS ---
- Files Extracted: 13
- Estimated Token Budget: ~25005 tokens
- Canonical Reference: https://codewiki.google/github.com/facebookresearch/audiocraft