Introduction

Generating high-fidelity audio from simple text descriptions has long been a challenge due to the extreme length of raw audio sequences. AudioCraft, a PyTorch-based library developed by Meta AI (Facebook Research), solves this by providing a unified framework for audio generation and compression. With over 23,000 GitHub stars, AudioCraft allows developers and researchers to generate music, sound effects, and high-fidelity audio tokens using state-of-the-art models like MusicGen and AudioGen.

What Is AudioCraft?

AudioCraft is a deep learning research library for audio processing and generation that provides a single-stop codebase for all generative audio needs. It is written in Python and built on PyTorch, released under the MIT license for the code, while the model weights are released under the CC-BY-NC 4.0 license for research purposes.

The library simplifies the design of generative models by using a single autoregressive Language Model (LM) that operates over streams of compressed discrete audio tokens. This approach allows it to capture long-term dependencies in audio while maintaining high quality, making it a powerful tool for anyone from AI researchers to content creators.

Why AudioCraft Matters

Before AudioCraft, audio generation often relied on symbolic representations like MIDI or complex, cascaded models that were difficult to open-source and adapt. AudioCraft fills this gap by providing the “full recipe”—the training code, inference pipelines, and model weights—in a single, accessible codebase.

The project has gained massive traction because it democratizes access to high-fidelity audio generation. By integrating the EnCodec neural audio codec, AudioCraft enables the generation of raw waveforms directly from text, bypassing the need for traditional synthesizers or manual sound design. This makes it an essential tool for those looking to prototype musical ideas or create custom sound effects on demand.

Key Features

• MusicGen: A controllable text-to-music model that generates high-quality music samples from text descriptions or melody conditioning.
• AudioGen: A text-to-sound model focused on environmental sounds and sound effects, such as dogs barking or cars honking.
• EnCodec: A high-fidelity neural audio codec that compresses audio into discrete tokens, serving as the foundation for the other generative models.
• Multi-Band Diffusion: An EnCodec-compatible decoder that uses diffusion to reduce artifacts and improve the perceived quality of reconstructed audio.
• MAGNeT: A non-autoregressive model designed for faster text-to-music and text-to-sound generation.
• AudioSeal: A state-of-the-art audio watermarking tool used to identify and protect AI-generated audio.
• JASCO: A high-quality text-to-music model conditioned on chords, melodies, and drum tracks for precise temporal control.
• Unified Training Framework: Provides PyTorch components and solvers for developing and training custom audio generative models.
• Comprehensive Metrics: Includes built-in support for audio quality metrics like CLAP score, FAD, and Visqol.

How AudioCraft Compares

AudioCraft differs from projects like Google Magenta by focusing on raw waveform generation rather than symbolic MIDI output. While Magenta is excellent for real-time interactive jamming and MIDI-based composition, AudioCraft is a “studio-in-a-box” that produces finished audio assets. The primary tradeoff is compute: AudioCraft requires a GPU with significant VRAM (typically 16GB+ for larger models) to run efficiently, whereas Magenta’s smaller models can often run on a CPU.

Compared to commercial tools like Stable Audio, AudioCraft’s primary advantage is its transparency. Because the training pipelines and model weights are available, researchers can fine-tune the models on their own datasets, a capability not available in closed-source alternatives.

Getting Started: Installation

To use AudioCraft, you must have ffmpeg installed on your system. This is a critical prerequisite for audio processing.

Using pip (Stable Release)

python -m pip install -U audiocraft

Using pip (Bleeding Edge)

python -m pip install -U git+https://git@github.com/facebookresearch/audiocraft#egg=audiocraft

Local Installation (For Training)

If you intend to train models, you must clone the repository locally:

git clone https://github.com/facebookresearch/audiocraft
cd audiocraft
python -m pip install -e .

Prerequisites

It is highly recommended to install PyTorch 2.1.0 or newer with CUDA support for GPU acceleration. For those using Anaconda, ffmpeg can be installed via:

conda install "ffmpeg<5" -c conda-forge

How to Use AudioCraft

The simplest way to start is by using the MusicGen model to generate a short audio clip from a text prompt. The process involves loading a pre-trained model and passing a description of the music you want to create.

For those who prefer a visual interface, AudioCraft includes a Gradio-based web app. You can launch it by running the following command in your terminal:

python -m demos.musicgen_app --share

Once the app is running, you can enter your text prompt, select the model size (small, medium, or melody), and generate audio in your browser.

Code Examples

Below are examples of how to generate audio using the Python API. All examples assume you have the audiocraft.models module available.

Basic Text-to-Music Generation

from audiocraft.models import MusicGen

# Initialize the model
model = MusicGen.get_pretrained('facebook/musicgen-small')

# Generate audio from text
descriptions = ["upbeat electronic dance music with synths"]
wav = model.generate(descriptions)

This snippet loads the small version of MusicGen and generates a waveform based on a single text description.

Melody-Conditioned Generation

from audiocraft.models import MusicGen

# Load the melody model
model = MusicGen.get_pretrained('facebook/musicgen-melody')

# Generate music based on a text prompt and an existing melody
descriptions = ["lo-fi hip hop with vinyl crackle"]
wav = model.generate(descriptions, melody_wav=melody_audio_file)

In this example, the melody_wav parameter allows the model to follow the structure of an existing audio melody, while applying the style described in the text prompt.

Real-World Use Cases

AudioCraft is particularly effective in scenarios where custom, royalty-free audio is needed quickly without the need for a full studio production.

• Game Development: Indie developers can use AudioGen to create specific environmental sound effects (e.g., “footsteps on a wooden floor” or “distant thunder”) to enhance the atmosphere of their virtual worlds.
• Content Creation: YouTubers and podcasters can generate unique background music beds for their segments using MusicGen, ensuring they avoid copyright strikes while maintaining a specific mood.
• Musical Prototyping: Composers can use the melody-conditioned model to quickly test how a simple hummed melody would sound with different instrumentation (e.g., changing a piano melody to a synth-wave style).
• AI Research: Machine learning researchers can use the EnCodec framework to develop new audio compression algorithms or explore the boundaries of autoregressive audio generation.

Contributing to AudioCraft

AudioCraft is an open-source project that encourages contributions from the community. To contribute, users should fork the repository and create a feature branch from main. All new code should be accompanied by tests to ensure stability.

If you are modifying APIs, you must update the corresponding documentation. The project follows standard GitHub flow: report bugs via the Issues tab and submit improvements via Pull Requests. Contributors are also required to complete a Contributor License Agreement (CLA) before their changes are merged.

Community and Support

The primary hub for AudioCraft is the official GitHub repository, where developers can find the latest updates and model weights. Detailed API documentation is available on the Meta AI research site, and the project is hosted on Hugging Face for easy model loading.

For community support, users can utilize GitHub Discussions and the various research papers associated with the models (MusicGen, AudioGen, and EnCodec). The project is highly active with thousands of forks and hundreds of open issues, indicating a strong developer interest in the AI audio space.

Conclusion

AudioCraft is a transformative tool for anyone working with generative audio. By providing a unified, open-source framework for music and sound effect generation, Meta AI has lowered the barrier to entry for AI-driven sound creation. Whether you are a researcher pushing the limits of neural audio codecs or a creator needing a custom soundtrack, AudioCraft provides the professional-grade tools necessary to achieve high-fidelity results.

While the model weights are restricted to non-commercial use (CC-BY-NC 4.0), the flexibility of the library allows for significant innovation. We recommend that researchers start with the MusicGen-small model to test their hardware capabilities before moving to the larger, more complex models.

Star the repo, try the quickstart, and join the community to start creating AI audio today.

Resources

• AudioCraft GitHub Repository
• Official API Documentation
• Meta AI AudioCraft Announcement