MMAudio

MMAudio generates sound for video footage and text prompts. It's for video creators adding audio to silent clips and researchers studying how generated sound matches on-screen action.

• Video sound generation: Create audio aligned with the timing of video frames, with text prompts to guide the result.
• Text-to-audio: Generate audio from a written prompt without supplying a video.
• Gradio interface: Use a browser interface for both video and text inputs.
• Training and evaluation: Researchers can use the included code to train models and evaluate their output.

The Python and PyTorch software runs locally on a GPU and has been tested on Ubuntu. Reported inference uses around 6GB of GPU memory in 16-bit mode. Hosted demos on Hugging Face, Colab and Replicate run outside your local setup.

The default output and training duration is eight seconds. Shorter or longer audio is possible, but quality may fall when the duration differs substantially. Speech-like sounds can be unintelligible, incidental background music may sound poor, and unfamiliar sound concepts can be difficult to reproduce.

The code is open source under the MIT license. The training datasets have separate licenses, and the project doesn't guarantee that its pretrained models are suitable for commercial use.