Wav2Lip

Wav2Lip changes a face's mouth movements in an existing video to match supplied speech. Its research version is for noncommercial projects, including experiments with lip-sync generation and model training.

• Faces and voices: Match speech across different identities and languages, including CGI faces and synthetic voices.
• Audio inputs: Use WAV, MP3 or other audio formats supported by FFmpeg, or take the audio from another video.
• Model choice: Wav2Lip favors synchronization accuracy, while Wav2Lip + GAN trades some accuracy for better visual quality.
• Research training: Use the supplied training code to train models on a dataset, alongside pretrained models for generating videos.

The Python tool runs locally, with a Docker option available. Google Colab notebooks offer a separate cloud route. Sync Labs provides a separate commercial service through the Sync API, which requires an API key.

Face framing and input resolution affect the output. The models learned from lower-resolution faces, so higher-resolution footage doesn't always produce better results. Training on another dataset may require code changes and requires a lip-sync discriminator trained for that dataset. Fine-tuning with only a few minutes of one speaker may produce poor results.

This listing covers the original downloadable Wav2Lip model, not the separate hosted commercial service. The repository usage notice limits the local release to personal, research and noncommercial use. It remains available through source installation and pretrained checkpoints.