LatentSync

LatentSync is an open-source AI lip-sync tool for video creators and researchers who want a face's mouth movements to match supplied audio. It processes video on your own GPU, with demos covering filmed faces and anime characters.

• Video lip-sync: Uses audio-conditioned diffusion based on Stable Diffusion to generate mouth movements that follow the supplied audio.
• Visual and scripted processing: A Gradio browser interface supports visual video processing, while a command-line interface supports scripted workflows.
• Custom training: Training code comes with a footage preparation pipeline that splits scenes, aligns faces, corrects audio timing and filters clips for quality and synchronization.
• Sync evaluation: SyncNet tools measure synchronization confidence in generated videos and model accuracy on datasets; evaluation data must go through the supplied preparation pipeline.

The code uses the Apache 2.0 license, and model checkpoints are available through Hugging Face. Inference needs at least 8 GB or 18 GB of VRAM, depending on the model. More generation steps improve visual quality but take longer. Stronger audio guidance improves synchronization but may cause distortion or jitter.

An efficient training setup needs 20 GB of VRAM and may slightly reduce visual quality and consistency between frames compared with the 30 GB setup.