MuseTalk

MuseTalk is a local AI lip-sync model for video creators and developers building virtual avatars. It edits a face to match supplied speech, so you can dub existing footage or animate a portrait.

• Video and image inputs: Use a video, still image or image sequence as the visual input alongside an audio file, including videos generated by MuseV.
• Multilingual dubbing: Match mouth movements to speech in languages including Chinese, English and Japanese.
• Real-time avatars: Run real-time inference, with reported performance above 30 frames per second on an NVIDIA Tesla V100.
• Face preview: A local Gradio browser interface lets you preview the first frame and adjust the face region before generating the full result.
• Custom training: Training code and data preparation tools let developers train a model on their own video datasets.

MuseTalk runs on Windows and Linux with an NVIDIA GPU and uses downloaded model weights. The reported real-time performance depends on the hardware; training requires substantially more GPU memory, with the supplied examples using NVIDIA H20 GPUs.

Facial edits cover a 256-by-256 region. You can combine MuseTalk with GFPGAN for higher-resolution output, but results can still show jitter or alter details such as mustaches, lip shape and lip color.

MuseTalk 1.5 is distributed through the repository and official model downloads. Its licensing notice permits academic and commercial use of the MIT-licensed code and trained MuseTalk model. Supporting models retain their own licenses; the supplied test data is for noncommercial research.