Open-Sora

Open-Sora is an AI video generation project for researchers, developers and creators who want to run or adapt a model on their own GPUs. It focuses on animating reference images with text guidance and also accepts text alone.

• Text and image inputs: One model handles both input types, with a Flux pipeline that can create an image from a text prompt before animating it.
• Motion control: A motion score lets you influence movement in generated clips, and a separate evaluator measures their motion.
• Repeatable generation: Seed controls support reproducible results, while multiple samples let you compare different outputs for the same prompt.
• Model training: The project includes tools for training and fine-tuning the video model, plus training and evaluating its video autoencoder.
• Optional prompt refinement: ChatGPT can refine prompts for either input type, but this feature sends prompts to an external service.

The Python code is open source under Apache 2.0. It uses ColossalAI for work across multiple GPUs and supports offloading to reduce GPU memory use. Published performance tests use NVIDIA H100 and H800 GPUs; the single-GPU test at the smaller output size reports 52.5 GB of peak GPU memory.