Open Sora Plan

Open Sora Plan is a video generation project for researchers and developers who want to work with model checkpoints on their own hardware. It aims to reproduce capabilities of OpenAI's Sora, with text-to-video and image-to-video models.

• Text-to-video: Generate clips from written prompts with checkpoints that support different video dimensions and lengths.
• Image-to-video: Use image-conditioned checkpoints to generate a video from a supplied image.
• Video model research: Study WFVAE video compression, which reduces diffusion transformer training costs, and SUV sparse attention, which the project reports runs faster with performance close to a dense diffusion transformer.
• Prompt refinement: The model collection includes a prompt refiner alongside generation checkpoints.

The Python code uses the MIT license. The main text-to-video model was trained entirely on Huawei Ascend hardware, and the project provides an NPU usage path. Input types and supported dimensions depend on the checkpoint you choose. Some checkpoints trained on Panda70M may produce watermarks because they haven't received final fine-tuning on higher-quality data.