Open-Sora Documentation

File: README.md Copy Chapter

Open-Sora: Democratizing Efficient Video Production for All Copy Chapter

We design and implement Open-Sora, an initiative dedicated to efficiently producing high-quality video. We hope to make the model,
tools and all details accessible to all. By embracing open-source principles,
Open-Sora not only democratizes access to advanced video generation techniques, but also offers a
streamlined and user-friendly platform that simplifies the complexities of video generation.
With Open-Sora, our goal is to foster innovation, creativity, and inclusivity within the field of content creation.

🎬 For a professional AI video-generation product, try Video Ocean — powered by a superior model.

📰 News Copy Chapter

• [2025.03.12] 🔥 We released Open-Sora 2.0 (11B). 🎬 11B model achieves on-par performance with 11B HunyuanVideo & 30B Step-Video on 📐VBench & 📊Human Preference. 🛠️ Fully open-source: checkpoints and training codes for training with only $200K. [report]
• [2025.02.20] 🔥 We released Open-Sora 1.3 (1B). With the upgraded VAE and Transformer architecture, the quality of our generated videos has been greatly improved 🚀. [checkpoints] [report] [demo]
• [2024.12.23] The development cost of video generation models has saved by 50%! Open-source solutions are now available with H200 GPU vouchers. [blog] [code] [vouchers]
• [2024.06.17] We released Open-Sora 1.2, which includes 3D-VAE, rectified flow, and score condition. The video quality is greatly improved. [checkpoints] [report] [arxiv]
• [2024.04.25] 🤗 We released the Gradio demo for Open-Sora on Hugging Face Spaces.
• [2024.04.25] We released Open-Sora 1.1, which supports 2s~15s, 144p to 720p, any aspect ratio text-to-image, text-to-video, image-to-video, video-to-video, infinite time generation. In addition, a full video processing pipeline is released. [checkpoints] [report]
• [2024.03.18] We released Open-Sora 1.0, a fully open-source project for video generation.
  Open-Sora 1.0 supports a full pipeline of video data preprocessing, training with
  
  acceleration,
  inference, and more. Our model can produce 2s 512x512 videos with only 3 days training. [checkpoints]
  [blog] [report]
• [2024.03.04] Open-Sora provides training with 46% cost reduction.
  [blog]

📍 Since Open-Sora is under active development, we remain different branches for different versions. The latest version is main. Old versions include: v1.0, v1.1, v1.2, v1.3.

🎥 Latest Demo Copy Chapter

Demos are presented in compressed GIF format for convenience. For original quality samples and their corresponding prompts, please visit our Gallery.

OpenSora 1.3 Demo

OpenSora 1.2 Demo

OpenSora 1.1 Demo

OpenSora 1.0 Demo

Videos are downsampled to .gif for display. Click for original videos. Prompts are trimmed for display,
see here for full prompts.

🔆 Reports Copy Chapter

• Tech Report of Open-Sora 2.0
• Step by step to train or finetune your own model
• Step by step to train and evaluate an video autoencoder
• Visit the high compression video autoencoder
• Reports of previous version (better see in according branch):
  • Open-Sora 1.3: shift-window attention, unified spatial-temporal VAE, etc.
  • Open-Sora 1.2, Tech Report: rectified flow, 3d-VAE, score condition, evaluation, etc.
  • Open-Sora 1.1: multi-resolution/length/aspect-ratio, image/video conditioning/editing, data preprocessing, etc.
  • Open-Sora 1.0: architecture, captioning, etc.

📍 Since Open-Sora is under active development, we remain different branches for different versions. The latest version is main. Old versions include: v1.0, v1.1, v1.2, v1.3.

Quickstart Copy Chapter

Installation Copy Section

# create a virtual env and activate (conda as an example)
conda create -n opensora python=3.10
conda activate opensora

# download the repo
git clone https://github.com/hpcaitech/Open-Sora
cd Open-Sora

# Ensure torch >= 2.4.0
pip install -v . # for development mode, `pip install -v -e .`
pip install xformers==0.0.27.post2 --index-url https://download.pytorch.org/whl/cu121 # install xformers according to your cuda version
pip install flash-attn --no-build-isolation

Optionally, you can install flash attention 3 for faster speed.

git clone https://github.com/Dao-AILab/flash-attention # 4f0640d5
cd flash-attention/hopper
python setup.py install

Model Download Copy Section

Our 11B model supports 256px and 768px resolution. Both T2V and I2V are supported by one model. 🤗 Huggingface 🤖 ModelScope.

Download from huggingface:

pip install "huggingface_hub[cli]"
huggingface-cli download hpcai-tech/Open-Sora-v2 --local-dir ./ckpts

Download from ModelScope:

pip install modelscope
modelscope download hpcai-tech/Open-Sora-v2 --local_dir ./ckpts

Text-to-Video Generation Copy Section

Our model is optimized for image-to-video generation, but it can also be used for text-to-video generation. To generate high quality videos, with the help of flux text-to-image model, we build a text-to-image-to-video pipeline. For 256x256 resolution:

# Generate one given prompt
torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/t2i2v_256px.py --save-dir samples --prompt "raining, sea"

# Save memory with offloading
torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/t2i2v_256px.py --save-dir samples --prompt "raining, sea" --offload True

# Generation with csv
torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/t2i2v_256px.py --save-dir samples --dataset.data-path assets/texts/example.csv

For 768x768 resolution:

# One GPU
torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/t2i2v_768px.py --save-dir samples --prompt "raining, sea"

# Multi-GPU with colossalai sp
torchrun --nproc_per_node 8 --standalone scripts/diffusion/inference.py configs/diffusion/inference/t2i2v_768px.py --save-dir samples --prompt "raining, sea"

You can adjust the generation aspect ratio by --aspect_ratio and the generation length by --num_frames. Candidate values for aspect_ratio includes 16:9, 9:16, 1:1, 2.39:1. Candidate values for num_frames should be 4k+1 and less than 129.

You can also run direct text-to-video by:

# One GPU for 256px
torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/256px.py --prompt "raining, sea"
# Multi-GPU for 768px
torchrun --nproc_per_node 8 --standalone scripts/diffusion/inference.py configs/diffusion/inference/768px.py --prompt "raining, sea"

Image-to-Video Generation Copy Section

Given a prompt and a reference image, you can generate a video with the following command:

# 256px
torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/256px.py --cond_type i2v_head --prompt "A plump pig wallows in a muddy pond on a rustic farm, its pink snout poking out as it snorts contentedly. The camera captures the pig's playful splashes, sending ripples through the water under the midday sun. Wooden fences and a red barn stand in the background, framed by rolling green hills. The pig's muddy coat glistens in the sunlight, showcasing the simple pleasures of its carefree life." --ref assets/texts/i2v.png

# 256px with csv
torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/256px.py --cond_type i2v_head --dataset.data-path assets/texts/i2v.csv

# Multi-GPU 768px
torchrun --nproc_per_node 8 --standalone scripts/diffusion/inference.py configs/diffusion/inference/768px.py --cond_type i2v_head --dataset.data-path assets/texts/i2v.csv

Advanced Usage Copy Chapter

Motion Score Copy Section

During training, we provide motion score into the text prompt. During inference, you can use the following command to generate videos with motion score (the default score is 4):

torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/t2i2v_256px.py --save-dir samples --prompt "raining, sea" --motion-score 4

We also provide a dynamic motion score evaluator. After setting your OpenAI API key, you can use the following command to evaluate the motion score of a video:

torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/t2i2v_256px.py --save-dir samples --prompt "raining, sea" --motion-score dynamic

Prompt Refine Copy Section

We take advantage of ChatGPT to refine the prompt. You can use the following command to refine the prompt. The function is available for both text-to-video and image-to-video generation.

export OPENAI_API_KEY=sk-xxxx
torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/t2i2v_256px.py --save-dir samples --prompt "raining, sea" --refine-prompt True

Reproductivity Copy Section

To make the results reproducible, you can set the random seed by:

torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/t2i2v_256px.py --save-dir samples --prompt "raining, sea" --sampling_option.seed 42 --seed 42

Use --num-sample k to generate k samples for each prompt.

Computational Efficiency Copy Chapter

We test the computational efficiency of text-to-video on H100/H800 GPU. For 256x256, we use colossalai's tensor parallelism, and --offload True is used. For 768x768, we use colossalai's sequence parallelism. All use number of steps 50. The results are presented in the format: $\color{blue}{\text{Total time (s)}}/\color{red}{\text{peak GPU memory (GB)}}$

Evaluation Copy Chapter

On VBench, Open-Sora 2.0 significantly narrows the gap with OpenAI’s Sora, reducing it from 4.52% → 0.69% compared to Open-Sora 1.2.

Human preference results show our model is on par with HunyuanVideo 11B and Step-Video 30B.

With strong performance, Open-Sora 2.0 is cost-effective.

Contribution Copy Chapter

Thanks goes to these wonderful contributors:

If you wish to contribute to this project, please refer to the Contribution Guideline.

Acknowledgement Copy Chapter

Here we only list a few of the projects. For other works and datasets, please refer to our report.

• ColossalAI: A powerful large model parallel acceleration and optimization
  system.
• DiT: Scalable Diffusion Models with Transformers.
• OpenDiT: An acceleration for DiT training. We adopt valuable acceleration
  strategies for training progress from OpenDiT.
• PixArt: An open-source DiT-based text-to-image model.
• Flux: A powerful text-to-image generation model.
• Latte: An attempt to efficiently train DiT for video.
• HunyuanVideo: Open-Source text-to-video model.
• StabilityAI VAE: A powerful image VAE model.
• DC-AE: Deep Compression AutoEncoder for image compression.
• CLIP: A powerful text-image embedding model.
• T5: A powerful text encoder.
• LLaVA: A powerful image captioning model based on Mistral-7B and Yi-34B.
• PLLaVA: A powerful video captioning model.
• MiraData: A large-scale video dataset with long durations and structured caption.

Citation Copy Chapter

@article{opensora,
  title={Open-sora: Democratizing efficient video production for all},
  author={Zheng, Zangwei and Peng, Xiangyu and Yang, Tianji and Shen, Chenhui and Li, Shenggui and Liu, Hongxin and Zhou, Yukun and Li, Tianyi and You, Yang},
  journal={arXiv preprint arXiv:2412.20404},
  year={2024}
}

@article{opensora2,
    title={Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k}, 
    author={Xiangyu Peng and Zangwei Zheng and Chenhui Shen and Tom Young and Xinying Guo and Binluo Wang and Hang Xu and Hongxin Liu and Mingyan Jiang and Wenjun Li and Yuhui Wang and Anbang Ye and Gang Ren and Qianran Ma and Wanying Liang and Xiang Lian and Xiwen Wu and Yuting Zhong and Zhuangyan Li and Chaoyu Gong and Guojun Lei and Leijun Cheng and Limin Zhang and Minghao Li and Ruijie Zhang and Silan Hu and Shijie Huang and Xiaokang Wang and Yuanheng Zhao and Yuqi Wang and Ziang Wei and Yang You},
    year={2025},
    journal={arXiv preprint arXiv:2503.09642},
}

Star History Copy Chapter

File: docs/zh_CN/report_v1.md Copy Chapter

Open-Sora v1 技术报告

OpenAI的Sora在生成一分钟高质量视频方面非常出色。然而，它几乎没有透露任何关于其细节的信息。为了使人工智能更加“开放”，我们致力于构建一个开源版本的Sora。这份报告描述了我们第一次尝试训练一个基于Transformer的视频扩散模型。

选择高效的架构 Copy Chapter

为了降低计算成本，我们希望利用现有的VAE模型。Sora使用时空VAE来减少时间维度。然而，我们发现没有开源的高质量时空VAE模型。MAGVIT的4x4x4 VAE并未开源，而VideoGPT的2x4x4 VAE在我们的实验中质量较低。因此，我们决定在我们第一个版本中使用2D VAE（来自Stability-AI）。

视频训练涉及大量的token。考虑到24fps的1分钟视频，我们有1440帧。通过VAE下采样4倍和patch大小下采样2倍，我们得到了1440x1024≈150万个token。在150万个token上进行全注意力计算将带来巨大的计算成本。因此，我们使用时空注意力来降低成本，这是遵循Latte的方法。

如图中所示，在STDiT（ST代表时空）中，我们在每个空间注意力之后立即插入一个时间注意力。这类似于Latte论文中的变种3。然而，我们并没有控制这些变体的相似数量的参数。虽然Latte的论文声称他们的变体比变种3更好，但我们在16x256x256视频上的实验表明，相同数量的迭代次数下，性能排名为：DiT（完整）> STDiT（顺序）> STDiT（并行）≈ Latte。因此，我们出于效率考虑选择了STDiT（顺序）。这里提供了速度基准测试。

为了专注于视频生成，我们希望基于一个强大的图像生成模型来训练我们的模型。PixArt-α是一个经过高效训练的高质量图像生成模型，具有T5条件化的DiT结构。我们使用PixArt-α初始化我们的模型，并将插入的时间注意力的投影层初始化为零。这种初始化在开始时保留了模型的图像生成能力，而Latte的架构则不能。插入的注意力将参数数量从5.8亿增加到7.24亿。

借鉴PixArt-α和Stable Video Diffusion的成功，我们还采用了渐进式训练策略：在366K预训练数据集上进行16x256x256的训练，然后在20K数据集上进行16x256x256、16x512x512和64x512x512的训练。通过扩展位置嵌入，这一策略极大地降低了计算成本。

我们还尝试在DiT中使用3D patch嵌入器。然而，在时间维度上2倍下采样后，生成的视频质量较低。因此，我们将在下一版本中将下采样留给时间VAE。目前，我们在每3帧采样一次进行16帧训练，以及在每2帧采样一次进行64帧训练。

数据是训练高质量模型的核心 Copy Chapter

我们发现数据的数量和质量对生成视频的质量有很大的影响，甚至比模型架构和训练策略的影响还要大。目前，我们只从HD-VG-130M准备了第一批分割（366K个视频片段）。这些视频的质量参差不齐，而且字幕也不够准确。因此，我们进一步从提供免费许可视频的Pexels收集了20k相对高质量的视频。我们使用LLaVA，一个图像字幕模型，通过三个帧和一个设计好的提示来标记视频。有了设计好的提示，LLaVA能够生成高质量的字幕。

由于我们更加注重数据质量，我们准备收集更多数据，并在下一版本中构建一个视频预处理流程。

训练细节 Copy Chapter

在有限的训练预算下，我们只进行了一些探索。我们发现学习率1e-4过大，因此将其降低到2e-5。在进行大批量训练时，我们发现fp16比bf16不太稳定，可能会导致生成失败。因此，我们在64x512x512的训练中切换到bf16。对于其他超参数，我们遵循了之前的研究工作。

损失曲线 Copy Chapter

16x256x256 预训练损失曲线

16x256x256 高质量训练损失曲线

16x512x512 高质量训练损失曲线

File: docs/zh_CN/report_v2.md Copy Chapter

Open-Sora 1.1 技术报告

• 模型架构修改
• 支持不同视频长度/分辨率/宽高比/帧率（fps）训练
• 使用Masked DiT作为图生视频/视频生视频模型
• 数据收集和流程
• 训练详情
• 结果和评价
• 不足和下一步计划

在Open-Sora1.1版本中，我们使用了10M数据来训练经过结构调优后的STDiT的700M模型（Open-Sora1.0版本仅用400K数据）。我们实现了Sora报告中提到的以下功能：

• 可变的视频时长、分辨率、宽高比（包括采样灵活性、改进的取景范围和构图）
• 提示词增加图片和视频选项（使图像动起来、生成式增长视频、视频到视频编辑、连接不同视频）
• 图像生成功能

为了实现这一目标，我们在预训练阶段使用了多任务学习。对于扩散模型来说，用不同的采样时间步长进行训练已经是一种多任务学习。我们将这一思想在图像和视频的条件生成模型上，进一步扩展到多分辨率、宽高比、帧长、fps以及不同的掩码策略。我们在0~15s、144p到720p、各种宽高比的视频上训练模型。虽然由于训练FLOPs不足的限制，生成的视频在时间一致性上的表现没有那么高，但我们仍然可以看到这个模型的巨大潜力。

模型架构修改 Copy Chapter

我们对原始ST-DiT模型进行了以下修改，以获得更好的训练稳定性和模型性能（ST-DiT-2）：

• 在时间注意力模块中添加旋转位置编码：遵循目前LLM的最佳实践，我们将时间注意力模块中的正弦位置编码更改为旋转位置编码，因为它也算一项序列预测任务。
• 在时间注意力模块中添加AdaIN和Layernormal：我们将时间注意力与AdaIN和Layer范数作为空间注意力包裹起来，以稳定训练。
• QK归一化与RMSNorm：和SD3类似地，我们应用QK归一化来提高半精度训练的稳定性。
• 支持动态输入大小和视频条件限定：为了支持多分辨率、宽高比和fps训练，我们ST-DiT-2来接受任何输入大小。延申PixArt-alpha的想法，我们支持限定视频的高度、宽度、宽高比、帧长和fps。
• 将T5token数量从120扩展到200：我们使用的视频描述通常少于200个token，我们发现模型也可以很好地处理更长的文本。

支持不同视频长度/分辨率/宽高比/帧率（fps）训练 Copy Chapter

正如Sora报告中提到的，使用原始无损视频的分辨率、宽高比和视频长度进行训练可以增加采样灵活性，改善取景和构图。我们找到了三种实现这一目标的方法：

• NaViT：通过不同掩码策略支持在同一训练批次内使用不同大小的数据，并且训练效率下降很少。然而，该系统实现起来有点复杂，并且可能无法兼容kernal优化技术（如flashattention）。
• 填充（FiT，Open-Sora-Plan）：通过填充支持同一批次内的不同大小的数据。然而，将不同的分辨率填充到相同的大小会导致效率降低。
• 分桶训练（SDXL、PixArt）：支持通过分桶的方式在不同批次中动态调整大小，但在同一批次内数据大小必须相同，只能应用固定数量的数据大小。在一个批次中，我们不需要实现复杂的掩码或填充。

为了更便捷的实现，我们选择分桶训练的方式。我们预先定义了一些固定的分辨率，并将不同的样本分配到不同的桶中。下面列出了分桶方案中值得注意的点。但我们可以看到，这些在我们的实验中并不是一个大问题。

查看注意事项

• 桶大小被限制为固定数量：首先，在实际应用中，通常只使用少数宽高比（9:16、3:4）和分辨率（240p、1080p）。其次，我们发现经过训练的模型可以很好地推广到未见过的解决方案。
• 每批的大小相同，打破了独立同分布（i.i.d.）假设：由于我们使用多个 GPU，因此不同 GPU 上的本地批次具有不同的大小。我们没有发现此问题导致性能显着下降。
• 可能没有足够的样本来填充每个桶，并且分布可能有偏差：首先，当本地批量大小不太大时，我们的数据集足够大以填充每个桶。其次，我们应该分析数据大小的分布并相应地定义桶大小。第三，分配不平衡并没有显着影响训练过程。
• 不同的分辨率和帧长可能有不同的处理速度：与PixArt只处理相似分辨率（相似token数）的宽高比不同，我们需要考虑不同分辨率和帧长的处理速度。我们可以使用“bucket_config”来定义每个桶的批量大小，以确保处理速度相似。

如图所示，桶是（分辨率，帧数量，宽高比）的三元组。我们为不同的分辨率提供预定义的宽高比，涵盖了大多数常见的视频宽高比。在每个epoch之前，我们打乱数据集并将样本分配到不同的桶中，如图所示。我们将样本放入最大分辨率和帧长度小于视频的桶中。

考虑到我们的计算资源有限，我们进一步为每个（分辨率，num_frame）二元组引入keep_prob和batch_size两个属性，以降低计算成本并实现多阶段训练。具体来说，高清视频将以概率1-keep_prob下采样到较低分辨率的桶中，并且每个桶的样本数量是由batch_size属性决定的。这样，我们可以控制不同桶中的样本数量，并通过为每个桶搜索合适的数据量来平衡GPU负载。

有关训练中桶使用的详细说明，请参阅配置文件.

使用Masked DiT作为图生视频/视频生视频模型 Copy Chapter

Transformer可以很容易地扩展到支持图生图和视频生视频的任务。我们提出了一种蒙版策略来支持图像和视频的调节。蒙版策略如下图所示。

在将图像或视频转换成另一个视频的过程中，我们通常会选择出需要作为条件的帧并取消其掩码（unmask）。在使用ST-DiT模型进行前向传播时，被选择取消掩码（unmask）的帧将被赋予时间步长0，而其他帧则保持它们原有的时间步长t。我们发现，如果直接将这种策略应用到训练好的模型上，会得到较差的结果，因为扩散模型在训练过程中并未学会如何处理一个样本中具有不同时间步长的帧。

受UL2的启发，我们在训练期间引入了随机掩码策略。具体来说，我们在训练期间随机取消掩码帧，包括取消掩码第一帧，前k帧，最后k帧，最后k帧，第一和最后k帧，随机帧等。基于Open-Sora 1.0模型，以50%的概率应用掩码策略，我们发现模型能够在10,000步的训练中学会处理图像条件（而30%的概率会导致处理能力变差），同时文本到视频的性能略有下降。因此，在Open-Sora 1.1版本中，我们从头开始预训练模型，并采用了掩码策略。

下图给出了用于推理的掩码策略配置的说明。五数字元组在定义掩码策略方面提供了极大的灵活性。

掩码策略用法的详细说明可在配置文件中查看.

数据收集和流程 Copy Chapter

正如我们在Sora1.0版本中看见的那样，数据数量和质量对于训练一个好的模型至关重要，因此，我们努力扩展数据集。首先，我们创建了一个遵循SVD的自动流水线，包括场景切割、字幕、各种评分和过滤以及数据集管理脚本和通用惯例。

我们计划使用panda-70M和其他数据来训练模型，大约包含3000万条数据。然而，我们发现磁盘输入输出（disk IO）在同时进行训练和数据处理时成为了一个瓶颈。因此，我们只能准备一个包含1000万条数据的数据集，并且没有完成我们构建的所有处理流程。最终，我们使用了包含970万视频和260万图像的数据集进行预训练，以及560,000视频和160万图像的数据集进行微调。预训练数据集的统计信息如下所示。

图像文本标记 (使用T5分词器)：

视频文本标记 (使用T5分词器)。我们直接使用Panda的短视频描述进行训练，并自己给其他数据集加视频描述。生成的字幕通常少于200个token。

视频时长：

训练详情 Copy Chapter

由于计算资源有限，我们必须仔细监控训练过程，并在推测模型学习不佳时更改训练策略，因为没有消融研究的计算。因此，Open-Sora1.1版本的训练包括多个更改，所以，指数移动平均（EMA）未被应用。

1. 首先，我们从Pixart-alpha-1024的模型checkpoint开始，使用不同分辨率的图像进行了6000步的微调。我们发现模型能够很容易地适应并生成不同分辨率的图像。为了加快扩散过程的训练，我们使用了SpeeDiT（iddpm-speed）技术。
2. [阶段一] 然后，我们使用梯度检查点（gradient-checkpointing）技术对模型进行了24,000步的预训练，这个过程在64个H800 GPU上运行了4天。尽管模型看到的数据样本数量相同，我们发现与使用较小批量大小相比，模型的学习速度较慢。我们推测，在训练的早期阶段，步数的数量对于训练更为重要。大多数视频的分辨率是240p，预训练时使用的配置与stage2.py相似。
3. [阶段一] 为了增加训练步数，我们改用了更小的批量大小，并且没有使用梯度检查点技术。在这个阶段，我们还引入了帧率（fps）条件。模型训练了40,000步，持续了2天。训练中使用的视频大多数是144p分辨率，使用的配置文件是stage1.py。我们使用较低的分辨率，因为我们在Open-Sora 1.0版本中发现模型可以以相对较低的分辨率学习时间知识。
4. [阶段一] 我们发现模型不能很好地学习长视频，并在Open-Sora1.0训练中发现了一个噪声生成结果，推测是半精度问题。因此，我们采用QK-归一化来稳定训练。我们还将iddpm-speed切换成iddpm。我们训练了17k步14小时。大多数视频的分辨率是144p，预训练时使用的配置是stage1.py。阶段1训练持续约一周，总步长81k。
5. [阶段二] 我们切换到更高的分辨率，其中大多数视频是240p和480p分辨率（stage2.py）。我们在所有预训练数据上训练了22000步，持续一天。
6. [阶段三] 我们切换到更高的分辨率，大多数视频的分辨率是480p和720p（stage3.py）。我们在高质量数据上训了4000步，用时一天。

结果和评价 Copy Chapter

不足和下一步计划 Copy Chapter

随着我们离Sora的复现又近了一步，我们发现当前模型存在许多不足，这些不足将在我们下阶段工作中得到改善。

• 噪音的生成和影响：我们发现生成的模型，特别是长视频中，有时很多噪点，不流畅。我们认为问题在于没有使用时间VAE。由于Pixart-Sigma发现适应新VAE很容易，我们计划在下一个版本中为模型开发时间VAE。
• 缺乏时间一致性：我们发现模型无法生成具有高时间一致性的视频，我们认为问题是由于缺乏训练FLOPs，我们计划收集更多数据并继续训练模型以提高时间一致性。
• 人像生成质量低：我们发现模型无法生成高质量的人类视频，我们认为问题是由于缺乏人类数据，我们计划收集更多的人类数据，并继续训练模型以提高人类生成。
• 美学得分低：我们发现模型的美学得分不高。问题在于缺少美学得分过滤，由于IO瓶颈没我们没有进行这一步骤。我们计划通过美学得分和微调模型来过滤数据，以提高美学得分。
• 长视频生成质量低：我们发现，使用同样的提示词，视频越长，质量越差。这意味着图像质量不能同等地被不同长度的序列所适应。

• 算法与加速实现：Zangwei Zheng, Xiangyu Peng, Shenggui Li, Hongxing Liu, Yukun Zhou
• 数据收集与处理：Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Young, Junjie Wang, Chenfeng Yu

File: docs/zh_CN/report_v3.md Copy Chapter

Open-Sora 1.2 报告

• 视频压缩网络
• 整流流和模型适应
• 更多数据和更好的多阶段训练
• 简单有效的模型调节
• 评估

在 Open-Sora 1.2 版本中，我们在 >30M 数据上训练了 一个1.1B 的模型，支持 0s~16s、144p 到 720p、各种宽高比的视频生成。我们的配置如下所列。继 1.1 版本之后，Open-Sora 1.2 还可以进行图像到视频的生成和视频扩展。

这里✅表示在训练期间可以看到数据，🆗表示虽然没有经过训练，但模型可以在该配置下进行推理。🆗的推理需要多个80G内存的GPU和序列并行。

除了 Open-Sora 1.1 中引入的功能外，Open-Sora 1.2 还有以下重磅更新：

• 视频压缩网络
• 整流流训练
• 更多数据和更好的多阶段训练
• 简单有效的模型调节
• 更好的评估指标

上述改进的所有实现（包括训练和推理）均可在 Open-Sora 1.2 版本中使用。以下部分将介绍改进的细节。我们还改进了代码库和文档，使其更易于使用。

视频压缩网络 Copy Chapter

对于 Open-Sora 1.0 & 1.1，我们使用了 stable-ai 的 83M 2D VAE，它仅在空间维度上压缩，将视频压缩 8x8 倍。为了减少时间维度，我们每三帧提取一帧。然而，这种方法导致生成的视频流畅度较低，因为牺牲了生成的帧率（fps）。因此，在这个版本中，我们引入了像 OpenAI 的 Sora 一样的视频压缩网络。该网络在时域上将视频大小压缩至四分之一，因此，我们不必再额外抽帧，而可以使用原有帧率生成模型。

考虑到训练 3D VAE 的计算成本很高，我们希望重新利用在 2D VAE 中学到的知识。我们注意到，经过 2D VAE 压缩后，时间维度上相邻的特征仍然高度相关。因此，我们提出了一个简单的视频压缩网络，首先将视频在空间维度上压缩 8x8 倍，然后将视频在时间维度上压缩 4 倍。网络如下所示：

我们用SDXL 的 VAE初始化 2D VAE ，它比我们以前使用的更好。对于 3D VAE，我们采用Magvit-v2中的 VAE 结构，它包含 300M 个参数。加上 83M 的 2D VAE，视频压缩网络的总参数为 384M。我们设定batch size 为 1， 对 3D VAE 进行了 1.2M 步的训练。训练数据是来自 pixels 和 pixabay 的视频，训练视频大小主要是 17 帧，256x256 分辨率。3D VAE 中使用causal convolotions使图像重建更加准确。

我们的训练包括三个阶段：

1. 对于前 380k 步，我们冻结 2D VAE并在 8 个 GPU 上进行训练。训练目标包括重建 2D VAE 的压缩特征（图中粉红色），并添加损失以使 3D VAE 的特征与 2D VAE 的特征相似（粉红色和绿色，称为identity loss）。我们发现后者的损失可以快速使整个 VAE 在图像上取得良好的性能，并在下一阶段更快地收敛。
2. 对于接下来的 260k 步，我们消除identity loss并仅学习 3D VAE。
3. 对于最后 540k 步，由于我们发现仅重建 2D VAE 的特征无法带来进一步的改进，因此我们移除了loss并训练整个 VAE 来重建原始视频。此阶段在 24 个 GPU 上进行训练。

对于训练的前半部分，我们采用 20% 的图像和 80% 的视频。按照Magvit-v2，我们使用 17 帧训练视频，同时对图像的前 16 帧进行零填充。然而，我们发现这种设置会导致长度不同于 17 帧的视频变得模糊。因此，在第 3 阶段，我们使用不超过34帧长度的任意帧长度视频进行混合视频长度训练,以使我们的 VAE 对不同视频长度更具鲁棒性（也就是说，如果我们希望训练含有n帧的视频，我们就把原视频中34-n帧用0进行填充）。我们的 训练和推理代码可在 Open-Sora 1.2 版本中找到。

当使用 VAE 进行扩散模型时，我们的堆叠 VAE 所需的内存较少，因为我们的 VAE 的输入已经经过压缩。我们还将输入视频拆分为几个 17 帧剪辑，以提高推理效率。我们的 VAE 与Open-Sora-Plan中的另一个开源 3D VAE 性能相当。

整流流和模型适应 Copy Chapter

最新的扩散模型 Stable Diffusion 3 为了获得更好的性能，采用了rectified flow替代了 DDPM。可惜 SD3 的 rectified flow 训练代码没有开源。不过 Open-Sora 1.2 提供了遵循 SD3 论文的训练代码，包括：

• 基本整流流训练
• 用于训练加速的 Logit-norm 采样
• 分辨率和视频长度感知时间步长采样

对于分辨率感知的时间步长采样，我们应该对分辨率较大的图像使用更多的噪声。我们将这个想法扩展到视频生成，对长度较长的视频使用更多的噪声。

Open-Sora 1.2 从PixArt-Σ 2K 模型checkpoint开始。请注意，此模型使用 DDPM 和 SDXL VAE 进行训练，分辨率也高得多。我们发现在小数据集上进行微调可以轻松地使模型适应我们的视频生成设置。适应过程如下，所有训练都在 8 个 GPU 上完成：

1. 多分辨率图像生成能力：我们训练模型以 20k 步生成从 144p 到 2K 的不同分辨率。
2. QK-norm：我们将 QK-norm 添加到模型中并训练 18k 步。
3. 整流流：我们从离散时间 DDPM 转变为连续时间整流流并训练 10k 步。
4. 使用 logit-norm 采样和分辨率感知时间步采样的整流流：我们训练 33k 步。
5. 较小的 AdamW epsilon：按照 SD3，使用 QK-norm，我们可以对 AdamW 使用较小的 epsilon（1e-15），我们训练 8k 步。
6. 新的 VAE 和 fps 调节：我们用自己的 VAE 替换原来的 VAE，并将 fps 调节添加到时间步调节中，我们训练 25k 步。请注意，对每个通道进行规范化对于整流流训练非常重要。
7. 时间注意力模块：我们添加时间注意力模块，其中没有初始化投影层。我们在图像上进行 3k 步训练。
8. 仅针对具有掩码策略的视频的时间块：我们仅在视频上训练时间注意力块，步长为 38k。

经过上述调整后，我们就可以开始在视频上训练模型了。上述调整保留了原始模型生成高质量图像的能力，并未后续的视频生成提供了许多助力：

• 通过整流，我们可以加速训练，将视频的采样步数从100步减少到30步，大大减少了推理的等待时间。
• 使用 qk-norm，训练更加稳定，并且可以使用积极的优化器。
• 采用新的VAE，时间维度压缩了4倍，使得训练更加高效。
• 该模型具有多分辨率图像生成能力，可以生成不同分辨率的视频。

更多数据和更好的多阶段训练 Copy Chapter

由于计算预算有限，我们精心安排了训练数据的质量从低到高，并将训练分为三个阶段。我们的训练涉及 12x8 GPU，总训练时间约为 2 周， 约70k步。

第一阶段 Copy Section

我们首先在 Webvid-10M 数据集（40k 小时）上训练模型，共 30k 步（2 个 epoch）。由于视频分辨率均低于 360p 且包含水印，因此我们首先在此数据集上进行训练。训练主要在 240p 和 360p 上进行，视频长度为 2s~16s。我们使用数据集中的原始字幕进行训练。训练配置位于stage1.py中。

第二阶段 Copy Section

然后我们在 Panda-70M 数据集上训练模型。这个数据集很大，但质量参差不齐。我们使用官方的 30M 子集，其中的片段更加多样化，并过滤掉美学评分低于 4.5 的视频。这产生了一个 20M 子集，包含 41k 小时。数据集中的字幕直接用于我们的训练。训练配置位于stage2.py中。

训练主要在 360p 和 480p 上进行。我们训练模型 23k 步，即 0.5 个 epoch。训练尚未完成，因为我们希望我们的新模型能早日与大家见面。

第三阶段 Copy Section

在此阶段，我们从各种来源收集了 200 万个视频片段，总时长 5000 小时，其中包括：

• 来自 Pexels、Pixabay、Mixkit 等的免费授权视频。
• MiraData：一个包含长视频的高质量数据集，主要来自游戏和城市/风景探索。
• Vript：一个密集注释的数据集。
• 还有一些其他数据集。

MiraData 和 Vript 有来自 GPT 的字幕，而我们使用PLLaVA为其余字幕添加字幕。与只能进行单帧/图像字幕的 LLaVA 相比，PLLaVA 是专门为视频字幕设计和训练的。加速版PLLaVA已在我们的tools/中发布。在实践中，我们使用预训练的 PLLaVA 13B 模型，并从每个视频中选择 4 帧生成字幕，空间池化形状为 2*2。

下面显示了此阶段使用的视频数据的一些统计数据。我们提供了持续时间和分辨率的基本统计数据，以及美学分数和光流分数分布。我们还从视频字幕中提取了对象和动作的标签并计算了它们的频率。

此阶段我们主要在 720p 和 1080p 上进行训练，以提高模型在高清视频上的表现力。在训练中，我们使用的掩码率为25%。训练配置位于stage3.py中。我们对模型进行 15k 步训练，大约为 2 个 epoch。

简单有效的模型调节 Copy Chapter

对于第 3 阶段，我们计算每个视频片段的美学分数和运动分数。但是，由于视频片段数量较少，我们不愿意过滤掉得分较低的片段，这会导致数据集较小。相反，我们将分数附加到字幕中并将其用作条件。我们发现这种方法可以让模型了解分数并遵循分数来生成质量更好的视频。

例如，一段美学评分为 5.5、运动评分为 10 且检测到摄像头运动向左平移的视频，其字幕将为：

[Original Caption] aesthetic score: 5.5, motion score: 10, camera motion: pan left.

在推理过程中，我们还可以使用分数来调节模型。对于摄像机运动，我们仅标记了 13k 个具有高置信度的剪辑，并且摄像机运动检测模块已在我们的工具中发布。

评估 Copy Chapter

之前，我们仅通过人工评估来监控训练过程，因为 DDPM 训练损失与生成的视频质量没有很好的相关性。但是，对于校正流，如 SD3 中所述，我们发现训练损失与生成的视频质量有很好的相关性。因此，我们跟踪了 100 张图像和 1k 个视频的校正流评估损失。

我们从 pixabay 中抽样了 1k 个视频作为验证数据集。我们计算了不同分辨率（144p、240p、360p、480p、720p）下图像和不同长度的视频（2s、4s、8s、16s）的评估损失。对于每个设置，我们等距采样 10 个时间步长。然后对所有损失取平均值。

此外，我们还会在训练过程中跟踪VBench得分。VBench 是用于短视频生成的自动视频评估基准。我们用 240p 2s 视频计算 vbench 得分。这两个指标验证了我们的模型在训练过程中持续改进。

所有评估代码均发布在eval文件夹中。查看评估指南了解更多详细信息。

序列并行 Copy Chapter

我们使用序列并行来支持长序列训练和推理。我们的实现基于Ulysses，工作流程如下所示。启用序列并行后，我们只需要将 all-to-all 通信应用于STDiT中的空间模块（spatial block），因为在序列维度上，只有对空间信息的计算是相互依赖的。

目前，由于训练数据分辨率较小，我们尚未使用序列并行进行训练，我们计划在下一个版本中使用。至于推理，我们可以使用序列并行，以防您的 GPU 内存不足。下表显示，序列并行可以实现加速：

File: docs/zh_CN/report_v4.md Copy Chapter

Open-Sora 1.3 报告

• 视频压缩网络
• 升级版带位移窗口注意力的STDiT
• 简单有效的模型条件控制
• 评估方法

在Open-Sora 1.3版本中，我们在超过60M（约85k小时）的数据上训练了一个1.1B参数的模型，训练耗时35k H100 GPU小时，支持0~113帧、360p和720p分辨率以及多种宽高比的视频生成。我们的配置如下。延续1.2版本的特性，Open-Sora 1.3同样支持图像到视频的生成和视频延展。

这里✅表示在训练过程中已经见过的数据。

除了Open-Sora 1.2中引入的特性外，Open-Sora 1.3的亮点包括：

• 视频压缩网络
• 升级版带位移窗口注意力的STDiT
• 更多数据和更好的多阶段训练
• 简单有效的模型条件控制
• 更好的评估指标

以上所有改进的实现（包括训练和推理）都在Open-Sora 1.3版本中提供。以下部分将详细介绍这些改进。我们还优化了代码库和文档以使其更易于使用和开发，并添加了LLM优化器来优化输入提示词并支持更多语言。

视频压缩网络 Copy Chapter

在Open-Sora 1.2中，视频压缩架构采用了模块化方法，分别处理空间和时间维度。基于Stability AI的SDXL VAE的空间VAE压缩单个帧的空间维度。时间VAE则处理来自空间VAE的潜在表示以实现时间压缩。这种两阶段设计实现了有效的空间和时间压缩，但也带来了一些限制。这些限制包括由于固定长度输入帧而导致的长视频处理效率低下、空间和时间特征之间缺乏无缝集成，以及在训练和推理过程中更高的内存需求。

Open-Sora 1.3引入了统一的视频压缩方法。通过将空间和时间处理结合到单一框架中，并利用诸如分块3D卷积和动态帧支持等高级特性，Open-Sora 1.3实现了更好的效率、可扩展性和重建质量。以下是Open-Sora 1.3 VAE的主要改进：

1. 统一的时空处理： 不同于使用独立的VAE进行空间和时间压缩，Open-Sora 1.3采用单一的编码器-解码器结构同时处理这两个维度。这种方法消除了中间表示和空间-时间模块之间的冗余数据传输的需求。

2. 分块3D卷积： Open-Sora 1.3在时间维度上引入了分块3D卷积支持。通过将视频分解成更小的时间块，该特性实现了对更长视频序列的高效编码和解码，而不会增加内存开销。这一改进解决了Open-Sora 1.2在处理大量帧时的限制，确保了更高的时间压缩灵活性。

3. 动态微批次和微帧处理： Open-Sora 1.3引入了新的微批次和微帧处理机制。这实现了：(1) 自适应时间重叠：时间编码和解码过程中的重叠帧帮助减少块边界的不连续性。(2) 动态帧大小支持：不再局限于固定长度序列（如Open-Sora 1.2中的17帧），Open-Sora 1.3支持动态序列长度，使其能够适应不同的视频长度。

4. 统一的归一化机制： Open-Sora 1.3中的归一化过程通过可调的缩放(scale)和平移(shift)参数得到了改进，确保了不同数据集间潜在空间分布的一致性。与Open-Sora 1.2特定于固定数据集的归一化不同，这个版本引入了更通用的参数并支持特定于帧的归一化策略。

改进总结

包含滑动窗口注意力的STDiT Copy Chapter

在Open-Sora 1.2取得成功的基础上，1.3版本引入了多项架构改进和新功能，以提升视频生成的质量和灵活性。本节概述了这两个版本之间的主要改进和差异。

最新的扩散模型（如Stable Diffusion 3）采用rectified flow代替DDPM以获得更好的性能。虽然SD3的rectified flow训练代码未开源，但OpenSora按照SD3论文提供了训练代码实现。OpenSora 1.2从SD3引入了几个关键策略：

1. 基础的rectified flow训练，实现连续时间扩散
2. Logit-norm采样用于加速训练（遵循SD3论文第3.1节），优先在中等噪声水平采样时间步
3. 分辨率和视频长度感知的时间步采样（遵循SD3论文第5.3.2节），对更大分辨率和更长视频使用更多噪声

在OpenSora 1.3中，我们在架构、功能和性能方面进行了显著改进：

1. 位移窗口注意力机制

• 引入可配置kernel_size的基于核的局部注意力，提高计算效率
• 实现类似Swin Transformer的位移窗口分区策略
• 增加带extra_pad_on_dims支持的窗口边界填充掩码处理
• 在局部窗口（时间、高度、宽度）内扩展3D相对位置编码

2. 增强的位置编码

• 改进RoPE实现，将rotation_dim降至原来的1/3以适应3D场景
• 为时间、高度和宽度维度添加独立的旋转嵌入
• 实现分辨率自适应的位置编码缩放
• 可选的空间RoPE以更好地建模空间关系

3. 灵活的生成能力

• 添加I2V和V2V功能，配备专门的条件控制机制
• 引入条件嵌入模块（x_embedder_cond和x_embedder_cond_mask）
• 零初始化条件嵌入以实现稳定训练
• 通过skip_temporal选项实现灵活的时序建模

4. 性能优化

• 改进Flash Attention触发条件（N > 128）以提高效率
• 添加torch.scaled_dot_product_attention (SDPA)作为替代后端
• 通过改进的填充和窗口分区优化内存使用
• 通过自适应高度填充增强序列并行性

从PixArt-Σ 2K的适应过程保持相似，但增加了额外步骤：
[第1-7点与v1.2相同：多分辨率训练、QK-norm、rectified flow、logit-norm采样、更小的AdamW epsilon、新VAE和基础时序注意力]

8. 增强的时序模块

• 添加带位移窗口支持的基于核的局部注意力
• 实现带分辨率自适应缩放的3D相对位置编码
• 采用改进的初始化策略进行投影层零初始化

相比专注于基础视频生成的v1.2，v1.3在三个关键领域带来了实质性改进：1. 质量：通过位移窗口注意力和3D位置编码增强时空建模。2. 灵活性：支持I2V/V2V任务和可配置的时序建模。3. 效率：优化注意力计算和内存使用

这些改进在保持v1.2核心功能的同时，扩展了模型在实际应用中的能力。模型保留了使用rectified flow生成高质量图像和视频的能力，同时在条件生成和长序列建模方面获得了新的优势。

简单有效的模型条件控制 Copy Chapter

我们对每个视频片段计算美学分数和运动分数，并过滤掉得分较低的片段，从而得到一个视频质量更好的数据集。此外，我们将这些分数附加到标题中并用作条件控制。具体来说，我们基于预定义的范围将数值分数转换为描述性语言。美学分数转换函数基于预定义范围将数值美学分数转换为描述标签：低于4分标记为"terrible"，依次通过"very poor"、"poor"、"fair"、"good"和"very good"，6.5分或更高标记为"excellent"。同样，运动分数转换函数将运动强度分数映射为描述符：低于0.5分标记为"very low"，依次通过"low"、"fair"、"high"和"very high"，20分或更高标记为"extremely high"。我们发现这种方法可以使模型意识到这些分数并遵循分数来生成更高质量的视频。

例如，对于一个美学分数为5.5，运动分数为10，检测到的相机运动为向左平移的视频，其标题将是：

[Original Caption] The aesthetic score is good, the motion strength is high, camera motion: pan left.

在推理过程中，我们也可以使用这些分数来控制模型。对于相机运动，我们只标记了13k个高置信度的片段，相机运动检测模块已在我们的工具中发布。

评估方法 Copy Chapter

此前，我们仅通过人工评估来监控训练过程，因为DDPM训练损失与生成视频的质量相关性不高。然而，对于rectified flow，我们发现正如SD3所述，训练损失与生成视频的质量有很好的相关性。因此，我们持续跟踪100张图像和1k个视频的rectified flow评估损失。

我们从pixabay采样了1k个视频作为验证数据集。我们计算了不同分辨率（360p，720p）下图像和不同长度视频（49帧、65帧、81帧、97帧、113帧）的评估损失。对于每种设置，我们等距采样10个时间步。然后对所有损失取平均值。

此外，我们还在训练期间跟踪VBench分数。VBench是一个用于短视频生成的自动视频评估基准。我们使用360p 49帧视频计算vbench分数。这两个指标验证了我们的模型在训练过程中持续改进。

所有评估代码都在eval文件夹中发布。查看README获取更多详细信息。

File: docs/ae.md Copy Chapter

Step by step to train and evaluate an video autoencoder (AE)

Inspired by SANA, we aim to drastically increase the compression ratio in the AE. We propose a video autoencoder architecture based on DC-AE, the Video DC-AE, which compression the video by 4x in the temporal dimension and 32x32 in the spatial dimension. Compared to HunyuanVideo's VAE of 4x8x8, our proposed AE has a much higher spatial compression ratio.
Thus, we can effectively reduce the token length in the diffusion model by a total of 16x (assuming the same patch sizes), drastically increase both training and inference speed.

Data Preparation Copy Chapter

Follow this guide to prepare the DATASET for training and inference. You may use our provided dataset or custom ones.

To use custom dataset, pass the argument --dataset.data_path <your_data_path> to the following training or inference command.

Training Copy Chapter

We train our Video DC-AE from scratch on 8xGPUs for 3 weeks.

We first train with the following command:

torchrun --nproc_per_node 8 scripts/vae/train.py configs/vae/train/video_dc_ae.py

When the model is almost converged, we add a discriminator and continue to train the model with the checkpoint model_ckpt using the following command:

torchrun --nproc_per_node 8 scripts/vae/train.py configs/vae/train/video_dc_ae_disc.py --model.from_pretrained <model_ckpt>

You may pass the flag --wandb True if you have a wandb account and wish to track the training progress online.

Inference Copy Chapter

Download the relevant weights following this guide. Alternatively, you may use your own trained model by passing the following flag --model.from_pretrained <your_model_ckpt_path>.

Video DC-AE Copy Section

Use the following code to reconstruct the videos using our trained Video DC-AE:

torchrun --nproc_per_node 1 --standalone scripts/vae/inference.py configs/vae/inference/video_dc_ae.py --save-dir samples/dcae

Hunyuan Video Copy Section

Alternatively, we have incorporated HunyuanVideo vae into our code, you may run inference with the following command:

torchrun --nproc_per_node 1 --standalone scripts/vae/inference.py configs/vae/inference/hunyuanvideo_vae.py --save-dir samples/hunyuanvideo_vae

Config Interpretation Copy Chapter

All AE configs are located in configs/vae/, divided into configs for training (configs/vae/train) and for inference (configs/vae/inference).

Training Config Copy Section

For training, the same config rules as those for the diffusion model are applied.

Loss Config
Our Video DC-AE is based on the DC-AE architecture, which doesn't have a variational component. Thus, our training simply composes of the reconstruction loss and the perceptual loss.
Experimentally, we found that setting a ratio of 0.5 for the perceptual loss is effective.

vae_loss_config = dict(
    perceptual_loss_weight=0.5, # weigh the perceptual loss by 0.5
    kl_loss_weight=0,           # no KL loss
)

In a later stage, we include a discriminator, and the training loss for the ae has an additional generator loss component, where we use a small ratio of 0.05 to weigh the loss calculated:

gen_loss_config = dict(
    gen_start=0,                # include generator loss from step 0 onwards          
    disc_weight=0.05,           # weigh the loss by 0.05
)

The discriminator we use is trained from scratch, and it's loss is simply the hinged loss:

disc_loss_config = dict(
    disc_start=0,               # update the discriminator from step 0 onwards
    disc_loss_type="hinge",     # the discriminator loss type
)

** Data Bucket Config **
For the data bucket, we used 32 frames of 256px videos to train our AE.

bucket_config = {
    "256px_ar1:1": {32: (1.0, 1)},
}

Train with more frames or higher resolutions

If you train with longer frames or larger resolutions, you may increase the spatial_tile_size and temporal_tile_size during inference without degrading the AE performance (see Inference Config). This may give you advantage of faster AE inference such as when training the diffusion model (although at the cost of slower AE training).

You may increase the video frames to 96 (although multiples of 4 works, we generally recommend to use frame numbers of multiples of 32):

bucket_config = {
    "256px_ar1:1": {96: (1.0, 1)},
}
grad_checkpoint = True

or train for higher resolution such as 512px:

bucket_config = {
    "512px_ar1:1": {32: (1.0, 1)},
}
grad_checkpoint = True

Note that gradient checkpoint needs to be turned on in order to avoid prevent OOM error.

Moreover, if grad_checkpointing is set to True in discriminator training, you need to pass the flag --model.disc_off_grad_ckpt True or simply set in the config:

grad_checkpoint = True
model = dict(
    disc_off_grad_ckpt = True, # set to true if your `grad_checkpoint` is True
)

This is to make sure the discriminator loss will have a gradient at the laster later during adaptive loss calculation.

Inference Config Copy Section

For AE inference, we have replicated the tiling mechanism in hunyuan to our Video DC-AE, which can be turned on with the following:

model = dict(
    ...,
    use_spatial_tiling=True,
    use_temporal_tiling=True,
    spatial_tile_size=256,
    temporal_tile_size=32,
    tile_overlap_factor=0.25,
    ...,
)

By default, both spatial tiling and temporal tiling are turned on for the best performance.
Since our Video DC-AE is trained on 256px videos of 32 frames only, spatial_tile_size should be set to 256 and temporal_tile_size should be set to 32.
If you train your own Video DC-AE with other resolutions and length, you may adjust the values accordingly.

You can specify the directory to store output samples with --save_dir <your_dir> or setting it in config, for instance:

save_dir = "./samples"

File: docs/hcae.md Copy Chapter

10× inference speedup with high-compression autoencoder

The high computational cost of training video generation models arises from the
large number of tokens and the dominance of attention computation. To further reduce training expenses,
we explore training video generation models with high-compression autoencoders (Video DC-AEs). As shown in the comparason below, by switching to the Video DC-AE with a much higher downsample ratio (4 x 32 x 32), we can afford to further reduce the patch size to 1 and still achieve 5.2× speedup in training throughput and 10x speedup during inference:

Nevertheless, despite the advantanges in drastically lower computation costs, other challenges remain. For instance, larger channels low down convergance. Our generation model adapted with a 128-channel Video DC-AE for 25K iterations achieves a loss level of 0.5, as compared to 0.1 from the initialization model. While the fast video generation model underperforms the original, it still captures spatial-temporal
relationships. We release this model to the research community for further exploration.

Checkout more details in our report.

Model Download Copy Chapter

Download from 🤗 Huggingface:

pip install "huggingface_hub[cli]"
huggingface-cli download hpcai-tech/Open-Sora-v2-Video-DC-AE --local-dir ./ckpts

Inference Copy Chapter

To inference on our fast video generation model:

torchrun --nproc_per_node 1 --standalone scripts/diffusion/inference.py configs/diffusion/inference/high_compression.py --prompt "The story of a robot's life in a cyberpunk setting."

Training Copy Chapter

Follow this guide to parepare the DATASET for training.
Then, you may train your own fast generation model with the following command:

torchrun --nproc_per_node 8 scripts/diffusion/train.py configs/diffusion/train/high_compression.py --dataset.data-path datasets/pexels_45k_necessary.csv

File: docs/report_01.md Copy Chapter

Open-Sora 1.0 Report

OpenAI's Sora is amazing at generating one minutes high quality videos. However, it reveals almost no information about its details. To make AI more "open", we are dedicated to build an open-source version of Sora. This report describes our first attempt to train a transformer-based video diffusion model.

Efficiency in choosing the architecture Copy Chapter

To lower the computational cost, we want to utilize existing VAE models. Sora uses spatial-temporal VAE to reduce the temporal dimensions. However, we found that there is no open-source high-quality spatial-temporal VAE model. MAGVIT's 4x4x4 VAE is not open-sourced, while VideoGPT's 2x4x4 VAE has a low quality in our experiments. Thus, we decided to use a 2D VAE (from Stability-AI) in our first version.

The video training involves a large amount of tokens. Considering 24fps 1min videos, we have 1440 frames. With VAE downsampling 4x and patch size downsampling 2x, we have 1440x1024≈1.5M tokens. Full attention on 1.5M tokens leads to a huge computational cost. Thus, we use spatial-temporal attention to reduce the cost following Latte.

As shown in the figure, we insert a temporal attention right after each spatial attention in STDiT (ST stands for spatial-temporal). This is similar to variant 3 in Latte's paper. However, we do not control a similar number of parameters for these variants. While Latte's paper claims their variant is better than variant 3, our experiments on 16x256x256 videos show that with same number of iterations, the performance ranks as: DiT (full) > STDiT (Sequential) > STDiT (Parallel) ≈ Latte. Thus, we choose STDiT (Sequential) out of efficiency. Speed benchmark is provided here.

To focus on video generation, we hope to train the model based on a powerful image generation model. PixArt-α is an efficiently trained high-quality image generation model with T5-conditioned DiT structure. We initialize our model with PixArt-α and initialize the projection layer of inserted temporal attention with zero. This initialization preserves model's ability of image generation at beginning, while Latte's architecture cannot. The inserted attention increases the number of parameter from 580M to 724M.

Drawing from the success of PixArt-α and Stable Video Diffusion, we also adopt a progressive training strategy: 16x256x256 on 366K pretraining datasets, and then 16x256x256, 16x512x512, and 64x512x512 on 20K datasets. With scaled position embedding, this strategy greatly reduces the computational cost.

We also try to use a 3D patch embedder in DiT. However, with 2x downsampling on temporal dimension, the generated videos have a low quality. Thus, we leave the downsampling to temporal VAE in our next version. For now, we sample at every 3 frames with 16 frames training and every 2 frames with 64 frames training.

Data is the key to high quality Copy Chapter

We find that the number and quality of data have a great impact on the quality of generated videos, even larger than the model architecture and training strategy. At this time, we only prepared the first split (366K video clips) from HD-VG-130M. The quality of these videos varies greatly, and the captions are not that accurate. Thus, we further collect 20k relatively high quality videos from Pexels, which provides free license videos. We label the video with LLaVA, an image captioning model, with three frames and a designed prompt. With designed prompt, LLaVA can generate good quality of captions.

As we lay more emphasis on the quality of data, we prepare to collect more data and build a video preprocessing pipeline in our next version.

Training Details Copy Chapter

With a limited training budgets, we made only a few exploration. We find learning rate 1e-4 is too large and scales down to 2e-5. When training with a large batch size, we find fp16 less stable than bf16 and may lead to generation failure. Thus, we switch to bf16 for training on 64x512x512. For other hyper-parameters, we follow previous works.

Loss curves Copy Chapter

16x256x256 Pretraining Loss Curve

16x256x256 HQ Training Loss Curve

16x512x512 HQ Training Loss Curve

Core Contributor: Zangwei Zheng*, Xiangyu Peng*, Shenggui Li, Hongxing Liu, Yang You

File: docs/report_02.md Copy Chapter

Open-Sora 1.1 Report

• Model Architecture Modification
• Support for Multi-time/resolution/aspect ratio/fps Training
• Masked DiT as Image/Video-to-Video Model
• Data Collection & Pipeline
• Training Details
• Limitation and Future Work

In Open-Sora 1.1 release, we train a 700M models on 10M data (Open-Sora 1.0 trained on 400K data) with a better STDiT architecture. We implement the following features mentioned in sora's report:

• Variable durations, resolutions, aspect ratios (Sampling flexibility, Improved framing and composition)
• Prompting with images and videos (Animating images, Extending generated videos, Video-to-video editing, Connecting videos)
• Image generation capabilities

To achieve this goal, we use multi-task learning in the pretraining stage. For diffusion models, training with different sampled timestep is already a multi-task learning. We further extend this idea to multi-resolution, aspect ratio, frame length, fps, and different mask strategies for image and video conditioned generation. We train the model on 0s~15s, 144p to 720p, various aspect ratios videos. Although the quality of time consistency is not that high due to limit training FLOPs, we can still see the potential of the model.

Model Architecture Modification Copy Chapter

We made the following modifications to the original ST-DiT for better training stability and performance (ST-DiT-2):

• Rope embedding for temporal attention: Following LLM's best practice, we change the sinusoidal positional encoding to rope embedding for temporal attention since it is also a sequence prediction task.
• AdaIN and Layernorm for temporal attention: we wrap the temporal attention with AdaIN and layernorm as the spatial attention to stabilize the training.
• QK-normalization with RMSNorm: Following SD3, we apply QK-normalization to the all attention for better training stability in half-precision.
• Dynamic input size support and video infomation condition: To support multi-resolution, aspect ratio, and fps training, we make ST-DiT-2 to accept any input size, and automatically scale positional embeddings. Extending PixArt-alpha's idea, we conditioned on video's height, width, aspect ratio, frame length, and fps.
• Extending T5 tokens from 120 to 200: our caption is usually less than 200 tokens, and we find the model can handle longer text well.

Support for Multi-time/resolution/aspect ratio/fps Training Copy Chapter

As mentioned in the sora's report, training with original video's resolution, aspect ratio, and length increase sampling flexibility and improve framing and composition. We found three ways to achieve this goal:

• NaViT: support dynamic size within the same batch by masking, with little efficiency loss. However, the system is a bit complex to implement, and may not benefit from optimized kernels such as flash attention.
• Padding (FiT, Open-Sora-Plan): support dynamic size within the same batch by padding. However, padding different resolutions to the same size is not efficient.
• Bucket (SDXL, PixArt): support dynamic size in different batches by bucketing, but the size must be the same within the same batch, and only a fixed number of size can be applied. With the same size in a batch, we do not need to implement complex masking or padding.

For the simplicity of implementation, we choose the bucket method. We pre-define some fixed resolution, and allocate different samples to different bucket. The concern for bucketing is listed below. But we can see that the concern is not a big issue in our case.

View the concerns

• The bucket size is limited to a fixed number: First, in real-world applications, only a few aspect ratios (9:16, 3:4) and resolutions (240p, 1080p) are commonly used. Second, we find trained models can generalize well to unseen resolutions.
• The size in each batch is the same, breaks the i.i.d. assumption: Since we are using multiple GPUs, the local batches on different GPUs have different sizes. We did not see a significant performance drop due to this issue.
• The may not be enough samples to fill each bucket and the distribution may be biased: First, our dataset is large enough to fill each bucket when local batch size is not too large. Second, we should analyze the data's distribution on sizes and define the bucket size accordingly. Third, an unbalanced distribution did not affect the training process significantly.
• Different resolutions and frame lengths may have different processing speed: Different from PixArt, which only deals with aspect ratios of similar resolutions (similar token numbers), we need to consider the processing speed of different resolutions and frame lengths. We can use the bucket_config to define the batch size for each bucket to ensure the processing speed is similar.

As shown in the figure, a bucket is a triplet of (resolution, num_frame, aspect_ratio). We provide pre-defined aspect ratios for different resolution that covers most of the common video aspect ratios. Before each epoch, we shuffle the dataset and allocate the samples to different buckets as shown in the figure. We put a sample into a bucket with largest resolution and frame length that is smaller than the video's.

Considering our computational resource is limited, we further introduce two attributes keep_prob and batch_size for each (resolution, num_frame) to reduce the computational cost and enable multi-stage training. Specifically, a high-resolution video will be downsampled to a lower resolution with probability 1-keep_prob and the batch size for each bucket is batch_size. In this way, we can control the number of samples in different buckets and balance the GPU load by search a good batch size for each bucket.

A detailed explanation of the bucket usage in training is available in docs/config.md.

Masked DiT as Image/Video-to-Video Model Copy Chapter

Transformers can be easily extended to support image-to-image and video-to-video tasks. We propose a mask strategy to support image and video conditioning. The mask strategy is shown in the figure below.

Typically, we unmask the frames to be conditioned on for image/video-to-video condition. During the ST-DiT forward, unmasked frames will have timestep 0, while others remain the same (t). We find directly apply the strategy to trained model yield poor results as the diffusion model did not learn to handle different timesteps in one sample during training.

Inspired by UL2, we introduce random mask strategy during training. Specifically, we randomly unmask the frames during training, including unmask the first frame, the first k frames, the last frame, the last k frames, the first and last k frames, random frames, etc. Based on Open-Sora 1.0, with 50% probability of applying masking, we see the model can learn to handle image conditioning (while 30% yields worse ability) for 10k steps, with a little text-to-video performance drop. Thus, for Open-Sora 1.1, we pretrain the model from scratch with masking strategy.

An illustration of masking strategy config to use in inference is given as follow. A five number tuple provides great flexibility in defining the mask strategy. By conditioning on generated frames, we can autogressively generate infinite frames (although error propagates).

A detailed explanation of the mask strategy usage is available in docs/config.md.

Data Collection & Pipeline Copy Chapter

As we found in Open-Sora 1.0, the data number and quality are crucial for training a good model, we work hard on scaling the dataset. First, we create an automatic pipeline following SVD, inlcuding scene cutting, captioning, various scoring and filtering, and dataset management scripts and conventions. More infomation can be found in docs/data_processing.md.

We plan to use panda-70M and other data to traing the model, which is approximately 30M+ data. However, we find disk IO a botteleneck for training and data processing at the same time. Thus, we can only prepare a 10M dataset and did not go through all processing pipeline that we built. Finally, we use a dataset with 9.7M videos + 2.6M images for pre-training, and 560k videos + 1.6M images for fine-tuning. The pretraining dataset statistics are shown below. More information about the dataset can be found in docs/datasets.md.

Image text tokens (by T5 tokenizer):

Video text tokens (by T5 tokenizer). We directly use panda's short caption for training, and caption other datasets by ourselves. The generated caption is usually less than 200 tokens.

Video duration:

Training Details Copy Chapter

With limited computational resources, we have to carefully monitor the training process, and change the training strategy if we speculate the model is not learning well since there is no computation for ablation study. Thus, Open-Sora 1.1's training includes multiple changes, and as a result, ema is not applied.

1. First, we fine-tune 6k steps with images of different resolution from Pixart-alpha-1024 checkpoints. We find the model easily adapts to generate images with different resolutions. We use SpeeDiT (iddpm-speed) to accelerate the diffusion training.
2. [Stage 1] Then, we pretrain the model with gradient-checkpointing for 24k steps, which takes 4 days on 64 H800 GPUs. Although the number of samples seen by the model is the same, we find the model learns slowly compared to a smaller batch size. We speculate that at an early stage, the number of steps is more important for training. The most videos are in 240p resolution, and the config is similar to stage2.py. The video looking is good, but the model does not know much about the temporal knowledge. We use mask ratio of 10%.
3. [Stage 1] To increase the number of steps, we switch to a smaller batch size without gradient-checkpointing. We also add fps conditioning at this point. We trained 40k steps for 2 days. The most videos are in 144p resolution, and the config file is stage1.py. We use a lower resolution as we find in Open-Sora 1.0 that the model can learn temporal knowledge with relatively low resolution.
4. [Stage 1] We find the model cannot learn well for long videos, and find a noised generation result as speculated to be half-precision problem found in Open-Sora 1.0 training. Thus, we adopt the QK-normalization to stabilize the training. Similar to SD3, we find the model quickly adapt to the QK-normalization. We also switch iddpm-speed to iddpm, and increase the mask ratio to 25% as we find image-condition not learning well. We trained for 17k steps for 14 hours. The most videos are in 144p resolution, and the config file is stage1.py. The stage 1 training lasts for approximately one week, with total step 81k.
5. [Stage 2] We switch to a higher resolution, where most videos are in 240p and 480p resolution (stage2.py). We trained 22k steps for one day on all pre-training data.
6. [Stage 3] We switch to a higher resolution, where most videos are in 480p and 720p resolution (stage3.py). We trained 4k with one day on high-quality data. We find loading previous stage's optimizer state can help the model learn faster.

To summarize, the training of Open-Sora 1.1 requires approximately 9 days on 64 H800 GPUs.

Limitation and Future Work Copy Chapter

As we get one step closer to the replication of Sora, we find many limitations for the current model, and these limitations point to the future work.

• Generation Failure: we find many cases (especially when the total token number is large or the content is complex), our model fails to generate the scene. There may be a collapse in the temporal attention and we have identified a potential bug in our code. We are working hard to fix it. Besides, we will increase our model size and training data to improve the generation quality in the next version.
• Noisy generation and influency: we find the generated model is sometimes noisy and not fluent, especially for long videos. We think the problem is due to not using a temporal VAE. As Pixart-Sigma finds that adapting to a new VAE is simple, we plan to develop a temporal VAE for the model in the next version.
• Lack of time consistency: we find the model cannot generate videos with high time consistency. We think the problem is due to the lack of training FLOPs. We plan to collect more data and continue training the model to improve the time consistency.
• Bad human generation: We find the model cannot generate high-quality human videos. We think the problem is due to the lack of human data. We plan to collect more human data and continue training the model to improve the human generation.
• Low aesthetic score: we find the model's aesthetic score is not high. The problem is due to the lack of aesthetic score filtering, which is not conducted due to IO bottleneck. We plan to filter the data by aesthetic score and finetuning the model to improve the aesthetic score.
• Worse quality for longer video generation: we find with a same prompt, the longer video has worse quality. This means the image quality is not equally adapted to different lengths of sequences.

• Algorithm & Acceleration: Zangwei Zheng, Xiangyu Peng, Shenggui Li, Hongxing Liu, Yukun Zhou, Tianyi Li
• Data Collection & Pipeline: Xiangyu Peng, Zangwei Zheng, Chenhui Shen, Tom Young, Junjie Wang, Chenfeng Yu

File: docs/report_03.md Copy Chapter

Open-Sora 1.2 Report

• Video compression network
• Rectified flow and model adaptation
• More data and better multi-stage training
• Easy and effective model conditioning
• Evaluation
• Sequence parallelism

In Open-Sora 1.2 release, we train a 1.1B models on >30M data (~80k hours), with training cost 35k H100 GPU hours, supporting 0s~16s, 144p to 720p, various aspect ratios video generation. Our configurations is listed below. Following our 1.1 version, Open-Sora 1.2 can also do image-to-video generation and video extension.

Here ✅ means that the data is seen during training, and 🆗 means although not trained, the model can inference at that config. Inference for 🆗 requires more than one 80G memory GPU and sequence parallelism.

Besides features introduced in Open-Sora 1.1, Open-Sora 1.2 highlights:

• Video compression network
• Rectifie-flow training
• More data and better multi-stage training
• Easy and effective model conditioning
• Better evaluation metrics

All implementations (both training and inference) of the above improvements are available in the Open-Sora 1.2 release. The following sections will introduce the details of the improvements. We also refine our codebase and documentation to make it easier to use and develop, and add a LLM to refine input prompts and support more languages.

Video compression network Copy Chapter

For Open-Sora 1.0 & 1.1, we used stability-ai's 83M 2D VAE, which compress the video only in the spatial dimension by 8x8 times. To reduce the temporal dimension, we extracted one frame in every three frames. However, this method led to the low fluency of generated video as the generated fps is sacrificed. Thus, in this release, we introduce the video compression network as OpenAI's Sora does. With a 4 times compression in the temporal dimension, we do not need to extract frames and can generate videos with the original fps.

Considering the high computational cost of training a 3D VAE, we hope to re-use the knowledge learnt in the 2D VAE. We notice that after 2D VAE's compression, the features adjacent in the temporal dimension are still highly correlated. Thus, we propose a simple video compression network, which first compress the video in the spatial dimension by 8x8 times, then compress the video in the temporal dimension by 4x times. The network is shown below:

We initialize the 2D VAE with SDXL's VAE, which is better than our previously used one. For the 3D VAE, we adopt the structure of VAE in Magvit-v2, which contains 300M parameters. Along with 83M 2D VAE, the total parameters of the video compression network is 384M. We train the 3D VAE for 1.2M steps with local batch size 1. The training data is videos from pixels and pixabay, and the training video size is mainly 17 frames, 256x256 resolution. Causal convolutions are used in the 3D VAE to make the image reconstruction more accurate.

Our training involves three stages:

1. For the first 380k steps, we train on 8 GPUs and freeze the 2D VAE. The training objective includes the reconstruction of the compressed features from 2D VAE (pink one in the figure) and also add a loss to make features from the 3D VAE similar to the features from the 2D VAE (pink one and green one, called identity loss). We find the latter loss can quickly make the whole VAE achieve a good performance for image and much faster to converge in the next stage.
2. For the next 260k steps, We remove the identity loss and just learn the 3D VAE.
3. For the last 540k steps , since we find only reconstruction 2D VAE's feature cannot lead to further improvement, we remove the loss and train the whole VAE to reconstruct the original videos. This stage is trained on on 24 GPUs.

For both stage 1 and stage 2 training, we adopt 20% images and 80% videos. Following Magvit-v2, we train video using 17 frames, while zero-padding the first 16 frames for image. However, we find that this setting leads to blurring of videos with length different from 17 frames. Thus, in stage 3, we use a random number within 34 frames for mixed video length training (a.k.a., zero-pad the first 43-n frames if we want to train a n frame video), to make our VAE more robust to different video lengths. Our training and inference code is available in the Open-Sora 1.2 release.

When using the VAE for diffusion model, our stacked VAE requires small memory as the our VAE's input is already compressed. We also split the input videos input several 17 frames clips to make the inference more efficient. The performance of our VAE is on par with another open-sourced 3D VAE in Open-Sora-Plan.

Rectified flow and model adaptation Copy Chapter

Lastest diffusion model like Stable Diffusion 3 adopts the rectified flow instead of DDPM for better performance. Pitiably, SD3's rectified flow training code is not open-sourced. However, Open-Sora 1.2 provides the training code following SD3's paper, including:

• Basic rectified flow training (original rectified flow paper)
• Logit-norm sampling for training acceleration (SD3 paper Section 3.1, intuitively it is more likely to sample timesteps at middle noise level)
• Resolution and video length aware timestep sampling (SD3 paper Section 5.3.2, intuitively it is more likely to sample timesteps with more noise for larger resolution, and we extend it to longer video)

For the resolution-aware timestep sampling, we should use more noise for images with larger resolution. We extend this idea to video generation and use more noise for videos with longer length.

Open-Sora 1.2 starts from the PixArt-Σ 2K checkpoint. Note that this model is trained with DDPM and SDXL VAE, also a much higher resolution. We find finetuning on a small dataset can easily adapt the model for our video generation setting. The adaptation process is as follows, all training is done on 8 GPUs (the adaptation for the diffusion model is quite fast and straightforward):

1. Multi-resolution image generation ability: we train the model to generate different resolution ranging from 144p to 2K for 20k steps.
2. QK-norm: we add the QK-norm to the model and train for 18k steps.
3. Rectified flow: we transform from discrete-time DDPM to continuous-time rectified flow and train for 10k steps.
4. Rectified flow with logit-norm sampling and resolution-aware timestep sampling: we train for 33k steps.
5. Smaller AdamW epsilon: following SD3, with QK-norm, we can use a smaller epsilon (1e-15) for AdamW, we train for 8k steps.
6. New VAE and fps conditioning: we replace the original VAE with ours and add fps conditioning to the timestep conditioning, we train for 25k steps. Note that normalizing each channel is important for rectified flow training.
7. Temporal attention blocks: we add temporal attention blocks with zero initialized projection layers. We train on images for 3k steps.
8. Temporal blocks only for video with mask strategy: we train the temporal attention blocks only on videos for 38k steps.

After the above adaptation, we are ready to train the model on videos. The adaptation above maintains the original model's ability to generate high-quality images, and brings multiple benefits for video generation:

• With rectified flow, we can accelerate the training and reduce the number of sampling steps for video from 100 to 30, which greatly reduces the waiting time for inference.
• With qk-norm, the training is more stablized and an aggressive optimizer can be used.
• With new VAE, the temporal dimension is compressed by 4 times, which makes the training more efficient.
• With multi-resolution image generation ability, the model can generate videos with different resolutions.

More data and better multi-stage training Copy Chapter

Due to a limited computational budget, we carefully arrange the training data from low to high quality and split our training into three stages. Our training involves 12x8 GPUs, and the total training time is about 2 weeks for about 70k steps.

First stage Copy Section

We first train the model on Webvid-10M datasets (40k hours) for 30k steps (2 epochs). Since the video is all lower than 360p resolution and contains watermark, we train on this dataset first. The training mainly happens on 240p and 360p, with video length 2s~16s. We use the original caption in the dataset for training. The training config locates in stage1.py.

Second stage Copy Section

Then we train the model on Panda-70M datasets. This dataset is large but the quality varies. We use the official 30M subset which clips are more diverse, and filter out videos with aesthetic score lower than 4.5. This leads to a 20M subset with 41k hours. The captions in the dataset are directly used for our training. The training config locates in stage2.py.

The training mainly happens on 360p and 480p. We train the model for 23k steps, which is 0.5 epoch. The training is not fully done since we hope our new model can meet you earlier.

Third stage Copy Section

In this stage, we collect ~2M video clips with a total length of 5K hours from all kinds of sources, including:

• Free-license videos, sourced from Pexels, Pixabay, Mixkit, etc.
• MiraData: a high-quality dataset with long videos, mainly from games and city/scenic exploration.
• Vript: a densely annotated dataset.
• And some other datasets.

While MiraData and Vript have captions from GPT, we use PLLaVA to caption the rest ones. Compared with LLaVA, which is only capable of single frame/image captioning, PLLaVA is specially designed and trained for video captioning. The accelerated PLLaVA is released in our tools/. In practice, we use the pretrained PLLaVA 13B model and select 4 frames from each video for captioning with a spatial pooling shape of 2*2.

Some statistics of the video data used in this stage are shown below. We present basic statistics of duration and resolution, as well as aesthetic score and optical flow score distribution.
We also extract tags for objects and actions from video captions and count their frequencies.

We mainly train 720p and 1080p videos in this stage, aiming to extend the model's ability to larger resolutions. We use a mask ratio of 25% during training. The training config locates in stage3.py. We train the model for 15k steps, which is approximately 2 epochs.

Easy and effective model conditioning Copy Chapter

For stage 3, we calculate the aesthetic score and motion score for each video clip. However, since the number of video clips is small, we are not willing to filter out clips with low scores, which leads to a smaller dataset. Instead, we append the scores to the captions and use them as conditioning. We find this method can make model aware of the scores and follows the scores to generate videos with better quality.

For example, a video with aesthetic score 5.5, motion score 10, and a detected camera motion pan left, the caption will be:

[Original Caption] aesthetic score: 5.5, motion score: 10, camera motion: pan left.

During inference, we can also use the scores to condition the model. For camera motion, we only label 13k clips with high confidence, and the camera motion detection module is released in our tools.

Evaluation Copy Chapter

Previously, we monitor the training process only by human evaluation, as DDPM traning loss is not well correlated with the quality of generated videos. However, for rectified flow, we find the training loss is well correlated with the quality of generated videos as stated in SD3. Thus, we keep track of rectified flow evaluation loss on 100 images and 1k videos.

We sampled 1k videos from pixabay as validation dataset. We calculate the evaluation loss for image and different lengths of videos (2s, 4s, 8s, 16s) for different resolution (144p, 240p, 360p, 480p, 720p). For each setting, we equidistantly sample 10 timesteps. Then all the losses are averaged. We also provide a video showing the sampled videos with a fixed prompt for different steps.

In addition, we also keep track of VBench scores during training. VBench is an automatic video evaluation benchmark for short video generation. We calcuate the vbench score with 240p 2s videos. The two metrics verify that our model continues to improve during training.

All the evaluation code is released in eval folder. Check the README for more details.

Sequence parallelism Copy Chapter

We use sequence parallelism to support long-sequence training and inference. Our implementation is based on Ulysses and the workflow is shown below. When sequence parallelism is enabled, we only need to apply the all-to-all communication to the spatial block in STDiT as only spatial computation is dependent on the sequence dimension.

Currently, we have not used sequence parallelism for training as data resolution is small and we plan to do so in the next release. As for inference, we can use sequence parallelism in case your GPU goes out of memory. A simple benchmark shows that sequence parallelism can achieve speedup