Diffusion based video generation has received extensive attention and
achieved considerable success within both the academic and industrial
communities. However, current efforts are mainly concentrated on
single-objective or single-task video generation, such as generation driven by
text, by image, or by a combination of text and image. This cannot fully meet
the needs of real-world application scenarios, as users are likely to input
images and text conditions in a flexible manner, either individually or in
combination. To address this, we propose a Unified-modal Video Genearation
system that is capable of handling multiple video generation tasks across text
and image modalities. To this end, we revisit the various video generation
tasks within our system from the perspective of generative freedom, and
classify them into high-freedom and low-freedom video generation categories.
For high-freedom video generation, we employ Multi-condition Cross Attention to
generate videos that align with the semantics of the input images or text. For
low-freedom video generation, we introduce Biased Gaussian Noise to replace the
pure random Gaussian Noise, which helps to better preserve the content of the
input conditions. Our method achieves the lowest Fr\'echet Video Distance (FVD)
on the public academic benchmark MSR-VTT, surpasses the current open-source
methods in human evaluations, and is on par with the current close-source
method Gen2. For more samples, visit this https URL

提出了一种统一的多模态视频生成系统，能够处理基于文本和图像模态的多个视频生成任务，其中高自由度视频生成采用多条件交叉注意力对齐输入图像或文本的语义，低自由度视频生成引入偏置高斯噪声以更好地保留输入条件的内容，该方法在 MSR-VTT 公共学术基准中取得了最低的 Fréchet 视频距离（FVD），在人类评估中超过了当前的开源方法，并与当前的闭源方法 Gen2 持平。