Diffusion models have achieved great success in image generation. However,
when leveraging this idea for video generation, we face significant challenges
in maintaining the consistency and continuity across video frames. This is
mainly caused by the lack of an effective framework to align frames of videos
with desired temporal features while preserving consistent semantic and
stochastic features. In this work, we propose a novel Sector-Shaped Diffusion
Model (S2DM) whose sector-shaped diffusion region is formed by a set of
ray-shaped reverse diffusion processes starting at the same noise point. S2DM
can generate a group of intrinsically related data sharing the same semantic
and stochastic features while varying on temporal features with appropriate
guided conditions. We apply S2DM to video generation tasks, and explore the use
of optical flow as temporal conditions. Our experimental results show that S2DM
outperforms many existing methods in the task of video generation without any
temporal-feature modelling modules. For text-to-video generation tasks where
temporal conditions are not explicitly given, we propose a two-stage generation
strategy which can decouple the generation of temporal features from
semantic-content features. We show that, without additional training, our model
integrated with another temporal conditions generative model can still achieve
comparable performance with existing works. Our results can be viewd at
this https URL

我们提出了一种新颖的扇形扩散模型（S2DM），通过一组从相同噪声点开始的射线状反扩散过程形成的扇形扩散区域，S2DM 可以生成一组具有相同语义和随机特征的内在相关数据，但在时间特征上有适当的引导条件所变化。我们将 S2DM 应用于视频生成任务，并探索了光流作为时间条件的使用。实验结果表明，S2DM 在视频生成任务中的表现优于许多现有方法，而不需要任何时间特征建模模块。对于文本到视频生成任务，在没有明确给定时间条件的情况下，我们提出了一种两阶段生成策略，可以将时间特征的生成与语义内容特征的生成分离开来。我们展示了在没有额外训练的情况下，将我们的模型与另一个时间条件生成模型集成仍然可以达到与现有作品相当的性能。

S2DM: 视频生成的扇形扩散模型

S2DM: Sector-Shaped Diffusion Models for Video Generation

Recent works have shown the potential of diffusion models in computer vision
and natural language processing. Apart from the classical supervised learning
fields, diffusion models have also shown strong competitiveness in
reinforcement learning (RL) by formulating decision-making as sequential
generation. However, incorporating temporal information of sequential data and
utilizing it to guide diffusion models to perform better generation is still an
open challenge. In this paper, we take one step forward to investigate
controllable generation with temporal conditions that are refined from temporal
information. We observe the importance of temporal conditions in sequential
generation in sufficient explorative scenarios and provide a comprehensive
discussion and comparison of different temporal conditions. Based on the
observations, we propose an effective temporally-conditional diffusion model
coined Temporally-Composable Diffuser (TCD), which extracts temporal
information from interaction sequences and explicitly guides generation with
temporal conditions. Specifically, we separate the sequences into three parts
according to time expansion and identify historical, immediate, and prospective
conditions accordingly. Each condition preserves non-overlapping temporal
information of sequences, enabling more controllable generation when we jointly
use them to guide the diffuser. Finally, we conduct extensive experiments and
analysis to reveal the favorable applicability of TCD in offline RL tasks,
where our method reaches or matches the best performance compared with prior
SOTA baselines.

本文提出了一种基于时间条件的扩散模型 (Temporally-Composable Diffuser)，该模型可以从交互序列中提取时间信息，并将其用于指导生成，以在离线强化学习任务中实现更好的性能。