Video outpainting is a challenging task, aiming at generating video content
outside the viewport of the input video while maintaining inter-frame and
intra-frame consistency. Existing methods fall short in either generation
quality or flexibility. We introduce MOTIA Mastering Video Outpainting Through
Input-Specific Adaptation, a diffusion-based pipeline that leverages both the
intrinsic data-specific patterns of the source video and the image/video
generative prior for effective outpainting. MOTIA comprises two main phases:
input-specific adaptation and pattern-aware outpainting. The input-specific
adaptation phase involves conducting efficient and effective pseudo outpainting
learning on the single-shot source video. This process encourages the model to
identify and learn patterns within the source video, as well as bridging the
gap between standard generative processes and outpainting. The subsequent
phase, pattern-aware outpainting, is dedicated to the generalization of these
learned patterns to generate outpainting outcomes. Additional strategies
including spatial-aware insertion and noise travel are proposed to better
leverage the diffusion model's generative prior and the acquired video patterns
from source videos. Extensive evaluations underscore MOTIA's superiority,
outperforming existing state-of-the-art methods in widely recognized
benchmarks. Notably, these advancements are achieved without necessitating
extensive, task-specific tuning.

通过输入特定的适应和模式感知的视频超出绘制，MOTIA 提供了一种基于扩散的流程，以有效地实现视觉外扩，进而在源视频中学习模式并生成外扩结果。

Be-Your-Outpainter: 通过特定输入的适应实现视频外推

Be-Your-Outpainter: Mastering Video Outpainting through Input-Specific  Adaptation

Video outpainting aims to adequately complete missing areas at the edges of
video frames. Compared to image outpainting, it presents an additional
challenge as the model should maintain the temporal consistency of the filled
area. In this paper, we introduce a masked 3D diffusion model for video
outpainting. We use the technique of mask modeling to train the 3D diffusion
model. This allows us to use multiple guide frames to connect the results of
multiple video clip inferences, thus ensuring temporal consistency and reducing
jitter between adjacent frames. Meanwhile, we extract the global frames of the
video as prompts and guide the model to obtain information other than the
current video clip using cross-attention. We also introduce a hybrid
coarse-to-fine inference pipeline to alleviate the artifact accumulation
problem. The existing coarse-to-fine pipeline only uses the infilling strategy,
which brings degradation because the time interval of the sparse frames is too
large. Our pipeline benefits from bidirectional learning of the mask modeling
and thus can employ a hybrid strategy of infilling and interpolation when
generating sparse frames. Experiments show that our method achieves
state-of-the-art results in video outpainting tasks. More results are provided
at our this https URL

视频外延的研究中，我们引入了一种基于遮罩的三维扩散模型，在保持填充区域的时间连续性方面，通过使用遮罩建模技术训练模型，利用多个引导帧连接多个视频剪辑推理结果，确保了时间连续性，并减少相邻帧之间的抖动。另外，我们通过交叉注意力，以全局帧为提示，引导模型获取当前视频剪辑之外的信息。同时，我们还引入了混合的由粗到细的推理流程来缓解伪影积累问题，与现有的只使用填充策略的由粗到细流程不同，我们的流程可以通过遮罩建模的双向学习，在生成稀疏帧时采用填充与插值的混合策略。实验证明我们的方法在视频外延任务中实现了最先进的结果。更多结果请参阅我们的此 URL。