Recent advancements in video generation have been remarkable, yet many
existing methods struggle with issues of consistency and poor text-video
alignment. Moreover, the field lacks effective techniques for text-guided video
inpainting, a stark contrast to the well-explored domain of text-guided image
inpainting. To this end, this paper proposes a novel text-guided video
inpainting model that achieves better consistency, controllability and
compatibility. Specifically, we introduce a simple but efficient motion capture
module to preserve motion consistency, and design an instance-aware region
selection instead of a random region selection to obtain better textual
controllability, and utilize a novel strategy to inject some personalized
models into our CoCoCo model and thus obtain better model compatibility.
Extensive experiments show that our model can generate high-quality video
clips. Meanwhile, our model shows better motion consistency, textual
controllability and model compatibility. More details are shown in
[cococozibojia.github.io](cococozibojia.github.io).

本文提出了一种新颖的文本导向的视频修复模型，实现了更好的一致性、可控性和兼容性。实验表明，该模型能够生成高质量的视频片段，并展示了更好的动作连贯性、文本可控性和模型兼容性。

CoCoCo：改进文本引导的视频修复以提升一致性、可控性和兼容性

CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency,  Controllability and Compatibility

The goal of conditional image-to-video (cI2V) generation is to create a
believable new video by beginning with the condition, i.e., one image and
text.The previous cI2V generation methods conventionally perform in RGB pixel
space, with limitations in modeling motion consistency and visual continuity.
Additionally, the efficiency of generating videos in pixel space is quite
low.In this paper, we propose a novel approach to address these challenges by
disentangling the target RGB pixels into two distinct components: spatial
content and temporal motions. Specifically, we predict temporal motions which
include motion vector and residual based on a 3D-UNet diffusion model. By
explicitly modeling temporal motions and warping them to the starting image, we
improve the temporal consistency of generated videos. This results in a
reduction of spatial redundancy, emphasizing temporal details. Our proposed
method achieves performance improvements by disentangling content and motion,
all without introducing new structural complexities to the model. Extensive
experiments on various datasets confirm our approach's superior performance
over the majority of state-of-the-art methods in both effectiveness and
efficiency.

通过将目标 RGB 像素分解成空间内容和时间运动两个不同的组成部分，我们提出了一种解决传统 RGB 像素空间方法中涉及建模动作一致性和视觉连贯性限制的新方法。通过显式建模时间运动并将其应用于起始图像，我们改进了生成视频的时间一致性，减少了空间冗余，突出了时间细节。大量实验证实了我们的方法相对于大多数最先进的方法在效果和效率上的卓越性能。

解耦内容与运动进行条件图像到视频的生成

Decouple Content and Motion for Conditional Image-to-Video Generation

Video anomaly detection (VAD) is a significant computer vision problem.
Existing deep neural network (DNN) based VAD methods mostly follow the route of
frame reconstruction or frame prediction. However, the lack of mining and
learning of higher-level visual features and temporal context relationships in
videos limits the further performance of these two approaches. Inspired by
video codec theory, we introduce a brand-new VAD paradigm to break through
these limitations: First, we propose a new task of video event restoration
based on keyframes. Encouraging DNN to infer missing multiple frames based on
video keyframes so as to restore a video event, which can more effectively
motivate DNN to mine and learn potential higher-level visual features and
comprehensive temporal context relationships in the video. To this end, we
propose a novel U-shaped Swin Transformer Network with Dual Skip Connections
(USTN-DSC) for video event restoration, where a cross-attention and a temporal
upsampling residual skip connection are introduced to further assist in
restoring complex static and dynamic motion object features in the video. In
addition, we propose a simple and effective adjacent frame difference loss to
constrain the motion consistency of the video sequence. Extensive experiments
on benchmarks demonstrate that USTN-DSC outperforms most existing methods,
validating the effectiveness of our method.

该研究通过引入新的视频事件修复任务，提出了一种新的视频异常检测范式，其基于关键帧，使用具有双跳连接的 U 形 Swin Transformer 网络进行视频事件修复，并使用相邻帧差异损失约束视频序列的运动一致性，实验结果表明该方法比大多数现有方法表现更优秀。