Generating a video given the first several static frames is challenging as it
anticipates reasonable future frames with temporal coherence. Besides video
prediction, the ability to rewind from the last frame or infilling between the
head and tail is also crucial, but they have rarely been explored for video
completion. Since there could be different outcomes from the hints of just a
few frames, a system that can follow natural language to perform video
completion may significantly improve controllability. Inspired by this, we
introduce a novel task, text-guided video completion (TVC), which requests the
model to generate a video from partial frames guided by an instruction. We then
propose Multimodal Masked Video Generation (MMVG) to address this TVC task.
During training, MMVG discretizes the video frames into visual tokens and masks
most of them to perform video completion from any time point. At inference
time, a single MMVG model can address all 3 cases of TVC, including video
prediction, rewind, and infilling, by applying corresponding masking
conditions. We evaluate MMVG in various video scenarios, including egocentric,
animation, and gaming. Extensive experimental results indicate that MMVG is
effective in generating high-quality visual appearances with text guidance for
TVC.

提供了一个新的任务，即文本引导下的视频完成（TVC），并提出了一种新的模型 —— 多模式蒙版视频生成（MMVG），通过文本输入生成高质量的视频完整图像，同时满足预测，倒带和填充的条件。

通过多模态遮挡视频生成实现统一的文本引导视频完成

Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation

We propose the onion-peel networks for video completion. Given a set of
reference images and a target image with holes, our network fills the hole by
referring the contents in the reference images. Our onion-peel network
progressively fills the hole from the hole boundary enabling it to exploit
richer contextual information for the missing regions every step. Given a
sufficient number of recurrences, even a large hole can be inpainted
successfully. To attend to the missing information visible in the reference
images, we propose an asymmetric attention block that computes similarities
between the hole boundary pixels in the target and the non-hole pixels in the
references in a non-local manner. With our attention block, our network can
have an unlimited spatial-temporal window size and fill the holes with globally
coherent contents. In addition, our framework is applicable to the image
completion guided by the reference images without any modification, which is
difficult to do with the previous methods. We validate that our method produces
visually pleasing image and video inpainting results in realistic test cases.

通过提出基于洋葱皮模型的神经网络，本文实现了针对具有遮挡区域的图像和视频的自适应修复，并应用非对称注意力机制来填充遮挡区域。实验结果表明，本方法可以在保持局部特征的同时实现全局一致性，具有较好的视觉效果。

使用洋葱皮网络进行深度视频修复

Onion-Peel Networks for Deep Video Completion

In this paper we propose novel methods for completion (from limited samples)
and de-noising of multilinear (tensor) data and as an application consider 3-D
and 4- D (color) video data completion and de-noising. We exploit the recently
proposed tensor-Singular Value Decomposition (t-SVD)[11]. Based on t-SVD, the
notion of multilinear rank and a related tensor nuclear norm was proposed in
[11] to characterize informational and structural complexity of multilinear
data. We first show that videos with linear camera motion can be represented
more efficiently using t-SVD compared to the approaches based on vectorizing or
flattening of the tensors. Since efficiency in representation implies
efficiency in recovery, we outline a tensor nuclear norm penalized algorithm
for video completion from missing entries. Application of the proposed
algorithm for video recovery from missing entries is shown to yield a superior
performance over existing methods. We also consider the problem of tensor
robust Principal Component Analysis (PCA) for de-noising 3-D video data from
sparse random corruptions. We show superior performance of our method compared
to the matrix robust PCA adapted to this setting as proposed in [4].

本研究提出了一种多线性数据完成和去噪的新方法，应用于 3D 和 4D 视频数据的完成和去噪，利用了最近提出的张量奇异值分解（t-SVD）方法，基于 t-SVD，提出了多线性秩和相关张量核范数的概念来表征多线性数据的信息和结构复杂性。在此基础上，我们提出了一个张量核范数惩罚算法，用于从缺失项中完成视频，并且相对于已有方法，该算法表现出更优秀的性能。我们还考虑了从稀疏随机损坏中去噪 3D 视频数据的张量鲁棒主成分分析（PCA）问题，相对于矩阵鲁棒 PCA 的方法，我们的方法表现更优秀。