GAN inversion is indispensable for applying the powerful editability of GAN
to real images. However, existing methods invert video frames individually
often leading to undesired inconsistent results over time. In this paper, we
propose a unified recurrent framework, named \textbf{R}ecurrent v\textbf{I}deo
\textbf{G}AN \textbf{I}nversion and e\textbf{D}iting (RIGID), to explicitly and
simultaneously enforce temporally coherent GAN inversion and facial editing of
real videos. Our approach models the temporal relations between current and
previous frames from three aspects. To enable a faithful real video
reconstruction, we first maximize the inversion fidelity and consistency by
learning a temporal compensated latent code. Second, we observe incoherent
noises lie in the high-frequency domain that can be disentangled from the
latent space. Third, to remove the inconsistency after attribute manipulation,
we propose an \textit{in-between frame composition constraint} such that the
arbitrary frame must be a direct composite of its neighboring frames. Our
unified framework learns the inherent coherence between input frames in an
end-to-end manner, and therefore it is agnostic to a specific attribute and can
be applied to arbitrary editing of the same video without re-training.
Extensive experiments demonstrate that RIGID outperforms state-of-the-art
methods qualitatively and quantitatively in both inversion and editing tasks.
The deliverables can be found in https://cnnlstm.github.io/RIGID

本文提出了一种名为 RIGID（Recurrent vIGAN Inversion and eDiting）的统一循环框架，用于显式并同时对实际视频进行时间一致的 GAN 反演和面部编辑。通过从三个方面建模当前帧与前一帧之间的时间关系，我们的方法可以实现忠实的实际视频重构。该统一框架以端到端的方式学习输入帧之间的内在一致性，因此对于特定属性是不可知的，并且可以用于同一视频的任意编辑而无需重新训练。经过广泛实验证明，RIGID 在反演和编辑任务中在定性和定量方面均优于现有的最先进方法。

RIGID: 实时人脸视频的循环生成对抗网络还原和编辑

RIGID: Recurrent GAN Inversion and Editing of Real Face Videos

Generating temporally coherent high fidelity video is an important milestone
in generative modeling research. We make progress towards this milestone by
proposing a diffusion model for video generation that shows very promising
initial results. Our model is a natural extension of the standard image
diffusion architecture, and it enables jointly training from image and video
data, which we find to reduce the variance of minibatch gradients and speed up
optimization. To generate long and higher resolution videos we introduce a new
conditional sampling technique for spatial and temporal video extension that
performs better than previously proposed methods. We present the first results
on a large text-conditioned video generation task, as well as state-of-the-art
results on established benchmarks for video prediction and unconditional video
generation. Supplementary material is available at
this https URL

本研究通过提出扩展图像扩散结构的扩散模型，使得可以自然地对图像和视频数据进行联合训练以生成高保真度的时空连续的视频，并引入一种更好的有条件采样技术，得到了在文本条件视频生成任务及视频预测和无条件视频生成方面的最先进结果。

视频扩散模型

Video Diffusion Models

This paper proposes a novel deep learning-based video object matting method
that can achieve temporally coherent matting results. Its key component is an
attention-based temporal aggregation module that maximizes image matting
networks' strength for video matting networks. This module computes temporal
correlations for pixels adjacent to each other along the time axis in feature
space, which is robust against motion noises. We also design a novel loss term
to train the attention weights, which drastically boosts the video matting
performance. Besides, we show how to effectively solve the trimap generation
problem by fine-tuning a state-of-the-art video object segmentation network
with a sparse set of user-annotated keyframes. To facilitate video matting and
trimap generation networks' training, we construct a large-scale video matting
dataset with 80 training and 28 validation foreground video clips with
ground-truth alpha mattes. Experimental results show that our method can
generate high-quality alpha mattes for various videos featuring appearance
change, occlusion, and fast motion. Our code and dataset can be found at:
this https URL

该论文提出了一种基于深度学习的新型视频对象抠图方法，能够实现时间上连贯的抠图效果，并使用注意力机制的时间聚合模块在特征空间中计算相邻像素沿时间轴的时序相关性，同时利用带有稀疏用户注释关键帧的先进视频对象分割网络进行细调以解决抠图问题。实验结果表明，该方法能够生成适用于各种视频的高质量阿尔法抠图。