Video-to-video synthesis (vid2vid) aims for converting high-level semantic
inputs to photorealistic videos. While existing vid2vid methods can achieve
short-term temporal consistency, they fail to ensure the long-term one. This is
because they lack knowledge of the 3D world being rendered and generate each
frame only based on the past few frames. To address the limitation, we
introduce a novel vid2vid framework that efficiently and effectively utilizes
all past generated frames during rendering. This is achieved by condensing the
3D world rendered so far into a physically-grounded estimate of the current
frame, which we call the guidance image. We further propose a novel neural
network architecture to take advantage of the information stored in the
guidance images. Extensive experimental results on several challenging datasets
verify the effectiveness of our approach in achieving world consistency - the
output video is consistent within the entire rendered 3D world.
this https URL

提出了一种新颖的 vid2vid 框架，它使用所渲染的所有过去帧来生成当前帧，并通过引入物理基础的 guidance image 来提高 3D 世界的一致性，并使用新颖的神经网络架构利用存储在 guidance image 中的信息。

基于世界一致性的视频到视频合成

World-Consistent Video-to-Video Synthesis

Deep generative models have shown success in automatically synthesizing
missing image regions using surrounding context. However, users cannot directly
decide what content to synthesize with such approaches. We propose an
end-to-end network for image inpainting that uses a different image to guide
the synthesis of new content to fill the hole. A key challenge addressed by our
approach is synthesizing new content in regions where the guidance image and
the context of the original image are inconsistent. We conduct four studies
that demonstrate our results yield more realistic image inpainting results over
seven baselines.

本文提出了一种基于端到端网络的图像修复方法，使用不同的图像来引导合成新的内容并填充空洞，并在四项研究中展示了优于七种基线方法的更逼真的图像修复结果。

引导式图像修补：通过另一幅图像提取内容替换图像区域

Guided Image Inpainting: Replacing an Image Region by Pulling Content  from Another Image

Joint image filters leverage the guidance image as a prior and transfer the
structural details from the guidance image to the target image for suppressing
noise or enhancing spatial resolution. Existing methods either rely on various
explicit filter constructions or hand-designed objective functions, thereby
making it difficult to understand, improve, and accelerate these filters in a
coherent framework. In this paper, we propose a learning-based approach for
constructing joint filters based on Convolutional Neural Networks. In contrast
to existing methods that consider only the guidance image, the proposed
algorithm can selectively transfer salient structures that are consistent with
both guidance and target images. We show that the model trained on a certain
type of data, e.g., RGB and depth images, generalizes well to other modalities,
e.g., flash/non-Flash and RGB/NIR images. We validate the effectiveness of the
proposed joint filter through extensive experimental evaluations with
state-of-the-art methods.

本研究提出了一种学习性方法，基于卷积神经网络构建联合滤波器，该算法可以选择性地传输与指导和目标图像一致的重要结构，验证表明该联合滤波器在抑制噪声或增强空间分辨率方面具有有效性，并且具有与现有最先进方法相当的性能。