We present a method for novel view synthesis from input images that are
freely distributed around a scene. Our method does not rely on a regular
arrangement of input views, can synthesize images for free camera movement
through the scene, and works for general scenes with unconstrained geometric
layouts. We calibrate the input images via SfM and erect a coarse geometric
scaffold via MVS. This scaffold is used to create a proxy depth map for a novel
view of the scene. Based on this depth map, a recurrent encoder-decoder network
processes reprojected features from nearby views and synthesizes the new view.
Our network does not need to be optimized for a given scene. After training on
a dataset, it works in previously unseen environments with no fine-tuning or
per-scene optimization. We evaluate the presented approach on challenging
real-world datasets, including Tanks and Temples, where we demonstrate
successful view synthesis for the first time and substantially outperform prior
and concurrent work.

本文提出了一种新的方法，可以从自由分布在场景周围的输入图像中合成新视角的图像，不依赖于输入视角的规则排列，可以为场景中的自由相机运动合成图像，并适用于具有任意几何布局的通用场景，该方法通过 SfM 对输入图像进行校准并通过 MVS 建立一个粗略的几何支架，然后基于透视图创建一个代理深度图，进而利用循环编解码网络处理从附近视角重新投影的特征，并合成新视角。该网络不需要针对特定场景进行优化，通过数据集训练后，可以在之前看不到的环境中工作，本文的方法在 Tanks and Temples 等具有挑战性的现实世界数据集上进行了评估和实验证明，表现出色，大大优于之前和同时进行的工作。

自由视角合成

Free View Synthesis

We propose a novel method for real-time face alignment in videos based on a
recurrent encoder-decoder network model. Our proposed model predicts 2D facial
point heat maps regularized by both detection and regression loss, while
uniquely exploiting recurrent learning at both spatial and temporal dimensions.
At the spatial level, we add a feedback loop connection between the combined
output response map and the input, in order to enable iterative coarse-to-fine
face alignment using a single network model, instead of relying on traditional
cascaded model ensembles. At the temporal level, we first decouple the features
in the bottleneck of the network into temporal-variant factors, such as pose
and expression, and temporal-invariant factors, such as identity information.
Temporal recurrent learning is then applied to the decoupled temporal-variant
features. We show that such feature disentangling yields better generalization
and significantly more accurate results at test time. We perform a
comprehensive experimental analysis, showing the importance of each component
of our proposed model, as well as superior results over the state of the art
and several variations of our method in standard datasets.

本论文提出了一种基于循环编码器 - 解码器网络模型的实时视频人脸对齐方法，该方法预测由检测和回归损失正则化的 2D 人脸关键点热力图，同时在空间和时间维度上独特地利用循环学习。在空间上，我们通过输出响应图和输入之间的反馈环连接，使单个网络模型实现迭代粗略到精细的人脸对齐，而不是依赖于传统的级联模型集成。在时间上，我们首先将网络瓶颈中的特征解耦为时变因素和时不变因素，例如姿态和表情和身份信息。然后应用于解耦的时变特征上的时态循环学习。我们展示了这种特征解耦产生更好的泛化效果，在标准数据集上优于现有技术和本方法的几种变体的全面实验分析。

RED-Net: 一种基于循环编码器 - 解码器网络的视频人脸对齐方法

RED-Net: A Recurrent Encoder-Decoder Network for Video-based Face  Alignment

We propose a novel recurrent encoder-decoder network model for real-time
video-based face alignment. Our proposed model predicts 2D facial point maps
regularized by a regression loss, while uniquely exploiting recurrent learning
at both spatial and temporal dimensions. At the spatial level, we add a
feedback loop connection between the combined output response map and the
input, in order to enable iterative coarse-to-fine face alignment using a
single network model. At the temporal level, we first decouple the features in
the bottleneck of the network into temporal-variant factors, such as pose and
expression, and temporal-invariant factors, such as identity information.
Temporal recurrent learning is then applied to the decoupled temporal-variant
features, yielding better generalization and significantly more accurate
results at test time. We perform a comprehensive experimental analysis, showing
the importance of each component of our proposed model, as well as superior
results over the state-of-the-art in standard datasets.

本研究提出了一种新颖的循环编码器 - 解码器网络模型，用于实时基于视频的面部对齐。我们的模型预测带有回归损失的 2D 面部关键点映射，同时在空间和时间维度上独特地利用循环学习。在空间层面上，我们在联合输出响应映射和输入之间添加反馈循环连接，以使用单个网络模型实现迭代式的粗到精面部对齐。在时间层面上，我们首先将网络瓶颈中的特征分解为时间可变因素和时间不变因素，例如姿势和表情以及身份信息。然后将时间循环学习应用于分解的时间可变特征，在测试时间产生更好的普适性和显着更精确的结果。通过全面实验分析，我们展示了提出的模型各组件的重要性，以及在标准数据集上的优异结果。