This paper presents a new method to synthesize an image from arbitrary views
and times given a collection of images of a dynamic scene. A key challenge for
the novel view synthesis arises from dynamic scene reconstruction where
epipolar geometry does not apply to the local motion of dynamic contents. To
address this challenge, we propose to combine the depth from single view (DSV)
and the depth from multi-view stereo (DMV), where DSV is complete, i.e., a
depth is assigned to every pixel, yet view-variant in its scale, while DMV is
view-invariant yet incomplete. Our insight is that although its scale and
quality are inconsistent with other views, the depth estimation from a single
view can be used to reason about the globally coherent geometry of dynamic
contents. We cast this problem as learning to correct the scale of DSV, and to
refine each depth with locally consistent motions between views to form a
coherent depth estimation. We integrate these tasks into a depth fusion network
in a self-supervised fashion. Given the fused depth maps, we synthesize a
photorealistic virtual view in a specific location and time with our deep
blending network that completes the scene and renders the virtual view. We
evaluate our method of depth estimation and view synthesis on diverse
real-world dynamic scenes and show the outstanding performance over existing
methods.

本文提出了一种新的方法，可以从动态场景的一组图像中合成任意视角和时间的图像，通过单视图深度（DSV）和多视图立体深度（DMV）的结合，重新估计视角和时间下的一致性和深度信息，从而精确的合成真实感极强的虚拟视图。

单目相机全局一致深度动态场景新视角合成

Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths  from a Monocular Camera

Depth from a monocular video can enable billions of devices and robots with a
single camera to see the world in 3D. In this paper, we present an approach
with a differentiable flow-to-depth layer for video depth estimation. The model
consists of a flow-to-depth layer, a camera pose refinement module, and a depth
fusion network. Given optical flow and camera pose, our flow-to-depth layer
generates depth proposals and the corresponding confidence maps by explicitly
solving an epipolar geometry optimization problem. Our flow-to-depth layer is
differentiable, and thus we can refine camera poses by maximizing the
aggregated confidence in the camera pose refinement module. Our depth fusion
network can utilize depth proposals and their confidence maps inferred from
different adjacent frames to produce the final depth map. Furthermore, the
depth fusion network can additionally take the depth proposals generated by
other methods to improve the results further. The experiments on three public
datasets show that our approach outperforms state-of-the-art depth estimation
methods, and has reasonable cross dataset generalization capability: our model
trained on KITTI still performs well on the unseen Waymo dataset.

该研究提出了一种基于可微的光流深度预测层的方法，包括光流深度预测层、相机姿态优化模块和深度融合网络，该方法在三个公共数据集上的实验结果表明其性能优于现有的深度估计方法，并具有合理的交叉数据集泛化能力。