We address the problem of novel view synthesis (NVS) from a few sparse source
view images. Conventional image-based rendering methods estimate scene geometry
and synthesize novel views in two separate steps. However, erroneous geometry
estimation will decrease NVS performance as view synthesis highly depends on
the quality of estimated scene geometry. In this paper, we propose an
end-to-end NVS framework to eliminate the error propagation issue. To be
specific, we construct a volume under the target view and design a source-view
visibility estimation (SVE) module to determine the visibility of the
target-view voxels in each source view. Next, we aggregate the visibility of
all source views to achieve a consensus volume. Each voxel in the consensus
volume indicates a surface existence probability. Then, we present a soft
ray-casting (SRC) mechanism to find the most front surface in the target view
(i.e. depth). Specifically, our SRC traverses the consensus volume along
viewing rays and then estimates a depth probability distribution. We then warp
and aggregate source view pixels to synthesize a novel view based on the
estimated source-view visibility and target-view depth. At last, our network is
trained in an end-to-end self-supervised fashion, thus significantly
alleviating error accumulation in view synthesis. Experimental results
demonstrate that our method generates novel views in higher quality compared to
the state-of-the-art.

本文提出了一种端到端的新视角合成框架，包括构建目标视角下的体积以及设计源视角的可见性估计模块和软射线投影机制，最终在端到端的自监督训练方式下，相较于现有方法提高了新视角的合成质量。