In this paper, we address the problem of simultaneous relighting and novel
view synthesis of a complex scene from multi-view images with a limited number
of light sources. We propose an analysis-synthesis approach called Relit-NeuLF.
Following the recent neural 4D light field network (NeuLF), Relit-NeuLF first
leverages a two-plane light field representation to parameterize each ray in a
4D coordinate system, enabling efficient learning and inference. Then, we
recover the spatially-varying bidirectional reflectance distribution function
(SVBRDF) of a 3D scene in a self-supervised manner. A DecomposeNet learns to
map each ray to its SVBRDF components: albedo, normal, and roughness. Based on
the decomposed BRDF components and conditioning light directions, a RenderNet
learns to synthesize the color of the ray. To self-supervise the SVBRDF
decomposition, we encourage the predicted ray color to be close to the
physically-based rendering result using the microfacet model. Comprehensive
experiments demonstrate that the proposed method is efficient and effective on
both synthetic data and real-world human face data, and outperforms the
state-of-the-art results. We publicly released our code on GitHub. You can find
it here: this https URL

本文提出了一种分析 - 合成方法 Relit-NeuLF，通过使用两平面光场表示对 4D 坐标系统的每条光线进行参数化，以实现对复杂场景的同时重照和新视角合成；通过自监督学习方法，该方法能够恢复三维场景的空间变化的双向反射分布函数（SVBRDF）；通过将每条光线映射到其 SVBRDF 组成部分（漫反射、法线和粗糙度）以及灯光方向的条件，实现光线颜色的合成；综合实验证明，该方法在合成数据和真实世界人脸数据上都具有高效和有效的性能，并且优于最先进的结果。