We present a method to edit complex indoor lighting from a single image with
its predicted depth and light source segmentation masks. This is an extremely
challenging problem that requires modeling complex light transport, and
disentangling HDR lighting from material and geometry with only a partial LDR
observation of the scene. We tackle this problem using two novel components: 1)
a holistic scene reconstruction method that estimates scene reflectance and
parametric 3D lighting, and 2) a neural rendering framework that re-renders the
scene from our predictions. We use physically-based indoor light
representations that allow for intuitive editing, and infer both visible and
invisible light sources. Our neural rendering framework combines
physically-based direct illumination and shadow rendering with deep networks to
approximate global illumination. It can capture challenging lighting effects,
such as soft shadows, directional lighting, specular materials, and
interreflections. Previous single image inverse rendering methods usually
entangle scene lighting and geometry and only support applications like object
insertion. Instead, by combining parametric 3D lighting estimation with neural
scene rendering, we demonstrate the first automatic method to achieve full
scene relighting, including light source insertion, removal, and replacement,
from a single image. All source code and data will be publicly released.

本文提出了一种使用神经渲染框架的方法，通过利用预测的深度数据和光源分割掩模，从单个图像中编辑复杂室内照明，解决了通过局部 LDR 场景观察从而对 HDR 照明进行建模和分离的难题，其特点在于综合使用场景重建和参数化 3D 照明的方法。