We introduce a high resolution spatially adaptive light source, or a
projector, into a neural reflectance field that allows to both calibrate the
projector and photo realistic light editing. The projected texture is fully
differentiable with respect to all scene parameters, and can be optimized to
yield a desired appearance suitable for applications in augmented reality and
projection mapping. Our neural field consists of three neural networks,
estimating geometry, material, and transmittance. Using an analytical BRDF
model and carefully selected projection patterns, our acquisition process is
simple and intuitive, featuring a fixed uncalibrated projected and a handheld
camera with a co-located light source. As we demonstrate, the virtual projector
incorporated into the pipeline improves scene understanding and enables various
projection mapping applications, alleviating the need for time consuming
calibration steps performed in a traditional setting per view or projector
location. In addition to enabling novel viewpoint synthesis, we demonstrate
state-of-the-art performance projector compensation for novel viewpoints,
improvement over the baselines in material and scene reconstruction, and three
simply implemented scenarios where projection image optimization is performed,
including the use of a 2D generative model to consistently dictate scene
appearance from multiple viewpoints. We believe that neural projection mapping
opens up the door to novel and exciting downstream tasks, through the joint
optimization of the scene and projection images.

本文提出利用神经投影映射技术在增强现实应用中实现对投影仪的高分辨率自适应校准和照片级别的光线编辑。通过将虚拟投影纹理与场景参数全微分，可以优化得到所需的外观。此外，本文展示了在材料和场景重建等方面，较传统方法更好的性能。