We present the first real-time method for inserting a rigid virtual object
into a neural radiance field, which produces realistic lighting and shadowing
effects, as well as allows interactive manipulation of the object. By
exploiting the rich information about lighting and geometry in a NeRF, our
method overcomes several challenges of object insertion in augmented reality.
For lighting estimation, we produce accurate, robust and 3D spatially-varying
incident lighting that combines the near-field lighting from NeRF and an
environment lighting to account for sources not covered by the NeRF. For
occlusion, we blend the rendered virtual object with the background scene using
an opacity map integrated from the NeRF. For shadows, with a precomputed field
of spherical signed distance field, we query the visibility term for any point
around the virtual object, and cast soft, detailed shadows onto 3D surfaces.
Compared with state-of-the-art techniques, our approach can insert virtual
object into scenes with superior fidelity, and has a great potential to be
further applied to augmented reality systems.

我们提出了第一个实时方法，用于将刚性虚拟对象插入到神经辐射场中，它能够产生逼真的光照和阴影效果，并且允许对对象进行交互操作。通过利用 NeRF 中有关光照和几何的丰富信息，我们的方法克服了增强现实中对象插入的几个挑战。对于光照估计，我们产生了精确、稳健和具有三维空间变化的入射光，结合了 NeRF 的近场光照和环境光，以考虑 NeRF 未覆盖的光源。对于遮挡，我们使用从 NeRF 中整合的不透明度图将渲染的虚拟对象与背景场景混合。对于阴影，我们使用一个预计算的球形符号距离场查询视点周围的任意点的可见性，并在三维表面上投射软、详细的阴影。与现有技术相比，我们的方法能够以更高的保真度将虚拟对象插入场景中，并有很大潜力进一步应用于增强现实系统。