We present a pipeline for realistic embedding of virtual objects into footage
of indoor scenes with focus on real-time AR applications. Our pipeline consists
of two main components: A light estimator and a neural soft shadow texture
generator. Our light estimation is based on deep neural nets and determines the
main light direction, light color, ambient color and an opacity parameter for
the shadow texture. Our neural soft shadow method encodes object-based
realistic soft shadows as light direction dependent textures in a small MLP. We
show that our pipeline can be used to integrate objects into AR scenes in a new
level of realism in real-time. Our models are small enough to run on current
mobile devices. We achieve runtimes of 9ms for light estimation and 5ms for
neural shadows on an iPhone 11 Pro.

我们提出了一个用于将虚拟对象逼真地嵌入到室内场景中的流程，重点关注实时增强现实应用。我们的流程包括两个主要组件：光估计器和神经软阴影纹理生成器。我们的光估计基于深度神经网络，确定主光方向、光颜色、环境颜色和阴影纹理的不透明度参数。我们的神经软阴影方法将基于物体的逼真软阴影编码为与光方向相关的纹理，存储在一个小型的多层感知机中。我们展示了我们的流程可以在实时中将对象融入增强现实场景中，实现新的逼真程度。我们的模型足够小，可以在现有的移动设备上运行，我们在 iPhone 11 Pro 上实现了 9ms 的光估计和 5ms 的神经阴影运行时间。