We integrate two powerful ideas, geometry and deep visual representation
learning, into recurrent network architectures for mobile visual scene
understanding. The proposed networks learn to "lift" and integrate 2D visual
features over time into latent 3D feature maps of the scene. They are equipped
with differentiable geometric operations, such as projection, unprojection,
egomotion estimation and stabilization, in order to compute a
geometrically-consistent mapping between the world scene and their 3D latent
feature state. We train the proposed architectures to predict novel camera
views given short frame sequences as input. Their predictions strongly
generalize to scenes with a novel number of objects, appearances and
configurations; they greatly outperform previous works that do not consider
egomotion stabilization or a space-aware latent feature state. We train the
proposed architectures to detect and segment objects in 3D using the latent 3D
feature map as input--as opposed to per frame features. The resulting object
detections persist over time: they continue to exist even when an object gets
occluded or leaves the field of view. Our experiments suggest the proposed
space-aware latent feature memory and egomotion-stabilized convolutions are
essential architectural choices for spatial common sense to emerge in
artificial embodied visual agents.

这篇论文提出了一种通过结合几何学和深度视觉表示学习的思想，将其嵌入移动视觉场景理解的递归网络架构中，以学习如何将 2D 视觉特征整合到场景的潜在 3D 特征映射中，通过不同 iable 几何操作进行预测和分割，十分成功。