Spatial memory, or the ability to remember and recall specific locations and
objects, is central to autonomous agents' ability to carry out tasks in real
environments. However, most existing artificial memory modules are not very
adept at storing spatial information. We propose a parameter-free module,
Egospheric Spatial Memory (ESM), which encodes the memory in an ego-sphere
around the agent, enabling expressive 3D representations. ESM can be trained
end-to-end via either imitation or reinforcement learning, and improves both
training efficiency and final performance against other memory baselines on
both drone and manipulator visuomotor control tasks. The explicit egocentric
geometry also enables us to seamlessly combine the learned controller with
other non-learned modalities, such as local obstacle avoidance. We further show
applications to semantic segmentation on the ScanNet dataset, where ESM
naturally combines image-level and map-level inference modalities. Through our
broad set of experiments, we show that ESM provides a general computation graph
for embodied spatial reasoning, and the module forms a bridge between real-time
mapping systems and differentiable memory architectures. Implementation at:
this https URL

提出使用无参数模块 Egospheric Spatial Memory (ESM) 来通过在自主代理周围的自我球内部编码记忆以实现表达三维表示来改善空间记忆的性能，并可以通过模仿或强化学习进行端对端训练，在训练效率和最后的控制方法表现方面明显优于其他记忆基线，同时可以无缝地将该控制器与其他未学习的模态相结合，从而实现对语义分割的应用。