Semantic 3D scene understanding is a problem of critical importance in robotics. While significant advances have been made in spatial perception, robots are still far from having the common-sense knowledge about household objects and locations of an average human. We thus investigate the use of large language models to impart common sense for scene understanding. Specifically, we introduce three paradigms for leveraging language for classifying rooms in indoor environments based on their contained objects: (i) a zero-shot approach, (ii) a feed-forward classifier approach, and (iii) a contrastive classifier approach. These methods operate on 3D scene graphs produced by modern spatial perception systems. We then analyze each approach, demonstrating notable zero-shot generalization and transfer capabilities stemming from their use of language. Finally, we show these approaches also apply to inferring building labels from contained rooms and demonstrate our zero-shot approach on a real environment. All code can be found at https://github.com/MIT-SPARK/llm_scene_understanding.

探讨使用大量语言模型来实现场景理解的常识；介绍了三种利用语言对包含对象的室内环境进行分类的范例：（i）零样本方法，（ii）前馈分类器方法和（iii）对比分类器方法，在现代空间感知系统生成的3D场景图上进行操作，并通过分析每种途径，展示了显着的零样本泛化和转移能力；最后，展示了这些方法也适用于推断包含房间的建筑标签，并在真实环境中展示了零样本方法。

利用大型语言模型进行机器人三维场景理解