We study open-world 3D scene understanding, a family of tasks that require
agents to reason about their 3D environment with an open-set vocabulary and
out-of-domain visual inputs - a critical skill for robots to operate in the
unstructured 3D world. Towards this end, we propose Semantic Abstraction
(SemAbs), a framework that equips 2D Vision-Language Models (VLMs) with new 3D
spatial capabilities, while maintaining their zero-shot robustness. We achieve
this abstraction using relevancy maps extracted from CLIP, and learn 3D spatial
and geometric reasoning skills on top of those abstractions in a
semantic-agnostic manner. We demonstrate the usefulness of SemAbs on two
open-world 3D scene understanding tasks: 1) completing partially observed
objects and 2) localizing hidden objects from language descriptions.
Experiments show that SemAbs can generalize to novel vocabulary,
materials/lighting, classes, and domains (i.e., real-world scans) from training
on limited 3D synthetic data. Code and data is available at
this https URL

本文介绍了一种基于视觉语言模型的语义抽象框架，可以为机器人在未结构化的三维环境中推理，并学会三维空间和几何推理技能，并在两个开放式三维场景理解任务上进行了实验测试。

语义抽象：从 2D 视觉语言模型实现开放式三维场景理解

Semantic Abstraction: Open-World 3D Scene Understanding from 2D  Vision-Language Models

Capturing the meaning of sentences has long been a challenging task. Current
models tend to apply linear combinations of word features to conduct semantic
composition for bigger-granularity units e.g. phrases, sentences, and
documents. However, the semantic linearity does not always hold in human
language. For instance, the meaning of the phrase `ivory tower' can not be
deduced by linearly combining the meanings of `ivory' and `tower'. To address
this issue, we propose a new framework that models different levels of semantic
units (e.g. sememe, word, sentence, and semantic abstraction) on a single
\textit{Semantic Hilbert Space}, which naturally admits a non-linear semantic
composition by means of a complex-valued vector word representation. An
end-to-end neural network~\footnote{this https URL} is
proposed to implement the framework in the text classification task, and
evaluation results on six benchmarking text classification datasets demonstrate
the effectiveness, robustness and self-explanation power of the proposed model.
Furthermore, intuitive case studies are conducted to help end users to
understand how the framework works.

通过在单个语义希尔伯特空间中对语义单元进行建模，本文提出了一种新框架，用于文本分类任务，并在六个基准文本分类数据集上展示了该模型的有效性、鲁棒性和自我解释能力。