Open-vocabulary 3D scene understanding (OV-3D) aims to localize and classify novel objects beyond the closed object classes. However, existing approaches and benchmarks primarily focus on the open vocabulary problem within the context of object classes, which is insufficient to provide a holistic evaluation to what extent a model understands the 3D scene. In this paper, we introduce a more challenging task called Generalized Open-Vocabulary 3D Scene Understanding (GOV-3D) to explore the open vocabulary problem beyond object classes. It encompasses an open and diverse set of generalized knowledge, expressed as linguistic queries of fine-grained and object-specific attributes. To this end, we contribute a new benchmark named OpenScan, which consists of 3D object attributes across eight representative linguistic aspects, including affordance, property, material, and more. We further evaluate state-of-the-art OV-3D methods on our OpenScan benchmark, and discover that these methods struggle to comprehend the abstract vocabularies of the GOV-3D task, a challenge that cannot be addressed by simply scaling up object classes during training. We highlight the limitations of existing methodologies and explore a promising direction to overcome the identified shortcomings. Data and code are available at https://github.com/YoujunZhao/OpenScan

本研究解决了现有3D场景理解方法未能充分评估开放词汇问题的缺陷，特别是在对象类以外的领域。提出了一个新颖的任务，即广义开放词汇3D场景理解(GOV-3D)，并贡献了基准OpenScan，涵盖多个细粒度对象属性。研究发现，现有方法在应对GOV-3D任务中的抽象词汇方面存在显著困难，强调了现有方法的局限性并探讨了克服这些短板的方向。

OpenScan：通用开放词汇3D场景理解基准