Learning descriptive 3D features is crucial for understanding 3D scenes with
diverse objects and complex structures. However, it is usually unknown whether
important geometric attributes and scene context obtain enough emphasis in an
end-to-end trained 3D scene understanding network. To guide 3D feature learning
toward important geometric attributes and scene context, we explore the help of
textual scene descriptions. Given some free-form descriptions paired with 3D
scenes, we extract the knowledge regarding the object relationships and object
attributes. We then inject the knowledge to 3D feature learning through three
classification-based auxiliary tasks. This language-assisted training can be
combined with modern object detection and instance segmentation methods to
promote 3D semantic scene understanding, especially in a label-deficient
regime. Moreover, the 3D feature learned with language assistance is better
aligned with the language features, which can benefit various 3D-language
multimodal tasks. Experiments on several benchmarks of 3D-only and 3D-language
tasks demonstrate the effectiveness of our language-assisted 3D feature
learning. Code is available at
this https URL

通过文本场景描述信息辅助 3D 特征学习，进而提升三维语义场景理解的效果，并构建更好的语言与三维结构的多模态任务。