Locating objects referred to in natural language poses a significant
challenge for autonomous agents. Existing CLIP-based open-vocabulary methods
successfully perform 3D object retrieval with simple (bare) queries but cannot
cope with ambiguous descriptions that demand an understanding of object
relations. To tackle this problem, we propose a modular approach called BBQ
(Beyond Bare Queries), which constructs 3D scene spatial graph representation
with metric edges and utilizes a large language model as a human-to-agent
interface through our deductive scene reasoning algorithm. BBQ employs robust
DINO-powered associations to form 3D objects, an advanced raycasting algorithm
to project them to 2D, and a vision-language model to describe them as graph
nodes. On Replica and ScanNet datasets, we show that the designed method
accurately constructs 3D object-centric maps. We have demonstrated that their
quality takes a leading place for open-vocabulary 3D semantic segmentation
against other zero-shot methods. Also, we show that leveraging spatial
relations is especially effective for scenes containing multiple entities of
the same semantic class. On Sr3D and Nr3D benchmarks, our deductive approach
demonstrates a significant improvement, enabling retrieving objects by complex
queries compared to other state-of-the-art methods. Considering our design
solutions, we achieved a processing speed approximately x3 times faster than
the closest analog. This promising performance enables our approach for usage
in applied intelligent robotics projects. We make the code publicly available
at linukc.github.io/bbq/.

通过提出模块化方法 BBQ（Beyond Bare Queries），结合构建 3D 场景空间图表示、套用大型语言模型以及深度推理算法，本研究成功实现了生成准确的 3D 物体中心地图、应用于开放词汇 3D 语义分割、并具备在复杂查询中通过使用空间关系检索物体的能力，并且其处理速度比最近的类似方法快大约三倍。