Seemingly simple natural language requests to a robot are generally
underspecified, for example "Can you bring me the wireless mouse?" Flat images
of candidate mice may not provide the discriminative information needed for
"wireless." The world, and objects in it, are not flat images but complex 3D
shapes. If a human requests an object based on any of its basic properties,
such as color, shape, or texture, robots should perform the necessary
exploration to accomplish the task. In particular, while substantial effort and
progress has been made on understanding explicitly visual attributes like color
and category, comparatively little progress has been made on understanding
language about shapes and contours. In this work, we introduce a novel
reasoning task that targets both visual and non-visual language about 3D
objects. Our new benchmark, ShapeNet Annotated with Referring Expressions
(SNARE) requires a model to choose which of two objects is being referenced by
a natural language description. We introduce several CLIP-based models for
distinguishing objects and demonstrate that while recent advances in jointly
modeling vision and language are useful for robotic language understanding, it
is still the case that these image-based models are weaker at understanding the
3D nature of objects -- properties which play a key role in manipulation. We
find that adding view estimation to language grounding models improves accuracy
on both SNARE and when identifying objects referred to in language on a robot
platform, but note that a large gap remains between these models and human
performance.

本文介绍了一个新的推理任务，旨在针对三维对象的视觉和非视觉语言，并介绍了用于区分对象的几种 CLIP 模型。虽然最近在联合建模视觉和语言方面取得了进展，但这些基于图像的模型仍然对对象的三维性质了解不足，此文发现，将视图估计添加到语言引理模型可以提高准确性。