Recent advances in vision-language models (VLMs) have led to improved
performance on tasks such as visual question answering and image captioning.
Consequently, these models are now well-positioned to reason about the physical
world, particularly within domains such as robotic manipulation. However,
current VLMs are limited in their understanding of the physical concepts (e.g.,
material, fragility) of common objects, which restricts their usefulness for
robotic manipulation tasks that involve interaction and physical reasoning
about such objects. To address this limitation, we propose PhysObjects, an
object-centric dataset of 36.9K crowd-sourced and 417K automated physical
concept annotations of common household objects. We demonstrate that
fine-tuning a VLM on PhysObjects improves its understanding of physical object
concepts, by capturing human priors of these concepts from visual appearance.
We incorporate this physically-grounded VLM in an interactive framework with a
large language model-based robotic planner, and show improved planning
performance on tasks that require reasoning about physical object concepts,
compared to baselines that do not leverage physically-grounded VLMs. We
additionally illustrate the benefits of our physically-grounded VLM on a real
robot, where it improves task success rates. We release our dataset and provide
further details and visualizations of our results at
this https URL

通过在 36.9K 个常见家居物体的数据集 PhysObjects 上针对视觉外观捕捉人类先验知识，我们提出了一种基于物理概念的视觉语言模型（VLM），并将其与基于大语言模型的机器人规划器结合使用，取得了在涉及与物理物体概念推理相关的任务中相比不使用物理概念的基线模型实现了改进的规划性能的结果。此外，在真实机器人上展示了物理概念视觉语言模型的好处，提高了任务成功率。

物理基础的视觉语言模型用于机器人操作

Physically Grounded Vision-Language Models for Robotic Manipulation

Language models~(LMs) gradually become general-purpose interfaces in the
interactive and embodied world, where the understanding of physical concepts is
an essential prerequisite. However, it is not yet clear whether LMs can
understand physical concepts in the human world. To investigate this, we design
a benchmark VEC that covers the tasks of (i) Visual concepts, such as the shape
and material of objects, and (ii) Embodied Concepts, learned from the
interaction with the world such as the temperature of objects. Our zero
(few)-shot prompting results show that the understanding of certain visual
concepts emerges as scaling up LMs, but there are still basic concepts to which
the scaling law does not apply. For example, OPT-175B performs close to humans
with a zero-shot accuracy of 85\% on the material concept, yet behaves like
random guessing on the mass concept. Instead, vision-augmented LMs such as CLIP
and BLIP achieve a human-level understanding of embodied concepts. Analysis
indicates that the rich semantics in visual representation can serve as a
valuable source of embodied knowledge. Inspired by this, we propose a
distillation method to transfer embodied knowledge from VLMs to LMs, achieving
performance gain comparable with that by scaling up the parameters of LMs 134x.
Our dataset is available at https://github.com/TobiasLee/VEC

该论文研究了自然语言模型是否能理解物理概念，并设计了一个基准测试 VEC 来探究视觉和具身概念。结果表明，视觉增强的语言模型，如 CLIP 和 BLIP，能够理解具身概念，而通过蒸馏方法将具身知识转移给 LMs 的性能也相当可观。