Robotic grasping is a fundamental aspect of robot functionality, defining how
robots interact with objects. Despite substantial progress, its
generalizability to counter-intuitive or long-tailed scenarios, such as objects
with uncommon materials or shapes, remains a challenge. In contrast, humans can
easily apply their intuitive physics to grasp skillfully and change grasps
efficiently, even for objects they have never seen before.
This work delves into infusing such physical commonsense reasoning into
robotic manipulation. We introduce PhyGrasp, a multimodal large model that
leverages inputs from two modalities: natural language and 3D point clouds,
seamlessly integrated through a bridge module. The language modality exhibits
robust reasoning capabilities concerning the impacts of diverse physical
properties on grasping, while the 3D modality comprehends object shapes and
parts. With these two capabilities, PhyGrasp is able to accurately assess the
physical properties of object parts and determine optimal grasping poses.
Additionally, the model's language comprehension enables human instruction
interpretation, generating grasping poses that align with human preferences. To
train PhyGrasp, we construct a dataset PhyPartNet with 195K object instances
with varying physical properties and human preferences, alongside their
corresponding language descriptions. Extensive experiments conducted in the
simulation and on the real robots demonstrate that PhyGrasp achieves
state-of-the-art performance, particularly in long-tailed cases, e.g., about
10% improvement in success rate over GraspNet. Project page:
this https URL

将物理常识推理融入机器人操作，通过 PhyGrasp 模型结合自然语言和 3D 点云输入，实现对对象的物理属性准确评估和最优抓取位姿决策。模型的语言理解能力使其能够解释人类指令，并生成符合人类偏好的抓取位姿。在长尾场景中，PhyGrasp 取得了最先进的性能，成功率比 GraspNet 提高了约 10%。