Generating natural human grasps necessitates consideration of not just object
geometry but also semantic information. Solely depending on object shape for
grasp generation confines the applications of prior methods in downstream
tasks. This paper presents a novel semantic-based grasp generation method,
termed SemGrasp, which generates a static human grasp pose by incorporating
semantic information into the grasp representation. We introduce a discrete
representation that aligns the grasp space with semantic space, enabling the
generation of grasp postures in accordance with language instructions. A
Multimodal Large Language Model (MLLM) is subsequently fine-tuned, integrating
object, grasp, and language within a unified semantic space. To facilitate the
training of SemGrasp, we have compiled a large-scale, grasp-text-aligned
dataset named CapGrasp, featuring about 260k detailed captions and 50k diverse
grasps. Experimental findings demonstrate that SemGrasp efficiently generates
natural human grasps in alignment with linguistic intentions. Our code, models,
and dataset are available publicly at: this https URL

本文提出了一种基于语义的抓取生成方法 SemGrasp，并且介绍了一个将目标、抓取和语言整合到一个统一的语义空间中的多模态大型语言模型 (Multimodal Large Language Model, MLLM)。实验结果表明，SemGrasp 能够高效地生成符合语言意图的自然人类抓取。

SemGrasp: 通过语言对齐离散化生成语义抓取

SemGrasp: Semantic Grasp Generation via Language Aligned Discretization

Grasping unseen objects in unconstrained, cluttered environments is an
essential skill for autonomous robotic manipulation. Despite recent progress in
full 6-DoF grasp learning, existing approaches often consist of complex
sequential pipelines that possess several potential failure points and
run-times unsuitable for closed-loop grasping. Therefore, we propose an
end-to-end network that efficiently generates a distribution of 6-DoF
parallel-jaw grasps directly from a depth recording of a scene. Our novel grasp
representation treats 3D points of the recorded point cloud as potential grasp
contacts. By rooting the full 6-DoF grasp pose and width in the observed point
cloud, we can reduce the dimensionality of our grasp representation to 4-DoF
which greatly facilitates the learning process. Our class-agnostic approach is
trained on 17 million simulated grasps and generalizes well to real world
sensor data. In a robotic grasping study of unseen objects in structured
clutter we achieve over 90% success rate, cutting the failure rate in half
compared to a recent state-of-the-art method.

通过一种端到端的神经网络，将深度记录的场景直接转化为 6 个自由度平行夹具抓取的分布，其独特的抓取表示法将点云的 3D 点作为潜在的抓取接触点，并将 6 个自由度的抓取姿态与宽度根据观察到的点云减少到 4 个自由度，成功率可达 90% 以上。