This paper presents a novel approach to enhance autonomous robotic
manipulation using the Large Language Model (LLM) for logical inference,
converting high-level language commands into sequences of executable motion
functions. The proposed system combines the advantage of LLM with YOLO-based
environmental perception to enable robots to autonomously make reasonable
decisions and task planning based on the given commands. Additionally, to
address the potential inaccuracies or illogical actions arising from LLM, a
combination of teleoperation and Dynamic Movement Primitives (DMP) is employed
for action correction. This integration aims to improve the practicality and
generalizability of the LLM-based human-robot collaboration system.

该论文介绍了一种利用大型语言模型（LLM）进行自主机器人操纵的新方法，通过逻辑推理将高层语言命令转化为可执行的运动函数序列。所提出的系统将 LLM 的优势与基于 YOLO 的环境感知相结合，使机器人能够根据给定的命令自主做出合理的决策和任务规划。此外，为了解决 LLM 可能出现的不准确性或不合逻辑的行为，采用了远程操作和动态运动原理（DMP）的组合进行行为校正。这种融合旨在提高 LLM 基础的人机协作系统的实用性和通用性。

基于 LLM 的人机协作框架用于操作任务

LLM-Based Human-Robot Collaboration Framework for Manipulation Tasks

Grasping unseen objects in unconstrained, cluttered environments is an
essential skill for autonomous robotic manipulation. Despite recent progress in
full 6-DoF grasp learning, existing approaches often consist of complex
sequential pipelines that possess several potential failure points and
run-times unsuitable for closed-loop grasping. Therefore, we propose an
end-to-end network that efficiently generates a distribution of 6-DoF
parallel-jaw grasps directly from a depth recording of a scene. Our novel grasp
representation treats 3D points of the recorded point cloud as potential grasp
contacts. By rooting the full 6-DoF grasp pose and width in the observed point
cloud, we can reduce the dimensionality of our grasp representation to 4-DoF
which greatly facilitates the learning process. Our class-agnostic approach is
trained on 17 million simulated grasps and generalizes well to real world
sensor data. In a robotic grasping study of unseen objects in structured
clutter we achieve over 90% success rate, cutting the failure rate in half
compared to a recent state-of-the-art method.

通过一种端到端的神经网络，将深度记录的场景直接转化为 6 个自由度平行夹具抓取的分布，其独特的抓取表示法将点云的 3D 点作为潜在的抓取接触点，并将 6 个自由度的抓取姿态与宽度根据观察到的点云减少到 4 个自由度，成功率可达 90% 以上。

Contact-GraspNet: 在混乱场景下高效生成 6 自由度抓取

Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes

Autonomous robotic manipulation in clutter is challenging. A large variety of
objects must be perceived in complex scenes, where they are partially occluded
and embedded among many distractors, often in restricted spaces. To tackle
these challenges, we developed a deep-learning approach that combines object
detection and semantic segmentation. The manipulation scenes are captured with
RGB-D cameras, for which we developed a depth fusion method. Employing
pretrained features makes learning from small annotated robotic data sets
possible. We evaluate our approach on two challenging data sets: one captured
for the Amazon Picking Challenge 2016, where our team NimbRo came in second in
the Stowing and third in the Picking task, and one captured in
disaster-response scenarios. The experiments show that object detection and
semantic segmentation complement each other and can be combined to yield
reliable object perception.

通过利用深度融合技术和深度学习的物体检测和语义分割算法，我们成功地解决了在复杂场景中感知目标物体的问题，并在 Amazon Picking Challenge2016 和紧急响应场景数据集上进行了验证。