In the context of rapid advancements in industrial automation, vision-based
robotic grasping plays an increasingly crucial role. In order to enhance visual
recognition accuracy, the utilization of large-scale datasets is imperative for
training models to acquire implicit knowledge related to the handling of
various objects. Creating datasets from scratch is a time and labor-intensive
process. Moreover, existing datasets often contain errors due to automated
annotations aimed at expediency, making the improvement of these datasets a
substantial research challenge. Consequently, several issues have been
identified in the annotation of grasp bounding boxes within the popular
Jacquard Grasp. We propose utilizing a Human-In-The-Loop(HIL) method to enhance
dataset quality. This approach relies on backbone deep learning networks to
predict object positions and orientations for robotic grasping. Predictions
with Intersection over Union (IOU) values below 0.2 undergo an assessment by
human operators. After their evaluation, the data is categorized into False
Negatives(FN) and True Negatives(TN). FN are then subcategorized into either
missing annotations or catastrophic labeling errors. Images lacking labels are
augmented with valid grasp bounding box information, whereas images afflicted
by catastrophic labeling errors are completely removed. The open-source tool
Labelbee was employed for 53,026 iterations of HIL dataset enhancement, leading
to the removal of 2,884 images and the incorporation of ground truth
information for 30,292 images. The enhanced dataset, named the Jacquard V2
Grasping Dataset, served as the training data for a range of neural networks.

利用人机互补方法改进 Jacquard 抓取数据集的研究，通过深度学习网络预测物体位置和方向，结合人工操作者的评估，改善了数据集的质量并用于训练神经网络模型。

Jacquard V2：使用人机协同数据校正方法优化数据集

Jacquard V2: Refining Datasets using the Human In the Loop Data  Correction Method

Extraterrestrial rovers with a general-purpose robotic arm have many
potential applications in lunar and planetary exploration. Introducing autonomy
into such systems is desirable for increasing the time that rovers can spend
gathering scientific data and collecting samples. This work investigates the
applicability of deep reinforcement learning for vision-based robotic grasping
of objects on the Moon. A novel simulation environment with
procedurally-generated datasets is created to train agents under challenging
conditions in unstructured scenes with uneven terrain and harsh illumination. A
model-free off-policy actor-critic algorithm is then employed for end-to-end
learning of a policy that directly maps compact octree observations to
continuous actions in Cartesian space. Experimental evaluation indicates that
3D data representations enable more effective learning of manipulation skills
when compared to traditionally used image-based observations. Domain
randomization improves the generalization of learned policies to novel scenes
with previously unseen objects and different illumination conditions. To this
end, we demonstrate zero-shot sim-to-real transfer by evaluating trained agents
on a real robot in a Moon-analogue facility.

此项研究探讨了使用深度强化学习对月球上基于视觉的机械臂进行抓取操作的智能化控制，通过虚拟环境的数据训练及域随机化技术，实现零样本迁移并成功验证实际机器人在类月环境中的操作效果。

利用深度强化学习从 3DOctree 观测中学习在月球表面抓取

Learning to Grasp on the Moon from 3D Octree Observations with Deep Reinforcement Learning

This paper presents a comprehensive survey on vision-based robotic grasping.
We conclude three key tasks during vision-based robotic grasping, which are
object localization, object pose estimation and grasp estimation. In detail,
the object localization task contains object localization without
classification, object detection and object instance segmentation. This task
provides the regions of the target object in the input data. The object pose
estimation task mainly refers to estimating the 6D object pose and includes
correspondence-based methods, template-based methods and voting-based methods,
which affords the generation of grasp poses for known objects. The grasp
estimation task includes 2D planar grasp methods and 6DoF grasp methods, where
the former is constrained to grasp from one direction. These three tasks could
accomplish the robotic grasping with different combinations. Lots of object
pose estimation methods need not object localization, and they conduct object
localization and object pose estimation jointly. Lots of grasp estimation
methods need not object localization and object pose estimation, and they
conduct grasp estimation in an end-to-end manner. Both traditional methods and
latest deep learning-based methods based on the RGB-D image inputs are reviewed
elaborately in this survey. Related datasets and comparisons between
state-of-the-art methods are summarized as well. In addition, challenges about
vision-based robotic grasping and future directions in addressing these
challenges are also pointed out.

本文针对基于视觉的机器人抓取进行了综合调查，总结了视觉基于机器人抓取的三个关键任务：物体定位、物体位姿估计和抓取估计，并详细介绍了这些任务中的传统方法和最新的基于 RGB-D 图像输入的深度学习方法，并总结了相关数据集和最先进方法之间的比较，同时也指出了视觉基于机器人抓取面临的挑战和未来方向。