Learning to manipulate 3D objects in an interactive environment has been a
challenging problem in Reinforcement Learning (RL). In particular, it is hard
to train a policy that can generalize over objects with different semantic
categories, diverse shape geometry and versatile functionality. Recently, the
technique of visual affordance has shown great prospects in providing
object-centric information priors with effective actionable semantics. As such,
an effective policy can be trained to open a door by knowing how to exert force
on the handle. However, to learn the affordance, it often requires
human-defined action primitives, which limits the range of applicable tasks. In
this study, we take advantage of visual affordance by using the contact
information generated during the RL training process to predict contact maps of
interest. Such contact prediction process then leads to an end-to-end
affordance learning framework that can generalize over different types of
manipulation tasks. Surprisingly, the effectiveness of such framework holds
even under the multi-stage and the multi-agent scenarios. We tested our method
on eight types of manipulation tasks. Results showed that our methods
outperform baseline algorithms, including visual-based affordance methods and
RL methods, by a large margin on the success rate. The demonstration can be
found at this https URL.

本研究使用视觉契合技术，通过预测关键点的接触信息，提出了一种端到端的控制策略学习框架，可以实现在不同类型的操纵任务中的泛化。结果显示，我们的算法在成功率上大大优于基线算法，包括基于视觉契合和强化学习方法。

机器人操作的端到端操作能力学习

End-to-End Affordance Learning for Robotic Manipulation

The physical and textural attributes of objects have been widely studied for
recognition, detection and segmentation tasks in computer vision. A number of
datasets, such as large scale ImageNet, have been proposed for feature learning
using data hungry deep neural networks and for hand-crafted feature extraction.
To intelligently interact with objects, robots and intelligent machines need
the ability to infer beyond the traditional physical/textural attributes, and
understand/learn visual cues, called visual affordances, for affordance
recognition, detection and segmentation. To date there is no publicly available
large dataset for visual affordance understanding and learning. In this paper,
we introduce a large scale multi-view RGBD visual affordance learning dataset,
a benchmark of 47210 RGBD images from 37 object categories, annotated with 15
visual affordance categories and 35 cluttered/complex scenes with different
objects and multiple affordances. To the best of our knowledge, this is the
first ever and the largest multi-view RGBD visual affordance learning dataset.
We benchmark the proposed dataset for affordance recognition and segmentation.
To achieve this we propose an Affordance Recognition Network a.k.a ARNet. In
addition, four state-of-the-art deep learning networks are evaluated for
affordance segmentation task. Our experimental results showcase the challenging
nature of the dataset and present definite prospects for new and robust
affordance learning algorithms. The dataset is available at:
this https URL.

本文介绍了一个大规模多视角 RGBD 视觉可供性学习数据集及其基准测试方法，它包含了来自 37 种对象类别、配有 15 种视觉可供性类别和 35 种杂乱 / 复杂的场景的 47210 个 RGBD 图像。我们提出了一个名为 ARNet 的【可供性识别网络】，证明了该数据集的挑战性以及提出的新的强大的可供性学习算法的明确前景。

走向视觉功能学习：一个为功能分割和识别设定的基准

Towards Visual Affordance Learning: A Benchmark for Affordance Segmentation and Recognition

The ability to understand the ways to interact with objects from visual cues,
a.k.a. visual affordance, is essential to vision-guided robotic research. This
involves categorizing, segmenting and reasoning of visual affordance. Relevant
studies in 2D and 2.5D image domains have been made previously, however, a
truly functional understanding of object affordance requires learning and
prediction in the 3D physical domain, which is still absent in the community.
In this work, we present a 3D AffordanceNet dataset, a benchmark of 23k shapes
from 23 semantic object categories, annotated with 18 visual affordance
categories. Based on this dataset, we provide three benchmarking tasks for
evaluating visual affordance understanding, including full-shape, partial-view
and rotation-invariant affordance estimations. Three state-of-the-art point
cloud deep learning networks are evaluated on all tasks. In addition we also
investigate a semi-supervised learning setup to explore the possibility to
benefit from unlabeled data. Comprehensive results on our contributed dataset
show the promise of visual affordance understanding as a valuable yet
challenging benchmark.

本论文介绍了一个名为 3D AffordanceNet 的数据集，其中包含来自 23 个语义对象类别的 23k 个形状，注释了 18 个视觉可用性类别，并提供了三个评估视觉可用性理解的基准测试任务。作者评估了三种最先进的点云深度学习网络，并研究了半监督学习设置探索利用未标记数据的可能性。综合结果表明，视觉可用性理解是一个有价值但具有挑战性的基准测试。