Does having visual priors (e.g. the ability to detect objects) facilitate
learning to perform vision-based manipulation (e.g. picking up objects)? We
study this problem under the framework of transfer learning, where the model is
first trained on a passive vision task, and adapted to perform an active
manipulation task. We find that pre-training on vision tasks significantly
improves generalization and sample efficiency for learning to manipulate
objects. However, realizing these gains requires careful selection of which
parts of the model to transfer. Our key insight is that outputs of standard
vision models highly correlate with affordance maps commonly used in
manipulation. Therefore, we explore directly transferring model parameters from
vision networks to affordance prediction networks, and show that this can
result in successful zero-shot adaptation, where a robot can pick up certain
objects with zero robotic experience. With just a small amount of robotic
experience, we can further fine-tune the affordance model to achieve better
results. With just 10 minutes of suction experience or 1 hour of grasping
experience, our method achieves ~80% success rate at picking up novel objects.

使用迁移学习框架，通过对视觉网络和物体操作网络进行预训练和微调，成功地实现了零经验机器人对物体的拾取任务，且只需少量的机器人经验即可获得更好的效果。

学习先看再动：用于操作的视觉预训练

Learning to See before Learning to Act: Visual Pre-training for  Manipulation

How much does having visual priors about the world (e.g. the fact that the
world is 3D) assist in learning to perform downstream motor tasks (e.g.
navigating a complex environment)? What are the consequences of not utilizing
such visual priors in learning? We study these questions by integrating a
generic perceptual skill set (a distance estimator, an edge detector, etc.)
within a reinforcement learning framework (see Fig. 1). This skill set
("mid-level vision") provides the policy with a more processed state of the
world compared to raw images.
Our large-scale study demonstrates that using mid-level vision results in
policies that learn faster, generalize better, and achieve higher final
performance, when compared to learning from scratch and/or using
state-of-the-art visual and non-visual representation learning methods. We show
that conventional computer vision objectives are particularly effective in this
regard and can be conveniently integrated into reinforcement learning
frameworks. Finally, we found that no single visual representation was
universally useful for all downstream tasks, hence we computationally derive a
task-agnostic set of representations optimized to support arbitrary downstream
tasks.

研究利用视觉先验在学习下游动作任务方面的帮助程度及其不使用的后果，通过在强化学习框架中整合感知技能集，探讨中级视觉如何提供比原始图像更加处理过的世界状态，从而实现更快、更好的学习和更高的终端性能，并通过计算导出任务不可知的一组优化的代表性来证明视觉表示不是适用于所有下游任务的普遍工具。