Humans inherently possess generalizable visual representations that empower
them to efficiently explore and interact with the environments in manipulation
tasks. We advocate that such a representation automatically arises from
simultaneously learning about multiple simple perceptual skills that are
critical for everyday scenarios (e.g., hand detection, state estimate, etc.)
and is better suited for learning robot manipulation policies compared to
current state-of-the-art visual representations purely based on self-supervised
objectives. We formalize this idea through the lens of human-oriented
multi-task fine-tuning on top of pre-trained visual encoders, where each task
is a perceptual skill tied to human-environment interactions. We introduce Task
Fusion Decoder as a plug-and-play embedding translator that utilizes the
underlying relationships among these perceptual skills to guide the
representation learning towards encoding meaningful structure for what's
important for all perceptual skills, ultimately empowering learning of
downstream robotic manipulation tasks. Extensive experiments across a range of
robotic tasks and embodiments, in both simulations and real-world environments,
show that our Task Fusion Decoder consistently improves the representation of
three state-of-the-art visual encoders including R3M, MVP, and EgoVLP, for
downstream manipulation policy-learning. Project page:
this https URL

人类具有内在的通用视觉表征，使其能够高效地探索和与环境进行物体操控。本研究提出使用多任务微调的方式在经过预训练的视觉编码器上学习感知技能，通过任务融合解码器指导表示学习，使得对于所有感知技能来说，学习编码的结构能够更好地表示重要信息，最终为下游的机器人操控任务提供帮助。大量实验验证了任务融合解码器在多个机器人任务和仿真及现实环境中对于三种最先进的视觉编码器（R3M、MVP 和 EgoVLP）的表示进行了改进，提升了下游操控策略的学习性能。

面向人类的机器人操作的表示学习

Human-oriented Representation Learning for Robotic Manipulation

How much does having visual priors about the world (e.g. the fact that the
world is 3D) assist in learning to perform downstream motor tasks (e.g.
navigating a complex environment)? What are the consequences of not utilizing
such visual priors in learning? We study these questions by integrating a
generic perceptual skill set (a distance estimator, an edge detector, etc.)
within a reinforcement learning framework (see Fig. 1). This skill set
("mid-level vision") provides the policy with a more processed state of the
world compared to raw images.
Our large-scale study demonstrates that using mid-level vision results in
policies that learn faster, generalize better, and achieve higher final
performance, when compared to learning from scratch and/or using
state-of-the-art visual and non-visual representation learning methods. We show
that conventional computer vision objectives are particularly effective in this
regard and can be conveniently integrated into reinforcement learning
frameworks. Finally, we found that no single visual representation was
universally useful for all downstream tasks, hence we computationally derive a
task-agnostic set of representations optimized to support arbitrary downstream
tasks.

研究利用视觉先验在学习下游动作任务方面的帮助程度及其不使用的后果，通过在强化学习框架中整合感知技能集，探讨中级视觉如何提供比原始图像更加处理过的世界状态，从而实现更快、更好的学习和更高的终端性能，并通过计算导出任务不可知的一组优化的代表性来证明视觉表示不是适用于所有下游任务的普遍工具。