Collecting and automatically obtaining reward signals from real robotic
visual data for the purposes of training reinforcement learning algorithms can
be quite challenging and time-consuming. Methods for utilizing unlabeled data
can have a huge potential to further accelerate robotic learning. We consider
here the problem of performing manipulation tasks from pixels. In such tasks,
choosing an appropriate state representation is crucial for planning and
control. This is even more relevant with real images where noise, occlusions
and resolution affect the accuracy and reliability of state estimation. In this
work, we learn a latent state representation implicitly with deep reinforcement
learning in simulation, and then adapt it to the real domain using unlabeled
real robot data. We propose to do so by optimizing sequence-based self
supervised objectives. These exploit the temporal nature of robot experience,
and can be common in both the simulated and real domains, without assuming any
alignment of underlying states in simulated and unlabeled real images. We
propose Contrastive Forward Dynamics loss, which combines dynamics model
learning with time-contrastive techniques. The learned state representation
that results from our methods can be used to robustly solve a manipulation task
in simulation and to successfully transfer the learned skill on a real system.
We demonstrate the effectiveness of our approaches by training a vision-based
reinforcement learning agent for cube stacking. Agents trained with our method,
using only 5 hours of unlabeled real robot data for adaptation, shows a clear
improvement over domain randomization, and standard visual domain adaptation
techniques for sim-to-real transfer.

使用深度强化学习隐式地在仿真环境中学习潜在的状态表示，并通过无标签的真实机器人数据将其调整到真实环境中，以实现从像素中执行操作任务。通过序列自监督目标优化对比正向动力学损失，提出了一种更加有效的方法，成功地训练基于视觉的强化学习智能体来堆叠方块。