Deep neural network based reinforcement learning (RL) can learn appropriate
visual representations for complex tasks like vision-based robotic grasping
without the need for manually engineering or prior learning a perception
system. However, data for RL is collected via running an agent in the desired
environment, and for applications like robotics, running a robot in the real
world may be extremely costly and time consuming. Simulated training offers an
appealing alternative, but ensuring that policies trained in simulation can
transfer effectively into the real world requires additional machinery.
Simulations may not match reality, and typically bridging the
simulation-to-reality gap requires domain knowledge and task-specific
engineering. We can automate this process by employing generative models to
translate simulated images into realistic ones. However, this sort of
translation is typically task-agnostic, in that the translated images may not
preserve all features that are relevant to the task. In this paper, we
introduce the RL-scene consistency loss for image translation, which ensures
that the translation operation is invariant with respect to the Q-values
associated with the image. This allows us to learn a task-aware translation.
Incorporating this loss into unsupervised domain translation, we obtain
RL-CycleGAN, a new approach for simulation-to-real-world transfer for
reinforcement learning. In evaluations of RL-CycleGAN on two vision-based
robotics grasping tasks, we show that RL-CycleGAN offers a substantial
improvement over a number of prior methods for sim-to-real transfer, attaining
excellent real-world performance with only a modest number of real-world
observations.

本论文提出了基于深度神经网络的强化学习在机器人抓取任务中的应用，以及利用生成模型进行图像仿真与真实环境之间的过渡。其中引入了 RL-scene 一致性损失用于图像翻译，可保证翻译操作是不变的，并在实现无监督领域转换的同时，训练出了 RL-CycleGAN 模型，该模型表现出了模拟到现实环境下最优的效果。