We investigate policy transfer using image-to-semantics translation to
mitigate learning difficulties in vision-based robotics control agents. This
problem assumes two environments: a simulator environment with semantics, that
is, low-dimensional and essential information, as the state space, and a
real-world environment with images as the state space. By learning mapping from
images to semantics, we can transfer a policy, pre-trained in the simulator, to
the real world, thereby eliminating real-world on-policy agent interactions to
learn, which are costly and risky. In addition, using image-to-semantics
mapping is advantageous in terms of the computational efficiency to train the
policy and the interpretability of the obtained policy over other types of
sim-to-real transfer strategies. To tackle the main difficulty in learning
image-to-semantics mapping, namely the human annotation cost for producing a
training dataset, we propose two techniques: pair augmentation with the
transition function in the simulator environment and active learning. We
observed a reduction in the annotation cost without a decline in the
performance of the transfer, and the proposed approach outperformed the
existing approach without annotation.

本研究探讨采用图像到语义翻译技术进行政策传递，缓解基于视觉的机器人控制代理的学习困难问题。通过学习从图像到语义的映射，我们可以将在模拟器中预先训练的政策传递到现实世界中，从而消除学习花费和风险高的现实世界上策略的实时交互学习。此外，使用图像到语义映射在训练策略时具有计算效率和可解释性优势。我们提出了两种技术：在模拟器环境中使用转换函数的配对增强技术和主动学习，以解决图像到语义映射中的主要困难，即为产生训练数据的人工注释成本。我们观察到注释成本的降低，而传递的性能不会下降。因此，所提出的方法优于现有的未经注释的方法。

强化学习中的策略转移的小样本图像到语义翻译

Few-Shot Image-to-Semantics Translation for Policy Transfer in Reinforcement Learning

This paper presents a benchmarking study of some of the state-of-the-art
reinforcement learning algorithms used for solving two simulated vision-based
robotics problems. The algorithms considered in this study include soft
actor-critic (SAC), proximal policy optimization (PPO), interpolated policy
gradients (IPG), and their variants with Hindsight Experience replay (HER). The
performances of these algorithms are compared against PyBullet's two simulation
environments known as KukaDiverseObjectEnv and RacecarZEDGymEnv respectively.
The state observations in these environments are available in the form of RGB
images and the action space is continuous, making them difficult to solve. A
number of strategies are suggested to provide intermediate hindsight goals
required for implementing HER algorithm on these problems which are essentially
single-goal environments. In addition, a number of feature extraction
architectures are proposed to incorporate spatial and temporal attention in the
learning process. Through rigorous simulation experiments, the improvement
achieved with these components are established. To the best of our knowledge,
such a benchmarking study is not available for the above two vision-based
robotics problems making it a novel contribution in the field.

本文通过比较常用强化学习算法及其变种在两个模拟视觉机器人问题中的表现，提出的策略包括提供实现 HER 算法所需中间预测目标，搭建空间和时间关注机制。结果表明，在这些环境中应用这些策略可以取得更好的结果。本文提出的基准测试方法为领域做出了新的贡献。