Simulation-to-reality (sim-to-real) transfer is a fundamental problem for
robot learning. Domain Randomization, which adds randomization during training,
is a powerful technique that effectively addresses the sim-to-real gap.
However, the noise in observations makes learning significantly harder.
Recently, studies have shown that employing a teacher-student learning paradigm
can accelerate training in randomized environments. Learned with privileged
information, a teacher agent can instruct the student agent to operate in noisy
environments. However, this approach is often not sample efficient as the
experience collected by the teacher is discarded completely when training the
student, wasting information revealed by the environment. In this work, we
extend the teacher-student learning paradigm by proposing a sample efficient
learning framework termed Learn to Teach (L2T) that recycles experience
collected by the teacher agent. We observe that the dynamics of the
environments for both agents remain unchanged, and the state space of the
teacher is coupled with the observation space of the student. We show that a
single-loop algorithm can train both the teacher and student agents under both
Reinforcement Learning and Inverse Reinforcement Learning contexts. We
implement variants of our methods, conduct experiments on the MuJoCo benchmark,
and apply our methods to the Cassie robot locomotion problem. Extensive
experiments show that our method achieves competitive performance while only
requiring environmental interaction with the teacher.

模拟到现实的迁移是机器人学习的一个基本问题，本文提出了一种学习框架，通过引入师生学习范式和样本高效性，解决了培训中噪声观测造成的困难，实现了模拟环境下机器人的高性能表现。

学习教学：提高师生学习中的样本效率，用于模拟到真实场景的迁移

Learn to Teach: Improve Sample Efficiency in Teacher-student Learning  for Sim-to-Real Transfer

Identifying the most suitable variables to represent the state is a
fundamental challenge in Reinforcement Learning (RL). These variables must
efficiently capture the information necessary for making optimal decisions. In
order to address this problem, in this paper, we introduce the Transfer Entropy
Redundancy Criterion (TERC), an information-theoretic criterion, which
determines if there is \textit{entropy transferred} from state variables to
actions during training. We define an algorithm based on TERC that provably
excludes variables from the state that have no effect on the final performance
of the agent, resulting in more sample efficient learning. Experimental results
show that this speed-up is present across three different algorithm classes
(represented by tabular Q-learning, Actor-Critic, and Proximal Policy
Optimization (PPO)) in a variety of environments. Furthermore, to highlight the
differences between the proposed methodology and the current state-of-the-art
feature selection approaches, we present a series of controlled experiments on
synthetic data, before generalizing to real-world decision-making tasks. We
also introduce a representation of the problem that compactly captures the
transfer of information from state variables to actions as Bayesian networks.

在本论文中，我们介绍了一种用于强化学习（RL）的信息论准则 Transfer Entropy Redundancy Criterion (TERC)，该准则确定了训练过程中是否从状态变量到动作变量传递了信息熵。基于 TERC 的算法能够排除对智能体最终性能没有影响的状态变量，从而提高样本利用效率。实验证明这种加速方式在不同算法类别和各种环境中都存在。为了突出与现有特征选择方法的区别，我们在合成数据上进行了一系列对照实验，并推广到现实世界的决策任务。我们还引入了一种用贝叶斯网络紧凑地捕捉状态变量到动作变量的信息传递的问题表征。