Despite success in many challenging problems, reinforcement learning (RL) is
still confronted with sample inefficiency, which can be mitigated by
introducing prior knowledge to agents. However, many transfer techniques in
reinforcement learning make the limiting assumption that the teacher is an
expert. In this paper, we use the action prior from the Reinforcement Learning
as Inference framework - that is, a distribution over actions at each state
which resembles a teacher policy, rather than a Bayesian prior - to recover
state-of-the-art policy distillation techniques. Then, we propose a class of
adaptive methods that can robustly exploit action priors by combining reward
shaping and auxiliary regularization losses. In contrast to prior work, we
develop algorithms for leveraging suboptimal action priors that may
nevertheless impart valuable knowledge - which we call soft action priors. The
proposed algorithms adapt by adjusting the strength of teacher feedback
according to an estimate of the teacher's usefulness in each state. We perform
tabular experiments, which show that the proposed methods achieve
state-of-the-art performance, surpassing it when learning from suboptimal
priors. Finally, we demonstrate the robustness of the adaptive algorithms in
continuous action deep RL problems, in which adaptive algorithms considerably
improved stability when compared to existing policy distillation methods.

通过引入动作先验，结合奖励整形和辅助正则化损失的方法，该论文提出了一种适应性算法，可以高效地利用先验知识，特别是软动作先验，取得了在 Reinforcement Learning 问题上的最新成果和深度 RL 的鲁棒性改进。