Reinforcement Learning (RL) generally suffers from poor sample complexity,
mostly due to the need to exhaustively explore the state space to find good
policies. On the other hand, we postulate that expert knowledge of the system
to control often allows us to design simple rules we expect good policies to
follow at all times. In this work, we hence propose a simple yet effective
modification of continuous actor-critic RL frameworks to incorporate such prior
knowledge in the learned policies and constrain them to regions of the state
space that are deemed interesting, thereby significantly accelerating their
convergence. Concretely, we saturate the actions chosen by the agent if they do
not comply with our intuition and, critically, modify the gradient update step
of the policy to ensure the learning process does not suffer from the
saturation step. On a room temperature control simulation case study, these
modifications allow agents to converge to well-performing policies up to one
order of magnitude faster than classical RL agents while retaining good final
performance.

本文提出了一种将先前的专家知识与连续 actor-critic RL 框架相结合的简单且有效的方法，以加速策略的学习过程并减少样本复杂度。结果显示，这种方法使得 RL agent 在保持最终性能的情况下，比传统的 RL agent 更快收敛一个数量级。