The naive application of Reinforcement Learning algorithms to continuous
control problems -- such as locomotion and manipulation -- often results in
policies which rely on high-amplitude, high-frequency control signals, known
colloquially as bang-bang control. Although such solutions may indeed maximize
task reward, they can be unsuitable for real world systems. Bang-bang control
may lead to increased wear and tear or energy consumption, and tends to excite
undesired second-order dynamics. To counteract this issue, multi-objective
optimization can be used to simultaneously optimize both the reward and some
auxiliary cost that discourages undesired (e.g. high-amplitude) control. In
principle, such an approach can yield the sought after, smooth, control
policies. It can, however, be hard to find the correct trade-off between cost
and return that results in the desired behavior. In this paper we propose a new
constraint-based reinforcement learning approach that ensures task success
while minimizing one or more auxiliary costs (such as control effort). We
employ Lagrangian relaxation to learn both (a) the parameters of a control
policy that satisfies the desired constraints and (b) the Lagrangian
multipliers for the optimization. Moreover, we demonstrate that we can satisfy
constraints either in expectation or in a per-step fashion, and can even learn
a single policy that is able to dynamically trade-off between return and cost.
We demonstrate the efficacy of our approach using a number of continuous
control benchmark tasks, a realistic, energy-optimized quadruped locomotion
task, as well as a reaching task on a real robot arm.

提出了一种基于 Lagrangian relaxation 的约束强化学习方法，通过同时优化任务奖励和某些辅助成本来确保任务成功，展示了在连续控制基准任务、优化能量的四足动物运动任务以及实际机器人臂夹取任务上，该方法的有效性。