Reinforcement learning (RL) has become a promising approach to developing
controllers for quadrupedal robots. Conventionally, an RL design for locomotion
follows a position-based paradigm, wherein an RL policy outputs target joint
positions at a low frequency that are then tracked by a high-frequency
proportional-derivative (PD) controller to produce joint torques. In contrast,
for the model-based control of quadrupedal locomotion, there has been a
paradigm shift from position-based control to torque-based control. In light of
the recent advances in model-based control, we explore an alternative to the
position-based RL paradigm, by introducing a torque-based RL framework, where
an RL policy directly predicts joint torques at a high frequency, thus
circumventing the use of a PD controller. The proposed learning torque control
framework is validated with extensive experiments, in which a quadruped is
capable of traversing various terrain and resisting external disturbances while
following user-specified commands. Furthermore, compared to learning position
control, learning torque control demonstrates the potential to achieve a higher
reward and is more robust to significant external disturbances. To our
knowledge, this is the first sim-to-real attempt for end-to-end learning torque
control of quadrupedal locomotion.

本研究探索了一种基于扭矩的强化学习框架，此模型成功验证了对于模型控制四足动物而言，相较于基于位置的强化学习方法，基于扭矩的强化学习方法能够产生更好的奖励和更强的抗干扰能力，并能够使四足动物在各种地形上移动，是关于四足机器人端到端学习扭矩控制的首次模拟到实际应用的尝试。