Many state-of-the art robotic applications utilize series elastic actuators
(SEAs) with closed-loop force control to achieve complex tasks such as walking,
lifting, and manipulation. Model-free PID control methods are more prone to
instability due to nonlinearities in the SEA where cascaded model-based robust
controllers can remove these effects to achieve stable force control. However,
these model-based methods require detailed investigations to characterize the
system accurately. Deep reinforcement learning (DRL) has proved to be an
effective model-free method for continuous control tasks, where few works deal
with hardware learning. This paper describes the training process of a DRL
policy on hardware of an SEA pendulum system for tracking force control
trajectories from 0.05 - 0.35 Hz at 50 N amplitude using the Proximal Policy
Optimization (PPO) algorithm. Safety mechanisms are developed and utilized for
training the policy for 12 hours (overnight) without an operator present within
the full 21 hours training period. The tracking performance is evaluated
showing improvements of $25$ N in mean absolute error when comparing the first
18 min. of training to the full 21 hours for a 50 N amplitude, 0.1 Hz sinusoid
desired force trajectory. Finally, the DRL policy exhibits better tracking and
stability margins when compared to a model-free PID controller for a 50 N chirp
force trajectory.

该研究使用深度强化学习（DRL）方法，通过 Proximal Policy Optimization（PPO）算法，在 SEA 振荡系统的硬件环境中训练一个 DRL 策略，以实现高精度的力控制任务。研究结果表明，该 DRL 策略优于传统的基于模型的 PID 控制器，并能够实现更好的跟踪和稳定性保障。