Traditional Reinforcement Learning (RL) algorithms are usually applied in robotics to learn controllers that act with a fixed control rate. Given the discrete nature of RL algorithms, they are oblivious to the effects of the choice of control rate: finding the correct control rate can be difficult and mistakes often result in excessive use of computing resources or even lack of convergence. We propose Soft Elastic Actor-Critic (SEAC), a novel off-policy actor-critic algorithm to address this issue. SEAC implements elastic time steps, time steps with a known, variable duration, which allow the agent to change its control frequency to adapt to the situation. In practice, SEAC applies control only when necessary, minimizing computational resources and data usage. We evaluate SEAC's capabilities in simulation in a Newtonian kinematics maze navigation task and on a 3D racing video game, Trackmania. SEAC outperforms the SAC baseline in terms of energy efficiency and overall time management, and most importantly without the need to identify a control frequency for the learned controller. SEAC demonstrated faster and more stable training speeds than SAC, especially at control rates where SAC struggled to converge. We also compared SEAC with a similar approach, the Continuous-Time Continuous-Options (CTCO) model, and SEAC resulted in better task performance. These findings highlight the potential of SEAC for practical, real-world RL applications in robotics.

传统强化学习算法通常在机器人技术中用于学习具有固定控制频率的控制器。本文提出Soft Elastic Actor-Critic（SEAC），这是一种新颖的离策略演员-评论家算法，旨在解决控制频率选择问题，通过实现弹性时间步长（具有已知可变时长的时间步长），使代理能够根据情况变化其控制频率，从而最大程度地减少计算资源和数据使用，提高能量效率和总体时间管理能力，尤其是在强化学习算法难以收敛的控制频率下，SEAC表现出更快速和更稳定的训练速度。此外，与类似方法Continuous-Time Continuous-Options（CTCO）模型相比，SEAC的任务性能更好，突出了SEAC在实际机器人强化学习应用中的潜力。

弹性时间步长的强化学习