Reinforcement learning for control over continuous spaces typically uses
high-entropy stochastic policies, such as Gaussian distributions, for local
exploration and estimating policy gradient to optimize performance. Many
robotic control problems deal with complex unstable dynamics, where applying
actions that are off the feasible control manifolds can quickly lead to
undesirable divergence. In such cases, most samples taken from the ambient
action space generate low-value trajectories that hardly contribute to policy
improvement, resulting in slow or failed learning. We propose to improve action
selection in this model-free RL setting by introducing additional adaptive
control steps based on Extremum-Seeking Control (ESC). On each action sampled
from stochastic policies, we apply sinusoidal perturbations and query for
estimated Q-values as the response signal. Based on ESC, we then dynamically
improve the sampled actions to be closer to nearby optima before applying them
to the environment. Our methods can be easily added in standard policy
optimization to improve learning efficiency, which we demonstrate in various
control learning environments.

通过引入基于极值搜索控制的自适应控制步骤，改进了模型自由强化学习中的动作选择，在标准策略优化中提高了学习效率。