The varying significance of distinct primitive behaviors during the policy
learning process has been overlooked by prior model-free RL algorithms.
Leveraging this insight, we explore the causal relationship between different
action dimensions and rewards to evaluate the significance of various primitive
behaviors during training. We introduce a causality-aware entropy term that
effectively identifies and prioritizes actions with high potential impacts for
efficient exploration. Furthermore, to prevent excessive focus on specific
primitive behaviors, we analyze the gradient dormancy phenomenon and introduce
a dormancy-guided reset mechanism to further enhance the efficacy of our
method. Our proposed algorithm, ACE: Off-policy Actor-critic with
Causality-aware Entropy regularization, demonstrates a substantial performance
advantage across 29 diverse continuous control tasks spanning 7 domains
compared to model-free RL baselines, which underscores the effectiveness,
versatility, and efficient sample efficiency of our approach. Benchmark results
and videos are available at this https URL

我们提出了 ACE 算法：基于因果关系的策略梯度法，通过引入因果关系熵项和梯度休眠现象引导重置机制，取得了在连续控制任务上显著的性能优势。