Exploration-exploitation dilemma has long been a crucial issue in
reinforcement learning. In this paper, we propose a new approach to
automatically balance between these two. Our method is built upon the Soft
Actor-Critic (SAC) algorithm, which uses an "entropy temperature" that balances
the original task reward and the policy entropy, and hence controls the
trade-off between exploitation and exploration. It is empirically shown that
SAC is very sensitive to this hyperparameter, and the follow-up work (SAC-v2),
which uses constrained optimization for automatic adjustment, has some
limitations. The core of our method, namely Meta-SAC, is to use metagradient
along with a novel meta objective to automatically tune the entropy temperature
in SAC. We show that Meta-SAC achieves promising performances on several of the
Mujoco benchmarking tasks, and outperforms SAC-v2 over 10% in one of the most
challenging tasks, humanoid-v2.

本文提出一种新方法 Meta-SAC，基于 Soft Actor-Critic (SAC) 算法，使用元梯度及元目标来自动调整熵温度，以平衡任务原始奖励与策略熵，从而解决了探索利用困境的问题，并在 Mujoco 基准测试任务中达到了令人满意的性能。