We propose a method for learning expressive energy-based policies for
continuous states and actions, which has been feasible only in tabular domains
before. We apply our method to learning maximum entropy policies, resulting
into a new algorithm, called soft Q-learning, that expresses the optimal policy
via a Boltzmann distribution. We use the recently proposed amortized Stein
variational gradient descent to learn a stochastic sampling network that
approximates samples from this distribution. The benefits of the proposed
algorithm include improved exploration and compositionality that allows
transferring skills between tasks, which we confirm in simulated experiments
with swimming and walking robots. We also draw a connection to actor-critic
methods, which can be viewed performing approximate inference on the
corresponding energy-based model.

提出了一种学习连续状态和动作表达性能量策略的方法，其中软 Q 学习表达了最佳策略，该方法使用 Boltzmann 分布近似样本。通过游泳和行走机器人的模拟实验，证实了该算法的改进探索和组合性，它允许在任务之间转移技能，并且与演员 - 评论员方法存在联系，可以视为对相应能量模型进行近似推断。