Deep reinforcement learning (RL) algorithms typically parameterize the policy
as a deep network that outputs either a deterministic action or a stochastic
one modeled as a Gaussian distribution, hence restricting learning to a single
behavioral mode. Meanwhile, diffusion models emerged as a powerful framework
for multimodal learning. However, the use of diffusion policies in online RL is
hindered by the intractability of policy likelihood approximation, as well as
the greedy objective of RL methods that can easily skew the policy to a single
mode. This paper presents Deep Diffusion Policy Gradient (DDiffPG), a novel
actor-critic algorithm that learns from scratch multimodal policies
parameterized as diffusion models while discovering and maintaining versatile
behaviors. DDiffPG explores and discovers multiple modes through off-the-shelf
unsupervised clustering combined with novelty-based intrinsic motivation.
DDiffPG forms a multimodal training batch and utilizes mode-specific Q-learning
to mitigate the inherent greediness of the RL objective, ensuring the
improvement of the diffusion policy across all modes. Our approach further
allows the policy to be conditioned on mode-specific embeddings to explicitly
control the learned modes. Empirical studies validate DDiffPG's capability to
master multimodal behaviors in complex, high-dimensional continuous control
tasks with sparse rewards, also showcasing proof-of-concept dynamic online
replanning when navigating mazes with unseen obstacles.

本研究提出了深度扩散策略梯度（DDiffPG）算法，它能够学习参数化为扩散模型的多模态策略，并通过聚类和内在动机探索来发现和维持多样化行为，同时减少强化学习的贪婪性质，实现了模态之间的均衡改善以及对学习模态的显式控制。经验证明，该算法在复杂的高维连续控制任务中能够掌握多模态行为，并展示了在导航未知障碍迷宫中的动态在线重新规划的概念验证。

从头学习多模态行为的扩散策略梯度

Learning Multimodal Behaviors from Scratch with Diffusion Policy  Gradient

In a physical system, changing parameters such as temperature can induce a
phase transition: an abrupt change from one state of matter to another.
Analogous phenomena have recently been observed in large language models.
Typically, the task of identifying phase transitions requires human analysis
and some prior understanding of the system to narrow down which low-dimensional
properties to monitor and analyze. Statistical methods for the automated
detection of phase transitions from data have recently been proposed within the
physics community. These methods are largely system agnostic and, as shown
here, can be adapted to study the behavior of large language models. In
particular, we quantify distributional changes in the generated output via
statistical distances, which can be efficiently estimated with access to the
probability distribution over next-tokens. This versatile approach is capable
of discovering new phases of behavior and unexplored transitions -- an ability
that is particularly exciting in light of the rapid development of language
models and their emergent capabilities.

通过统计距离量化输出生成中的分布变化，本研究利用物理学界提出的自动检测相变的统计方法，成功地将其应用于大型语言模型中，以发现新的行为状态和未探索的转变。