This paper contributes a new approach for distributional reinforcement
learning which elucidates a clean separation of transition structure and reward
in the learning process. Analogous to how the successor representation (SR)
describes the expected consequences of behaving according to a given policy,
our distributional successor measure (SM) describes the distributional
consequences of this behaviour. We formulate the distributional SM as a
distribution over distributions and provide theory connecting it with
distributional and model-based reinforcement learning. Moreover, we propose an
algorithm that learns the distributional SM from data by minimizing a two-level
maximum mean discrepancy. Key to our method are a number of algorithmic
techniques that are independently valuable for learning generative models of
state. As an illustration of the usefulness of the distributional SM, we show
that it enables zero-shot risk-sensitive policy evaluation in a way that was
not previously possible.

该研究提出了一种新的分布式强化学习方法，它清晰地将过渡结构和奖励在学习过程中分开，通过最小化两层最大均值差异来学习分布式后继度量，进而实现零 - shot 风险敏感策略评估。