This paper proves that the episodic learning environment of every
finite-horizon decision task has a unique steady state under any behavior
policy, and that the marginal distribution of the agent's input indeed
converges to the steady-state distribution in essentially all episodic learning
processes. This observation supports an interestingly reversed mindset against
conventional wisdom: While the existence of unique steady states was often
presumed in continual learning but considered less relevant in episodic
learning, it turns out their existence is guaranteed for the latter. Based on
this insight, the paper unifies episodic and continual RL around several
important concepts that have been separately treated in these two RL
formalisms. Practically, the existence of unique and approachable steady state
enables a general way to collect data in episodic RL tasks, which the paper
applies to policy gradient algorithms as a demonstration, based on a new
steady-state policy gradient theorem. Finally, the paper also proposes and
experimentally validates a perturbation method that facilitates rapid
steady-state convergence in real-world RL tasks.

本文证明了每个有限时间决策任务的情节学习环境在任何行为策略下都有一个独特的稳态，并且代理输入的边缘分布在几乎所有情节学习过程中确实会收敛到稳态分布。此观察支持一种反转常规智慧的思维方式。基于这个观察，本文围绕着几个重要的概念统一了情节式和持续式强化学习，并提出并验证了一种有助于在现实 RL 任务中实现快速稳态收敛的扰动方法。

情节性强化学习的稳态分析

Steady State Analysis of Episodic Reinforcement Learning

In this paper we introduce a mathematical model that captures some of the
salient features of recommender systems that are based on popularity and that
try to exploit social ties among the users. We show that, under very general
conditions, the market always converges to a steady state, for which we are
able to give an explicit form. Thanks to this we can tell rather precisely how
much a market is altered by a recommendation system, and determine the power of
users to influence others. Our theoretical results are complemented by
experiments with real world social networks showing that social graphs prevent
large market distortions in spite of the presence of highly influential users.

通过一个数学模型研究了基于流行度和社交关系的推荐系统。结果表明，在一些普遍的条件下，市场总会收敛到一个稳定状态。通过实验进一步证明，实际社交网络中的社交图表现出一定的防止市场失真的作用，即使存在高度具有影响力的用户。