One of the main challenges in model-based reinforcement learning (RL) is to
decide which aspects of the environment should be modeled. The
value-equivalence (VE) principle proposes a simple answer to this question: a
model should capture the aspects of the environment that are relevant for
value-based planning. Technically, VE distinguishes models based on a set of
policies and a set of functions: a model is said to be VE to the environment if
the Bellman operators it induces for the policies yield the correct result when
applied to the functions. As the number of policies and functions increase, the
set of VE models shrinks, eventually collapsing to a single point corresponding
to a perfect model. A fundamental question underlying the VE principle is thus
how to select the smallest sets of policies and functions that are sufficient
for planning. In this paper we take an important step towards answering this
question. We start by generalizing the concept of VE to order-$k$ counterparts
defined with respect to $k$ applications of the Bellman operator. This leads to
a family of VE classes that increase in size as $k \rightarrow \infty$. In the
limit, all functions become value functions, and we have a special
instantiation of VE which we call proper VE or simply PVE. Unlike VE, the PVE
class may contain multiple models even in the limit when all value functions
are used. Crucially, all these models are sufficient for planning, meaning that
they will yield an optimal policy despite the fact that they may ignore many
aspects of the environment. We construct a loss function for learning PVE
models and argue that popular algorithms such as MuZero can be understood as
minimizing an upper bound for this loss. We leverage this connection to propose
a modification to MuZero and show that it can lead to improved performance in
practice.

本文研究模型为基础的强化学习中的价值等价原则及其在 MuZero 算法优化中的应用。

适当价值等效性

Proper Value Equivalence

Reinforcement learning tasks are typically specified as Markov decision
processes. This formalism has been highly successful, though specifications
often couple the dynamics of the environment and the learning objective. This
lack of modularity can complicate generalization of the task specification, as
well as obfuscate connections between different task settings, such as episodic
and continuing. In this work, we introduce the RL task formalism, that provides
a unification through simple constructs including a generalization to
transition-based discounting. Through a series of examples, we demonstrate the
generality and utility of this formalism. Finally, we extend standard learning
constructs, including Bellman operators, and extend some seminal theoretical
results, including approximation errors bounds. Overall, we provide a
well-understood and sound formalism on which to build theoretical results and
simplify algorithm use and development.

介绍了一种 RL 任务形式化的方法，通过简单的构造实现了不同任务设置的统一，包括状态转移折扣的概括；并拓展了标准学习构造和部分理论结果，提供了一个可理解和可靠的形式化框架来构建理论和简化算法使用和开发。