One of the key promises of model-based reinforcement learning is the ability
to generalize using an internal model of the world to make predictions in novel
environments and tasks. However, the generalization ability of model-based
agents is not well understood because existing work has focused on model-free
agents when benchmarking generalization. Here, we explicitly measure the
generalization ability of model-based agents in comparison to their model-free
counterparts. We focus our analysis on MuZero (Schrittwieser et al., 2020), a
powerful model-based agent, and evaluate its performance on both procedural and
task generalization. We identify three factors of procedural generalization --
planning, self-supervised representation learning, and procedural data
diversity -- and show that by combining these techniques, we achieve
state-of-the art generalization performance and data efficiency on Procgen
(Cobbe et al., 2019). However, we find that these factors do not always provide
the same benefits for the task generalization benchmarks in Meta-World (Yu et
al., 2019), indicating that transfer remains a challenge and may require
different approaches than procedural generalization. Overall, we suggest that
building generalizable agents requires moving beyond the single-task,
model-free paradigm and towards self-supervised model-based agents that are
trained in rich, procedural, multi-task environments.

在本文中，我们评估了 MuZero 代理在程序化和任务通用性方面的性能，并通过比较自由模型代理，确定了模型代理的泛化能力。我们发现，计划、自监督表示学习和程序数据多样性是程序化通用性的三个因素，并表明结合这些技术可以实现出色的通用性性能和数据效率。然而，我们发现这些因素不总是为 Meta-World 中的任务通用性基准提供相同的好处。