We present an approach to model-based RL that achieves a new state of the art performance on the challenging Craftax-classic benchmark, an open-world 2D survival game that requires agents to exhibit a wide range of general abilities -- such as strong generalization, deep exploration, and long-term reasoning. With a series of careful design choices aimed at improving sample efficiency, our MBRL algorithm achieves a reward of 67.4% after only 1M environment steps, significantly outperforming DreamerV3, which achieves 53.2%, and, for the first time, exceeds human performance of 65.0%. Our method starts by constructing a SOTA model-free baseline, using a novel policy architecture that combines CNNs and RNNs. We then add three improvements to the standard MBRL setup: (a) "Dyna with warmup", which trains the policy on real and imaginary data, (b) "nearest neighbor tokenizer" on image patches, which improves the scheme to create the transformer world model (TWM) inputs, and (c) "block teacher forcing", which allows the TWM to reason jointly about the future tokens of the next timestep.

本研究解决了强化学习中样本效率不足的问题，提出了一种新的模型基础强化学习（MBRL）算法，通过精心设计的选择以改善样本效率，并在Craftax-classic基准测试中取得了67.4%的奖励，超越了DreamerV3（53.2%）和人类表现（65.0%）。该方法使用了一种结合CNN和RNN的新型策略架构，并通过“热身的动态”、“最近邻标记器”和“区块教师强制”等三项改进提升了模型表现。

提高变压器世界模型以实现数据高效的强化学习