The model-based reinforcement learning paradigm, which uses planning
algorithms and neural network models, has recently achieved unprecedented
results in diverse applications, leading to what is now known as deep
reinforcement learning. These agents are quite complex and involve multiple
components, factors that can create challenges for research. In this work, we
propose a new modular software architecture suited for these types of agents,
and a set of building blocks that can be easily reused and assembled to
construct new model-based reinforcement learning agents. These building blocks
include planning algorithms, policies, and loss functions.
We illustrate the use of this architecture by combining several of these
building blocks to implement and test agents that are optimized to three
different test environments: Cartpole, Minigrid, and Tictactoe. One particular
planning algorithm, made available in our implementation and not previously
used in reinforcement learning, which we called averaged minimax, achieved good
results in the three tested environments.
Experiments performed with this architecture have shown that the best
combination of planning algorithm, policy, and loss function is heavily problem
dependent. This result provides evidence that the proposed architecture, which
is modular and reusable, is useful for reinforcement learning researchers who
want to study new environments and techniques.

本文提出基于模型的强化学习范式中的新型模块化软件架构和一组可以方便地重复使用和组装以构建新的基于模型的强化学习智能体的构建块，包括规划算法，策略以及损失函数，实验证明了该架构是有用的，可用于研究新环境和技术。