In this paper we describe a new conceptual framework that connects
approximate Dynamic Programming (DP), Model Predictive Control (MPC), and
Reinforcement Learning (RL). This framework centers around two algorithms,
which are designed largely independently of each other and operate in synergy
through the powerful mechanism of Newton's method. We call them the off-line
training and the on-line play algorithms. The names are borrowed from some of
the major successes of RL involving games; primary examples are the recent
(2017) AlphaZero program (which plays chess, [SHS17], [SSS17]), and the
similarly structured and earlier (1990s) TD-Gammon program (which plays
backgammon, [Tes94], [Tes95], [TeG96]). In these game contexts, the off-line
training algorithm is the method used to teach the program how to evaluate
positions and to generate good moves at any given position, while the on-line
play algorithm is the method used to play in real time against human or
computer opponents.
Significantly, the synergy between off-line training and on-line play also
underlies MPC (as well as other major classes of sequential decision problems),
and indeed the MPC design architecture is very similar to the one of AlphaZero
and TD-Gammon. This conceptual insight provides a vehicle for bridging the
cultural gap between RL and MPC, and sheds new light on some fundamental issues
in MPC. These include the enhancement of stability properties through rollout,
the treatment of uncertainty through the use of certainty equivalence, the
resilience of MPC in adaptive control settings that involve changing system
parameters, and the insights provided by the superlinear performance bounds
implied by Newton's method.

本文描述了一个将近似动态规划 (DP)、模型预测控制 (MPC) 和强化学习 (RL) 连接起来的新概念框架，其中通过牛顿法的强大机制，离线训练和在线应用算法相互独立地设计，协同运作。这一理论架构为强化学习和模型预测控制之间的文化差距提供了桥梁，并对模型预测控制中的一些基本问题提供了新的见解。