Continuous-time nonlinear optimal control problems hold great promise in
real-world applications. After decades of development, reinforcement learning
(RL) has achieved some of the greatest successes as a general nonlinear control
design method. However, a recent comprehensive analysis of state-of-the-art
continuous-time RL (CT-RL) methods, namely, adaptive dynamic programming
(ADP)-based CT-RL algorithms, reveals they face significant design challenges
due to their complexity, numerical conditioning, and dimensional scaling
issues. Despite advanced theoretical results, existing ADP CT-RL synthesis
methods are inadequate in solving even small, academic problems. The goal of
this work is thus to introduce a suite of new CT-RL algorithms for control of
affine nonlinear systems. Our design approach relies on two important factors.
First, our methods are applicable to physical systems that can be partitioned
into smaller subproblems. This constructive consideration results in reduced
dimensionality and greatly improved intuitiveness of design. Second, we
introduce a new excitation framework to improve persistence of excitation (PE)
and numerical conditioning performance via classical input/output insights.
Such a design-centric approach is the first of its kind in the ADP CT-RL
community. In this paper, we progressively introduce a suite of (decentralized)
excitable integral reinforcement learning (EIRL) algorithms. We provide
convergence and closed-loop stability guarantees, and we demonstrate these
guarantees on a significant application problem of controlling an unstable,
nonminimum phase hypersonic vehicle (HSV).

连续时间非线性最优控制问题中的强化学习方法面临复杂性、数值条件和维度扩展等挑战。该论文介绍了新的强化学习算法，应用于仿射非线性系统的控制，并引入了新的激发框架以优化性能。