Integral reinforcement learning (IntRL) demands the precise computation of
the utility function's integral at its policy evaluation (PEV) stage. This is
achieved through quadrature rules, which are weighted sums of utility functions
evaluated from state samples obtained in discrete time. Our research reveals a
critical yet underexplored phenomenon: the choice of the computational method
-- in this case, the quadrature rule -- can significantly impact control
performance. This impact is traced back to the fact that computational errors
introduced in the PEV stage can affect the policy iteration's convergence
behavior, which in turn affects the learned controller. To elucidate how
computation impacts control, we draw a parallel between IntRL's policy
iteration and Newton's method applied to the Hamilton-Jacobi-Bellman equation.
In this light, computational error in PEV manifests as an extra error term in
each iteration of Newton's method, with its upper bound proportional to the
computational error. Further, we demonstrate that when the utility function
resides in a reproducing kernel Hilbert space (RKHS), the optimal quadrature is
achievable by employing Bayesian quadrature with the RKHS-inducing kernel
function. We prove that the local convergence rates for IntRL using the
trapezoidal rule and Bayesian quadrature with a Mat\'ern kernel to be
$O(N^{-2})$ and $O(N^{-b})$, where $N$ is the number of evenly-spaced samples
and $b$ is the Mat\'ern kernel's smoothness parameter. These theoretical
findings are finally validated by two canonical control tasks.

积分强化学习中的计算方法选择（如求积法则）对控制性能产生显著影响，并通过与哈密顿 - 雅可比 - 贝尔曼方程的牛顿法相比较，揭示了计算误差在策略迭代中的额外误差项，并且在使用再生核希尔伯特空间中的效用函数时，通过贝叶斯求积法与诱导核函数结合可以实现最佳求积，其中理论发现最终通过两个经典控制任务得到验证。

连续时间控制中积分增强学习的计算影响

Impact of Computation in Integral Reinforcement Learning for  Continuous-Time Control

We propose the use of machine learning techniques to find optimal quadrature
rules for the construction of stiffness and mass matrices in isogeometric
analysis (IGA). We initially consider 1D spline spaces of arbitrary degree
spanned over uniform and non-uniform knot sequences, and then the generated
optimal rules are used for integration over higher-dimensional spaces using
tensor product sense. The quadrature rule search is posed as an optimization
problem and solved by a machine learning strategy based on gradient-descent.
However, since the optimization space is highly non-convex, the success of the
search strongly depends on the number of quadrature points and the parameter
initialization. Thus, we use a dynamic programming strategy that initializes
the parameters from the optimal solution over the spline space with a lower
number of knots. With this method, we found optimal quadrature rules for spline
spaces when using IGA discretizations with up to 50 uniform elements and
polynomial degrees up to 8, showing the generality of the approach in this
scenario. For non-uniform partitions, the method also finds an optimal rule in
a reasonable number of test cases. We also assess the generated optimal rules
in two practical case studies, namely, the eigenvalue problem of the Laplace
operator and the eigenfrequency analysis of freeform curved beams, where the
latter problem shows the applicability of the method to curved geometries. In
particular, the proposed method results in savings with respect to traditional
Gaussian integration of up to 44% in 1D, 68% in 2D, and 82% in 3D spaces.

该研究使用机器学习技术提出了一种优化静态和动态刚度和质量矩阵构造的四面体数值积分规则方法，实现对不规则网格几何建模的精度处理。