We introduce a novel Proximal Policy Optimization (PPO) algorithm aimed at
addressing the challenge of maintaining a uniform proton beam intensity
delivery in the Muon to Electron Conversion Experiment (Mu2e) at Fermi National
Accelerator Laboratory (Fermilab). Our primary objective is to regulate the
spill process to ensure a consistent intensity profile, with the ultimate goal
of creating an automated controller capable of providing real-time feedback and
calibration of the Spill Regulation System (SRS) parameters on a millisecond
timescale. We treat the Mu2e accelerator system as a Markov Decision Process
suitable for Reinforcement Learning (RL), utilizing PPO to reduce bias and
enhance training stability. A key innovation in our approach is the integration
of a neuralized Proportional-Integral-Derivative (PID) controller into the
policy function, resulting in a significant improvement in the Spill Duty
Factor (SDF) by 13.6%, surpassing the performance of the current PID controller
baseline by an additional 1.6%. This paper presents the preliminary offline
results based on a differentiable simulator of the Mu2e accelerator. It paves
the groundwork for real-time implementations and applications, representing a
crucial step towards automated proton beam intensity control for the Mu2e
experiment.

我们引入了一种创新的近端策略优化（PPO）算法，旨在解决费米国家加速器实验室（Fermilab）的 Muon to Electron Conversion Experiment（Mu2e）中保持均匀质子束强度传递的挑战。我们的主要目标是调节流出过程以确保一致的强度分布，最终目标是创建一个能够提供实时反馈和校准 Spill Regulation System（SRS）参数的自动控制器，达到毫秒级的时间尺度。我们将 Mu2e 加速器系统视为适合强化学习（RL）的马尔可夫决策过程，利用 PPO 来减少偏差并增强训练稳定性。我们方法的一个关键创新是将神经化的比例积分微分（PID）控制器整合到策略函数中，导致 Spill Duty Factor（SDF）明显提高了 13.6%，超过了当前 PID 控制器基准的额外 1.6％。本文介绍了基于 Mu2e 加速器的可微模拟器的初步离线结果，为实时实现和应用奠定了基础，是朝着 Mu2e 实验的自动质子束强度控制迈出的关键一步。

超越 PID 控制器：Mu2e 中用神经化 PID 策略的 PPO 质子束强度控制

Beyond PID Controllers: PPO with Neuralized PID Policy for Proton Beam  Intensity Control in Mu2e

While ensuring stability for linear systems is well understood, it remains a
major challenge for systems with nonlinear dynamics. A general approach in such
cases is to leverage Lyapunov stability theory to compute a combination of a
Lyapunov control function and an associated control policy. However, finding
Lyapunov functions for general nonlinear systems is a challenging task. To
address this challenge, several methods have been recently proposed that
represent Lyapunov functions using neural networks. However, such approaches
have been designed exclusively for continuous-time systems. We propose the
first approach for learning neural Lyapunov control in discrete-time systems.
Three key ingredients enable us to effectively learn provably stable control
policies. The first is a novel mixed-integer linear programming approach for
verifying the stability conditions in discrete-time systems. The second is a
novel approach for computing sub-level sets which characterize the region of
attraction. Finally, we rely on a heuristic gradient-based approach for quickly
finding counterexamples to significantly speed up Lyapunov function learning.
Our experiments on four standard benchmarks demonstrate that our approach
significantly outperforms state-of-the-art baselines. For example, on the path
tracking benchmark, we outperform recent neural Lyapunov control baselines by
an order of magnitude in both running time and the size of the region of
attraction, and on two of the four benchmarks (cartpole and PVTOL), ours is the
first automated approach to return a provably stable controller.

提出了一种针对离散时间系统学习神经 Laypunov 控制器的方法，其中包括计算 Lyapunov 控制函数的混合整数线性规划方法、计算子水平集的新方法以及基于启发式梯度的方法以加速学习 Lyapunov 函数。实验结果表明，该方法在四个标准测试中均优于目前的基线方法，并且在车杆和 PVTOL 测试中是第一个自动化控制器的学习方法。