Ensuring safety of nonlinear systems under model uncertainty and external
disturbances is crucial, especially for real-world control tasks. Predictive
methods such as robust model predictive control (RMPC) require solving
nonconvex optimization problems online, which leads to high computational
burden and poor scalability. Reinforcement learning (RL) works well with
complex systems, but pays the price of losing rigorous safety guarantee. This
paper presents a theoretical framework that bridges the advantages of both RMPC
and RL to synthesize safety filters for nonlinear systems with state- and
action-dependent uncertainty. We decompose the robust invariant set (RIS) into
two parts: a target set that aligns with terminal region design of RMPC, and a
reach-avoid set that accounts for the rest of RIS. We propose a policy
iteration approach for robust reach-avoid problems and establish its monotone
convergence. This method sets the stage for an adversarial actor-critic deep RL
algorithm, which simultaneously synthesizes a reach-avoid policy network, a
disturbance policy network, and a reach-avoid value network. The learned
reach-avoid policy network is utilized to generate nominal trajectories for
online verification, which filters potentially unsafe actions that may drive
the system into unsafe regions when worst-case disturbances are applied. We
formulate a second-order cone programming (SOCP) approach for online
verification using system level synthesis, which optimizes for the worst-case
reach-avoid value of any possible trajectories. The proposed safety filter
requires much lower computational complexity than RMPC and still enjoys
persistent robust safety guarantee. The effectiveness of our method is
illustrated through a numerical example.

提出了一个理论框架，通过捕捉非线性系统中的状态和动作相关的不确定性，综合应用稳健模型预测控制（RMPC）和强化学习（RL），以合成非线性系统的安全过滤器，其计算复杂度较低且具有持久的鲁棒安全保证。

通过鲁棒不变集的分解学习预测性安全过滤器

Learning Predictive Safety Filter via Decomposition of Robust Invariant  Set

Safety is a primary concern when applying reinforcement learning to
real-world control tasks, especially in the presence of external disturbances.
However, existing safe reinforcement learning algorithms rarely account for
external disturbances, limiting their applicability and robustness in practice.
To address this challenge, this paper proposes a robust safe reinforcement
learning framework that tackles worst-case disturbances. First, this paper
presents a policy iteration scheme to solve for the robust invariant set, i.e.,
a subset of the safe set, where persistent safety is only possible for states
within. The key idea is to establish a two-player zero-sum game by leveraging
the safety value function in Hamilton-Jacobi reachability analysis, in which
the protagonist (i.e., control inputs) aims to maintain safety and the
adversary (i.e., external disturbances) tries to break down safety. This paper
proves that the proposed policy iteration algorithm converges monotonically to
the maximal robust invariant set. Second, this paper integrates the proposed
policy iteration scheme into a constrained reinforcement learning algorithm
that simultaneously synthesizes the robust invariant set and uses it for
constrained policy optimization. This algorithm tackles both optimality and
safety, i.e., learning a policy that attains high rewards while maintaining
safety under worst-case disturbances. Experiments on classic control tasks show
that the proposed method achieves zero constraint violation with learned
worst-case adversarial disturbances, while other baseline algorithms violate
the safety constraints substantially. Our proposed method also attains
comparable performance as the baselines even in the absence of the adversary.

该论文提出了一种鲁棒安全强化学习框架，解决了在真实控制任务中应用强化学习时外部干扰的安全性问题，该框架通过建立鲁棒不变集合来保证安全，并采用约束强化学习算法进行策略优化。