The field of risk-constrained reinforcement learning (RCRL) has been
developed to effectively reduce the likelihood of worst-case scenarios by
explicitly handling risk-measure-based constraints. However, the nonlinearity
of risk measures makes it challenging to achieve convergence and optimality. To
overcome the difficulties posed by the nonlinearity, we propose a spectral risk
measure-constrained RL algorithm, spectral-risk-constrained policy optimization
(SRCPO), a bilevel optimization approach that utilizes the duality of spectral
risk measures. In the bilevel optimization structure, the outer problem
involves optimizing dual variables derived from the risk measures, while the
inner problem involves finding an optimal policy given these dual variables.
The proposed method, to the best of our knowledge, is the first to guarantee
convergence to an optimum in the tabular setting. Furthermore, the proposed
method has been evaluated on continuous control tasks and showed the best
performance among other RCRL algorithms satisfying the constraints.

我们提出了一种基于谱风险度量约束的增强学习算法，该算法利用了谱风险度量的对偶性，通过双层优化结构来实现收敛性和最优性，从而在表格设置中保证了最佳性能，并在连续控制任务中展现出了最好的性能。

具收敛保证的谱风险安全强化学习

Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees

We consider the problem of estimating a spectral risk measure (SRM) from
i.i.d. samples, and propose a novel method that is based on numerical
integration. We show that our SRM estimate concentrates exponentially, when the
underlying distribution has bounded support. Further, we also consider the case
when the underlying distribution is either Gaussian or exponential, and derive
a concentration bound for our estimation scheme. We validate the theoretical
findings on a synthetic setup, and in a vehicular traffic routing application.

提出一种基于数值积分的估算谱风险度量值的新方法，通过限定支撑集，实现估算浓度的指数级集中，并在合成和交通路由应用中验证理论发现。