The field of risk-constrained reinforcement learning (RCRL) has been
developed to effectively reduce the likelihood of worst-case scenarios by
explicitly handling risk-measure-based constraints. However, the nonlinearity
of risk measures makes it challenging to achieve convergence and optimality. To
overcome the difficulties posed by the nonlinearity, we propose a spectral risk
measure-constrained RL algorithm, spectral-risk-constrained policy optimization
(SRCPO), a bilevel optimization approach that utilizes the duality of spectral
risk measures. In the bilevel optimization structure, the outer problem
involves optimizing dual variables derived from the risk measures, while the
inner problem involves finding an optimal policy given these dual variables.
The proposed method, to the best of our knowledge, is the first to guarantee
convergence to an optimum in the tabular setting. Furthermore, the proposed
method has been evaluated on continuous control tasks and showed the best
performance among other RCRL algorithms satisfying the constraints.

我们提出了一种基于谱风险度量约束的增强学习算法，该算法利用了谱风险度量的对偶性，通过双层优化结构来实现收敛性和最优性，从而在表格设置中保证了最佳性能，并在连续控制任务中展现出了最好的性能。