Constrained reinforcement learning (RL) is an area of RL whose objective is
to find an optimal policy that maximizes expected cumulative return while
satisfying a given constraint. Most of the previous constrained RL works
consider expected cumulative sum cost as the constraint. However, optimization
with this constraint cannot guarantee a target probability of outage event that
the cumulative sum cost exceeds a given threshold. This paper proposes a
framework, named Quantile Constrained RL (QCRL), to constrain the quantile of
the distribution of the cumulative sum cost that is a necessary and sufficient
condition to satisfy the outage constraint. This is the first work that tackles
the issue of applying the policy gradient theorem to the quantile and provides
theoretical results for approximating the gradient of the quantile. Based on
the derived theoretical results and the technique of the Lagrange multiplier,
we construct a constrained RL algorithm named Quantile Constrained Policy
Optimization (QCPO). We use distributional RL with the Large Deviation
Principle (LDP) to estimate quantiles and tail probability of the cumulative
sum cost for the implementation of QCPO. The implemented algorithm satisfies
the outage probability constraint after the training period.

本文提出了一种框架，名为 Quantile Constrained RL (QCRL)，用于约束累积成本总和的分布分位数，并利用 LDP 来估计 QCPO 中的分位数和尾概率。