We propose a reinforcement learning (RL) framework under a broad class of risk objectives, characterized by convex scoring functions. This class covers many common risk measures, such as variance, Expected Shortfall, entropic Value-at-Risk, and mean-risk utility. To resolve the time-inconsistency issue, we consider an augmented state space and an auxiliary variable and recast the problem as a two-state optimization problem. We propose a customized Actor-Critic algorithm and establish some theoretical approximation guarantees. A key theoretical contribution is that our results do not require the Markov decision process to be continuous. Additionally, we propose an auxiliary variable sampling method inspired by the alternating minimization algorithm, which is convergent under certain conditions. We validate our approach in simulation experiments with a financial application in statistical arbitrage trading, demonstrating the effectiveness of the algorithm.

本文解决了风险目标下强化学习的问题，采用了一类广泛的凸评分函数，该方法涵盖了多种常见风险衡量标准。通过引入辅助变量和增强状态空间，提出了一种定制的演员-评论家算法，为不需要连续马尔可夫决策过程的理论贡献奠定基础，实验证明该算法在统计套利交易中的有效性。

基于凸评分函数的风险敏感强化学习