In real-world decision-making problems, for instance in the fields of
finance, robotics or autonomous driving, keeping uncertainty under control is
as important as maximizing expected returns. Risk aversion has been addressed
in the reinforcement learning literature through risk measures related to the
variance of returns. However, in many cases, the risk is measured not only on a
long-term perspective, but also on the step-wise rewards (e.g., in trading, to
ensure the stability of the investment bank, it is essential to monitor the
risk of portfolio positions on a daily basis). In this paper, we define a novel
measure of risk, which we call reward volatility, consisting of the variance of
the rewards under the state-occupancy measure. We show that the reward
volatility bounds the return variance so that reducing the former also
constrains the latter. We derive a policy gradient theorem with a new objective
function that exploits the mean-volatility relationship, and develop an
actor-only algorithm. Furthermore, thanks to the linearity of the Bellman
equations defined under the new objective function, it is possible to adapt the
well-known policy gradient algorithms with monotonic improvement guarantees
such as TRPO in a risk-averse manner. Finally, we test the proposed approach in
two simulated financial environments.

本文提出了一种新的风险评估指标 —— 奖励波动率，并建立了一个基于改进后的奖励波动率指标的策略梯度定理。通过在两个模拟的金融环境下进行测试，证明了该方法的有效性。