We present a mean-variance policy iteration (MVPI) framework for risk-averse
control in a discounted infinite horizon MDP optimizing the variance of a
per-step reward random variable. MVPI enjoys great flexibility in that any
policy evaluation method and risk-neutral control method can be dropped in for
risk-averse control off the shelf, in both on- and off-policy settings. This
flexibility reduces the gap between risk-neutral control and risk-averse
control and is achieved by working on a novel augmented MDP directly. We
propose risk-averse TD3 as an example instantiating MVPI, which outperforms
vanilla TD3 and many previous risk-averse control methods in challenging Mujoco
robot simulation tasks under a risk-aware performance metric. This risk-averse
TD3 is the first to introduce deterministic policies and off-policy learning
into risk-averse reinforcement learning, both of which are key to the
performance boost we show in Mujoco domains.

本研究提出了一种基于平均方差策略迭代 (MVPI) 框架的风险规避控制方法，采用任意策略评估方法和风险中立控制方法，通过对一个新颖的扩展 MDP 直接进行处理，减少风险中立控制与风险规避控制之间的差距，并介绍了一种风险规避 TD3 方法作为 MVPI 的示例。该方法在 Mujoco 机器人仿真任务中优于传统 TD3 方法和其他风险规避控制方法。