Prior work on safe Reinforcement Learning (RL) has studied risk-aversion to randomness in dynamics (aleatory) and to model uncertainty (epistemic) in isolation. We propose and analyze a new framework to jointly model the risk associated with epistemic and aleatory uncertainties in finite-horizon and discounted infinite-horizon MDPs. We call this framework that combines Risk-Averse and Soft-Robust methods RASR. We show that when the risk-aversion is defined using either EVaR or the entropic risk, the optimal policy in RASR can be computed efficiently using a new dynamic program formulation with a time-dependent risk level. As a result, the optimal risk-averse policies are deterministic but time-dependent, even in the infinite-horizon discounted setting. We also show that particular RASR objectives reduce to risk-averse RL with mean posterior transition probabilities. Our empirical results show that our new algorithms consistently mitigate uncertainty as measured by EVaR and other standard risk measures.

提出一种新的框架，称为Risk-Averse and Soft-Robust methods (RASR)，用于联合建模有限和无限时域rMarkov决策过程（MDP）中的认识不确定性和aleatory不确定性与刻画风险。在RASR中，当风险是使用EVaR或熵风险定义时，可以使用时间依赖的风险级别使用动态程序形式化地计算最优风险规避策略。我们还发现特定的RASR目标简化为具有后验平均转移概率的风险规避RL。实验结果表明，我们新的算法通过EVaR和其他标准风险度量一致地缓解不确定性。

RASR: 带有EVaR和熵风险的风险规避型软鲁棒MDPs