We consider a robust reinforcement learning problem, where a learning agent
learns from a simulated training environment. To account for the model
mis-specification between this training environment and the real environment
due to lack of data, we adopt a formulation of Bayesian risk MDP (BRMDP) with
infinite horizon, which uses Bayesian posterior to estimate the transition
model and impose a risk functional to account for the model uncertainty.
Observations from the real environment that is out of the agent's control
arrive periodically and are utilized by the agent to update the Bayesian
posterior to reduce model uncertainty. We theoretically demonstrate that BRMDP
balances the trade-off between robustness and conservativeness, and we further
develop a multi-stage Bayesian risk-averse Q-learning algorithm to solve BRMDP
with streaming observations from real environment. The proposed algorithm
learns a risk-averse yet optimal policy that depends on the availability of
real-world observations. We provide a theoretical guarantee of strong
convergence for the proposed algorithm.

本研究提出一种用于强化学习中的贝叶斯风险 MDP 问题的解决方法，旨在平衡鲁棒性和保守性。研究通过采用后验贝叶斯来估计转移模型，结合风险功能来控制模型的不确定性，并开发了多级贝叶斯风险敏感的 Q-learning 算法来解决问题。