We study reinforcement learning under model misspecification, where we do not have access to the true environment but only to a reasonably close approximation to it. We address this problem by extending the framework of robust MDPs to the model-free Reinforcement Learning setting, where we do not have access to the model parameters, but can only sample states from it. We define robust versions of Q-learning, SARSA, and TD-learning and prove convergence to an approximately optimal robust policy and approximate value function respectively. We scale up the robust algorithms to large MDPs via function approximation and prove convergence under two different settings. We prove convergence of robust approximate policy iteration and robust approximate value iteration for linear architectures (under mild assumptions). We also define a robust loss function, the mean squared robust projected Bellman error and give stochastic gradient descent algorithms that are guaranteed to converge to a local minimum.

论文研究了缺失真实环境信息的强化学习问题，将鲁棒MDP框架扩展到无模型参数条件下的RL设置中，提出了三个具有鲁棒性的Q-learning、SARSA和TD-learning算法，并通过函数逼近扩展到大规模MDPs，证明了其收敛性，并给出了保证局部最小的随机梯度下降算法。

模型不匹配下的强化学习