Off-policy evaluation (OPE) in reinforcement learning is an important problem in settings where experimentation is limited, such as education and healthcare. But, in these very same settings, observed actions are often confounded by unobserved variables making OPE even more difficult. We study an OPE problem in an infinite-horizon, ergodic Markov decision process with unobserved confounders, where states and actions can act as proxies for the unobserved confounders. We show how, given only a latent variable model for states and actions, policy value can be identified from off-policy data. Our method involves two stages. In the first, we show how to use proxies to estimate stationary distribution ratios, extending recent work on breaking the curse of horizon to the confounded setting. In the second, we show optimal balancing can be combined with such learned ratios to obtain policy value while avoiding direct modeling of reward functions. We establish theoretical guarantees of consistency, and benchmark our method empirically.

针对强化学习领域中的Off-policy Evaluation问题，本研究提出了一种基于无法观测到的干扰变量的无限时域Markov决策过程方法，并通过使用代理估计稳态分布比率以及最优化平衡等阶段性的方法，从Off-policy数据中识别政策价值。

具有潜在混淆因素的无限时段强化学习中的离线评估