We study offline reinforcement learning (RL) in partially observable Markov
decision processes. In particular, we aim to learn an optimal policy from a
dataset collected by a behavior policy which possibly depends on the latent
state. Such a dataset is confounded in the sense that the latent state
simultaneously affects the action and the observation, which is prohibitive for
existing offline RL algorithms. To this end, we propose the \underline{P}roxy
variable \underline{P}essimistic \underline{P}olicy \underline{O}ptimization
(\texttt{P3O}) algorithm, which addresses the confounding bias and the
distributional shift between the optimal and behavior policies in the context
of general function approximation. At the core of \texttt{P3O} is a coupled
sequence of pessimistic confidence regions constructed via proximal causal
inference, which is formulated as minimax estimation. Under a partial coverage
assumption on the confounded dataset, we prove that \texttt{P3O} achieves a
$n^{-1/2}$-suboptimality, where $n$ is the number of trajectories in the
dataset. To our best knowledge, \texttt{P3O} is the first provably efficient
offline RL algorithm for POMDPs with a confounded dataset.

研究通过行为策略收集的数据集来学习优化策略的离线强化学习算法，并针对潜在状态的影响所产生的混淆偏差和最优策略与行为策略之间的分布转换问题，提出了代理变量悲观策略优化（P3O）算法。

面对混淆因素的悲观主义：部分可观察马尔可夫决策过程中可证明高效的离线强化学习

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes

Skepticism about the assumption of no unmeasured confounding, also known as
exchangeability, is often warranted in making causal inferences from
observational data; because exchangeability hinges on an investigator's ability
to accurately measure covariates that capture all potential sources of
confounding. In practice, the most one can hope for is that covariate
measurements are at best proxies of the true underlying confounding mechanism
operating in a given observational study. In this paper, we consider the
framework of proximal causal inference introduced by Miao et al. (2018);
Tchetgen Tchetgen et al. (2020), which while explicitly acknowledging covariate
measurements as imperfect proxies of confounding mechanisms, offers an
opportunity to learn about causal effects in settings where exchangeability on
the basis of measured covariates fails. We make a number of contributions to
proximal inference including (i) an alternative set of conditions for
nonparametric proximal identification of the average treatment effect; (ii)
general semiparametric theory for proximal estimation of the average treatment
effect including efficiency bounds for key semiparametric models of interest;
(iii) a characterization of proximal doubly robust and locally efficient
estimators of the average treatment effect. Moreover, we provide analogous
identification and efficiency results for the average treatment effect on the
treated. Our approach is illustrated via simulation studies and a data
application on evaluating the effectiveness of right heart catheterization in
the intensive care unit of critically ill patients.

本文介绍了一种基于测量协变量的局部因果推断框架，其中正式承认协变量测量是难以完美匹配真实混淆机制的，为在基于观测数据的因果推断中提供新的思路。作者提出了一套非参数近端推断的条件，建立了平均处理效应的半参数近端推断理论，以及多种相容性分析方案，并在模拟情境和实际数据集评估中予以展示。