Offline reinforcement learning is important in many settings with available
observational data but the inability to deploy new policies online due to
safety, cost, and other concerns. Many recent advances in causal inference and
machine learning target estimation of causal contrast functions such as CATE,
which is sufficient for optimizing decisions and can adapt to potentially
smoother structure. We develop a dynamic generalization of the R-learner (Nie
and Wager 2021, Lewis and Syrgkanis 2021) for estimating and optimizing the
difference of $Q^\pi$-functions, $Q^\pi(s,1)-Q^\pi(s,0)$ (which can be used to
optimize multiple-valued actions). We leverage orthogonal estimation to improve
convergence rates in the presence of slower nuisance estimation rates and prove
consistency of policy optimization under a margin condition. The method can
leverage black-box nuisance estimators of the $Q$-function and behavior policy
to target estimation of a more structured $Q$-function contrast.

离线强化学习是许多具有可用观测数据但由于安全、成本和其他方面的考虑而无法在线部署新策略的设置中的重要问题。本文提出了 R-learner 的动态推广，用于估计和优化 Q 函数之间的差异，方法可以利用黑盒辅助估计器和行为策略来估计更结构化的 Q 函数对比。