Off-policy evaluation (OPE) is the task of estimating the expected reward of
a given policy based on offline data previously collected under different
policies. Therefore, OPE is a key step in applying reinforcement learning to
real-world domains such as medical treatment, where interactive data collection
is expensive or even unsafe. As the observed data tends to be noisy and
limited, it is essential to provide rigorous uncertainty quantification, not
just a point estimation, when applying OPE to make high stakes decisions. This
work considers the problem of constructing non-asymptotic confidence intervals
in infinite-horizon off-policy evaluation, which remains a challenging open
question. We develop a practical algorithm through a primal-dual
optimization-based approach, which leverages the kernel Bellman loss (KBL) of
Feng et al.(2019) and a new martingale concentration inequality of KBL
applicable to time-dependent data with unknown mixing conditions. Our algorithm
makes minimum assumptions on the data and the function class of the Q-function,
and works for the behavior-agnostic settings where the data is collected under
a mix of arbitrary unknown behavior policies. We present empirical results that
clearly demonstrate the advantages of our approach over existing methods.

本文提出一种基于原始 - 对偶优化的算法，用于构建非渐进置信区间，该算法利用了 Feng 等人（2019 年）的核贝尔曼损失（KBL）和适用于具有未知混合条件的时间依赖数据的新的鞅集中不等式，明确展示了算法的优势。