We consider the problem of off-policy evaluation---estimating the value of a target policy using data collected by another policy---under the contextual bandit model. We establish a minimax lower bound on the mean squared error (MSE), and show that it is matched up to constant factors by the inverse propensity scoring (IPS) estimator. Since in the multi-armed bandit problem the IPS is suboptimal (Li et. al, 2015), our result highlights the difficulty of the contextual setting with non-degenerate context distributions. We further consider improvements on this minimax MSE bound, given access to a reward model. We show that the existing doubly robust approach, which utilizes such a reward model, may continue to suffer from high variance even when the reward model is perfect. We propose a new estimator called SWITCH which more effectively uses the reward model and achieves a superior bias-variance tradeoff compared with prior work. We prove an upper bound on its MSE and demonstrate its benefits empirically on a diverse collection of datasets, often seeing orders of magnitude improvements over a number of baselines.

本研究旨在研究在没有相应奖励模型的情况下，通过使用由另一政策收集的数据来估算目标政策的价值的，上下文Bandit模型下的离线政策评估问题。所提出的 SWITCH estimator 可以使用现有的（不一定一致的）奖励模型来实现更好的偏差-方差平衡，从而在多个数据集上实现更好的表现。

基于上下文的赌博机中最优的自适应离线评估