Learning effective contextual-bandit policies from past actions of a deployed
system is highly desirable in many settings (e.g. voice assistants,
recommendation, search), since it enables the reuse of large amounts of log
data. State-of-the-art methods for such off-policy learning, however, are based
on inverse propensity score (IPS) weighting. A key theoretical requirement of
IPS weighting is that the policy that logged the data has "full support", which
typically translates into requiring non-zero probability for any action in any
context. Unfortunately, many real-world systems produce support deficient data,
especially when the action space is large, and we show how existing methods can
fail catastrophically. To overcome this gap between theory and applications, we
identify three approaches that provide various guarantees for IPS-based
learning despite the inherent limitations of support-deficient data:
restricting the action space, reward extrapolation, and restricting the policy
space. We systematically analyze the statistical and computational properties
of these three approaches, and we empirically evaluate their effectiveness. In
addition to providing the first systematic analysis of support-deficiency in
contextual-bandit learning, we conclude with recommendations that provide
practical guidance.

本文针对每日实际情况中，由于行动空间的限制，不能在所有情境下为每一个行动确定非零概率而产生的不足支持数据的问题，提出在限制行动空间、奖励外推和限制策略空间三个方法领域内提出的 IPS 补丁方法，分析了三种方法的统计性和计算性属性，并进行实证研究，提供了 I PS 方法学习时的实践指南。