Performing policy evaluation in education, healthcare and online commerce can be challenging, because it can require waiting substantial amounts of time to observe outcomes over the desired horizon of interest. While offline evaluation methods can be used to estimate the performance of a new decision policy from historical data in some cases, such methods struggle when the new policy involves novel actions or is being run in a new decision process with potentially different dynamics. Here we consider how to estimate the full-horizon value of a new decision policy using only short-horizon data from the new policy, and historical full-horizon data from a different behavior policy. We introduce two new estimators for this setting, including a doubly robust estimator, and provide formal analysis of their properties. Our empirical results on two realistic simulators, of HIV treatment and sepsis treatment, show that our methods can often provide informative estimates of a new decision policy ten times faster than waiting for the full horizon, highlighting that it may be possible to quickly identify if a new decision policy, involving new actions, is better or worse than existing past policies.

本研究解决了在教育、医疗和在线商业中政策评估面临的挑战，特别是需要长时间观察结果的问题。通过引入两种新的评估估计器，研究展示了如何仅利用短期数据和历史全周期数据，快速有效地估计新决策政策的全周期价值，从而在十倍速上实现比传统方法更快的政策评估。

使用软替代品预测长期顺序政策价值