Off-policy evaluation (OPE) is to evaluate a target policy with data
generated by other policies. Most previous OPE methods focus on precisely
estimating the true performance of a policy. We observe that in many
applications, (1) the end goal of OPE is to compare two or multiple candidate
policies and choose a good one, which is a much simpler task than precisely
evaluating their true performance; and (2) there are usually multiple policies
that have been deployed to serve users in real-world systems and thus the true
performance of these policies can be known. Inspired by the two observations,
in this work, we study a new problem, supervised off-policy ranking (SOPR),
which aims to rank a set of target policies based on supervised learning by
leveraging off-policy data and policies with known performance. We propose a
method to solve SOPR, which learns a policy scoring model by minimizing a
ranking loss of the training policies rather than estimating the precise policy
performance. The scoring model in our method, a hierarchical Transformer based
model, maps a set of state-action pairs to a score, where the state of each
pair comes from the off-policy data and the action is taken by a target policy
on the state in an offline manner. Extensive experiments on public datasets
show that our method outperforms baseline methods in terms of rank correlation,
regret value, and stability. Our code is publicly available at GitHub.

本研究提出了一种监督式非政策排名方法，利用离线政策数据和已知性能的政策对一组目标政策进行排名，并通过一个基于层次 Transformer 的政策得分模型来实现。实验证明，这种方法在排名相关性、遗憾值和稳定性等指标上优于基准方法。