Many selection processes such as finding patients qualifying for a medical trial or retrieval pipelines in search engines consist of multiple stages, where an initial screening stage focuses the resources on shortlisting the most promising candidates. In this paper, we investigate what guarantees a screening classifier can provide, independently of whether it is constructed manually or trained. We find that current solutions do not enjoy distribution-free theoretical guarantees -- we show that, in general, even for a perfectly calibrated classifier, there always exist specific pools of candidates for which its shortlist is suboptimal. Then, we develop a distribution-free screening algorithm -- called Calibrated Subset Selection (CSS) -- that, given any classifier and some amount of calibration data, finds near-optimal shortlists of candidates that contain a desired number of qualified candidates in expectation. Moreover, we show that a variant of our algorithm that calibrates a given classifier multiple times across specific groups can create shortlists with provable diversity guarantees. Experiments on US Census survey data validate our theoretical results and show that the shortlists provided by our algorithm are superior to those provided by several competitive baselines.

本文考察了筛选分类器能否提供独立于手动或训练方式构建的理论保证。我们发现目前的解决方案没有分布无关的理论保证，开发了一种分布无关的筛选算法-- Calibrated Subset Selection（CSS），可以在任何分类器和某些校准数据的情况下找到包含所需合格候选人数量的最佳候选人短列表。实验验证了我们算法提供的清单优于几种竞争基线。

通过校准子集选择改善筛选流程