Decision support systems for classification tasks are predominantly designed
to predict the value of the ground truth labels. However, since their
predictions are not perfect, these systems also need to make human experts
understand when and how to use these predictions to update their own
predictions. Unfortunately, this has been proven challenging. In this context,
it has been recently argued that an alternative type of decision support
systems may circumvent this challenge. Rather than providing a single label
prediction, these systems provide a set of label prediction values constructed
using a conformal predictor, namely a prediction set, and forcefully ask
experts to predict a label value from the prediction set. However, the design
and evaluation of these systems have so far relied on stylized expert models,
questioning their promise. In this paper, we revisit the design of this type of
systems from the perspective of online learning and develop a methodology that
does not require, nor assumes, an expert model. Our methodology leverages the
nested structure of the prediction sets provided by any conformal predictor and
a natural counterfactual monotonicity assumption on the experts' predictions
over the prediction sets to achieve an exponential improvement in regret in
comparison with vanilla bandit algorithms. We conduct a large-scale human
subject study ($n = 2{,}751$) to verify our counterfactual monotonicity
assumption and compare our methodology to several competitive baselines. The
results suggest that decision support systems that limit experts' level of
agency may be practical and may offer greater performance than those allowing
experts to always exercise their own agency.

本文提出了一种新型决策支持系统，通过嵌套结构的预测集和专家的设想单调性假设，实现比普通 bandit 算法更好的结果；大规模人类学科研究 ($n=2751$) 表明，这种限制专家代理水平的决策支持系统实用且可能提供比允许专家始终行使其代理权的系统更好的性能。

使用反事实预测集设计决策支持系统

Designing Decision Support Systems Using Counterfactual Prediction Sets

Pre-trained language models (PLMs) have been shown effective for zero-shot
(0shot) text classification. 0shot models based on natural language inference
(NLI) and next sentence prediction (NSP) employ cross-encoder architecture and
infer by making a forward pass through the model for each label-text pair
separately. This increases the computational cost to make inferences linearly
in the number of labels. In this work, we improve the efficiency of such
cross-encoder-based 0shot models by restricting the number of likely labels
using another fast base classifier-based conformal predictor (CP) calibrated on
samples labeled by the 0shot model. Since a CP generates prediction sets with
coverage guarantees, it reduces the number of target labels without excluding
the most probable label based on the 0shot model. We experiment with three
intent and two topic classification datasets. With a suitable CP for each
dataset, we reduce the average inference time for NLI- and NSP-based models by
25.6% and 22.2% respectively, without dropping performance below the predefined
error rate of 1%.

通过使用基于 conformal predictor 的快速基分类器限制潜在的标签数，我们成功地提高了基于交叉编码器的零样本模型的效率，实验结果显示我们可以在性能不降低的情况下，将自然语言推理和下一句话预测的模型推理时间分别降低了 25.6％和 22.2％。