With the increasing deployment of machine learning models in many
socially-sensitive tasks, there is a growing demand for reliable and
trustworthy predictions. One way to accomplish these requirements is to allow a
model to abstain from making a prediction when there is a high risk of making
an error. This requires adding a selection mechanism to the model, which
selects those examples for which the model will provide a prediction. The
selective classification framework aims to design a mechanism that balances the
fraction of rejected predictions (i.e., the proportion of examples for which
the model does not make a prediction) versus the improvement in predictive
performance on the selected predictions. Multiple selective classification
frameworks exist, most of which rely on deep neural network architectures.
However, the empirical evaluation of the existing approaches is still limited
to partial comparisons among methods and settings, providing practitioners with
little insight into their relative merits. We fill this gap by benchmarking 18
baselines on a diverse set of 44 datasets that includes both image and tabular
data. Moreover, there is a mix of binary and multiclass tasks. We evaluate
these approaches using several criteria, including selective error rate,
empirical coverage, distribution of rejected instance's classes, and
performance on out-of-distribution instances. The results indicate that there
is not a single clear winner among the surveyed baselines, and the best method
depends on the users' objectives.

通过对包含图像和表格数据的多样化数据集进行实证评估，该研究对 18 种基线模型进行基准测试，比较了它们在选择错误率、实证覆盖率、被拒绝实例类别分布以及在分布外实例上的性能等多个方面的表现，结果显示没有一个明确的最佳方法，最好的方法取决于用户的目标。

深度神经网络选择性分类基准测试

Deep Neural Network Benchmarks for Selective Classification

In-context learning (ICL) is a new paradigm for natural language processing
that utilizes Generative Pre-trained Transformer (GPT)-like models. This
approach uses prompts that include in-context demonstrations to generate the
corresponding output for a new query input. However, applying ICL in real cases
does not scale with the number of samples, and lacks robustness to different
prompt templates and demonstration permutations. In this paper, we first show
that GPT-like models using ICL result in unreliable predictions based on a new
metric based on Shannon entropy. Then, to solve this problem, we propose a new
technique called the Linear Probe Calibration (LinC), a method that calibrates
the model's output probabilities, resulting in reliable predictions and
improved performance, while requiring only minimal additional samples (as few
as five labeled data samples). LinC significantly enhances the ICL test
performance of GPT models on various benchmark datasets, with an average
improvement of up to 21%, and up to a 50% improvement in some cases, and
significantly boosts the performance of PEFT methods, especially in the low
resource regime. Moreover, LinC achieves lower expected calibration error, and
is highly robust to varying label proportions, prompt templates, and
demonstration permutations. Our code is available at
https://github.com/mominabbass/LinC.

通过使用新的线性校准技术（LinC），在仅需少量数据样本（如仅五个标记数据样本）的情况下，对于以 in-context learning（ICL）为基础的 GPT 模型进行概率输出校准，从而达到可靠的预测和改善性能，并显著提升 GPT 模型在各种基准数据集上的测试性能，平均改善率高达 21％，某些情况下高达 50％，尤其在资源有限的情况下提升 PEFT 方法的性能，同时具有较低的期望校准误差，并对不同的标签比例、提示模板和演示排列都具有高度的鲁棒性。