Estimating the out-of-distribution performance in regimes where labels are
scarce is critical to safely deploy foundation models. Recently, it was shown
that ensembles of neural networks observe the phenomena
``agreement-on-the-line'', which can be leveraged to reliably predict OOD
performance without labels. However, in contrast to classical neural networks
that are trained on in-distribution data from scratch for numerous epochs,
foundation models undergo minimal finetuning from heavily pretrained weights,
which may reduce the ensemble diversity needed to observe
agreement-on-the-line. In our work, we demonstrate that when lightly finetuning
multiple runs from a $\textit{single}$ foundation model, the choice of
randomness during training (linear head initialization, data ordering, and data
subsetting) can lead to drastically different levels of agreement-on-the-line
in the resulting ensemble. Surprisingly, only random head initialization is
able to reliably induce agreement-on-the-line in finetuned foundation models
across vision and language benchmarks. Second, we demonstrate that ensembles of
$\textit{multiple}$ foundation models pretrained on different datasets but
finetuned on the same task can also show agreement-on-the-line. In total, by
careful construction of a diverse ensemble, we can utilize
agreement-on-the-line-based methods to predict the OOD performance of
foundation models with high precision.

通过构建多样性的集合，我们可以利用基于线上一致性的方法以高精度预测基础模型的 OOD 性能。

通过线上协议预测基础模型的性能

Predicting the Performance of Foundation Models via  Agreement-on-the-Line

Ensembles of separate neural networks (NNs) have shown superior accuracy and
confidence calibration over single NN across tasks. Recent methods compress
ensembles within a single network via early exits or multi-input multi-output
frameworks. However, the landscape of these methods is fragmented thus far,
making it difficult to choose the right approach for a given task. Furthermore,
the algorithmic performance of these methods is behind the ensemble of separate
NNs and requires extensive architecture tuning. We propose a novel methodology
unifying these approaches into a Single Architecture Ensemble (SAE). Our method
learns the optimal number and depth of exits per ensemble input in a single NN.
This enables the SAE framework to flexibly tailor its configuration for a given
architecture or application. We evaluate SAEs on image classification and
regression across various network architecture types and sizes. We demonstrate
competitive accuracy or confidence calibration to baselines while reducing the
compute operations or parameter count by up to $1.5{\sim}3.7\times$.

通过单个神经网络中的早期退出和多输入多输出框架，将分散的神经网络集成为一个单一的体系结构集合，提高准确度和置信度校准，并减少运算和参数量。