Test-time adaptation (TTA) has emerged as a viable solution to adapt
pre-trained models to domain shifts using unlabeled test data. However, TTA
faces challenges of adaptation failures due to its reliance on blind adaptation
to unknown test samples in dynamic scenarios. Traditional methods for
out-of-distribution performance estimation are limited by unrealistic
assumptions in the TTA context, such as requiring labeled data or re-training
models. To address this issue, we propose AETTA, a label-free accuracy
estimation algorithm for TTA. We propose the prediction disagreement as the
accuracy estimate, calculated by comparing the target model prediction with
dropout inferences. We then improve the prediction disagreement to extend the
applicability of AETTA under adaptation failures. Our extensive evaluation with
four baselines and six TTA methods demonstrates that AETTA shows an average of
19.8%p more accurate estimation compared with the baselines. We further
demonstrate the effectiveness of accuracy estimation with a model recovery case
study, showcasing the practicality of our model recovery based on accuracy
estimation. The source code is available at this https URL

我们提出了一种适用于测试时间自适应（TTA）的无标签准确度估计算法 AETTA，通过与辍学推断比较目标模型预测来改进预测不一致性以扩展 AETTA 在适应失败下的适用性。我们进行了广泛的评估，并使用四个基线模型和六种 TTA 方法进行实验证明了 AETTA 相比基线方法具有平均高出 19.8％的准确度估计，此外通过模型恢复案例研究进一步证明了准确度估计的有效性。

AETTA: 测试时适应的无标签精度估计

AETTA: Label-Free Accuracy Estimation for Test-Time Adaptation

Recent instruction-finetuned large language models (LMs) have achieved
notable performances in various tasks, such as question-answering (QA).
However, despite their ability to memorize a vast amount of general knowledge
across diverse tasks, they might be suboptimal on specific tasks due to their
limited capacity to transfer and adapt knowledge to target tasks. Moreover,
further finetuning LMs with labeled datasets is often infeasible due to their
absence, but it is also questionable if we can transfer smaller LMs having
limited knowledge only with unlabeled test data. In this work, we show and
investigate the capabilities of smaller self-adaptive LMs, only with unlabeled
test data. In particular, we first stochastically generate multiple answers,
and then ensemble them while filtering out low-quality samples to mitigate
noise from inaccurate labels. Our proposed self-adaption strategy demonstrates
significant performance improvements on benchmark QA datasets with higher
robustness across diverse prompts, enabling LMs to stay stable. Code is
available at: this https URL

通过使用未标记的测试数据，我们展示并研究了仅凭借未标记的测试数据的自适应语言模型的能力。我们首先随机生成多个答案，然后在过滤掉低质量样本的同时将它们集成在一起，以减轻不准确标签引入的噪声。我们提出的自适应策略在基准问答数据集上表现出了显著的性能改进，对于多样的提示更具鲁棒性，使语言模型保持稳定。

测试时间自适应的小型语言模型在问答中的应用

Test-Time Self-Adaptive Small Language Models for Question Answering

To calculate the model accuracy on a computer vision task, e.g., object
recognition, we usually require a test set composing of test samples and their
ground truth labels. Whilst standard usage cases satisfy this requirement, many
real-world scenarios involve unlabeled test data, rendering common model
evaluation methods infeasible. We investigate this important and under-explored
problem, Automatic model Evaluation (AutoEval). Specifically, given a labeled
training set and a classifier, we aim to estimate the classification accuracy
on unlabeled test datasets. We construct a meta-dataset: a dataset comprised of
datasets generated from the original images via various transformations such as
rotation, background substitution, foreground scaling, etc. As the
classification accuracy of the model on each sample (dataset) is known from the
original dataset labels, our task can be solved via regression. Using the
feature statistics to represent the distribution of a sample dataset, we can
train regression models (e.g., a regression neural network) to predict model
performance. Using synthetic meta-dataset and real-world datasets in training
and testing, respectively, we report a reasonable and promising prediction of
the model accuracy. We also provide insights into the application scope,
limitation, and potential future direction of AutoEval.

本文讨论了在存在未标记测试数据时自动模型评估的问题，并提出了一种利用采样数据和回归模型估计模型精度的方法，称之为 AutoEval。