Test-time adaptation (TTA) methods aim to improve robustness to distribution
shifts by adapting models using unlabeled data from the shifted test
distribution. However, there remain unresolved challenges that undermine the
reliability of TTA, which include difficulties in evaluating TTA performance,
miscalibration after TTA, and unreliable hyperparameter tuning for adaptation.
In this work, we make a notable and surprising observation that TTAed models
strongly show the agreement-on-the-line phenomenon (Baek et al., 2022) across a
wide range of distribution shifts. We find such linear trends occur
consistently in a wide range of models adapted with various hyperparameters,
and persist in distributions where the phenomenon fails to hold in vanilla
models (i.e., before adaptation). We leverage these observations to make TTA
methods more reliable in three perspectives: (i) estimating OOD accuracy
(without labeled data) to determine when TTA helps and when it hurts, (ii)
calibrating TTAed models without label information, and (iii) reliably
determining hyperparameters for TTA without any labeled validation data.
Through extensive experiments, we demonstrate that various TTA methods can be
precisely evaluated, both in terms of their improvements and degradations.
Moreover, our proposed methods on unsupervised calibration and hyperparameters
tuning for TTA achieve results close to the ones assuming access to
ground-truth labels, in terms of both OOD accuracy and calibration error.

通过对测试数据的自适应方法（TTA）进行评估和校准，解决了分布转移时的困难，同时提出了关于 OOD 准确性估计和超参数调整的可靠方法。