Test-time adaptation (TTA) is a technique aimed at enhancing the
generalization performance of models by leveraging unlabeled samples solely
during prediction. Given the need for robustness in neural network systems when
faced with distribution shifts, numerous TTA methods have recently been
proposed. However, evaluating these methods is often done under different
settings, such as varying distribution shifts, backbones, and designing
scenarios, leading to a lack of consistent and fair benchmarks to validate
their effectiveness. To address this issue, we present a benchmark that
systematically evaluates 13 prominent TTA methods and their variants on five
widely used image classification datasets: CIFAR-10-C, CIFAR-100-C, ImageNet-C,
DomainNet, and Office-Home. These methods encompass a wide range of adaptation
scenarios (e.g. online adaptation v.s. offline adaptation, instance adaptation
v.s. batch adaptation v.s. domain adaptation). Furthermore, we explore the
compatibility of different TTA methods with diverse network backbones. To
implement this benchmark, we have developed a unified framework in PyTorch,
which allows for consistent evaluation and comparison of the TTA methods across
the different datasets and network architectures. By establishing this
benchmark, we aim to provide researchers and practitioners with a reliable
means of assessing and comparing the effectiveness of TTA methods in improving
model robustness and generalization performance. Our code is available at
this https URL

通过建立一个基准测试来评估和比较测试时间调整方法在提高模型稳健性和泛化性能方面的有效性，我们提供了一个可靠的评估方式，并探索了不同测试时间调整方法与不同网络骨干的兼容性。