Test Time Adaptation (TTA) addresses the problem of distribution shift by
enabling pretrained models to learn new features on an unseen domain at test
time. However, it poses a significant challenge to maintain a balance between
learning new features and retaining useful pretrained features. In this paper,
we propose Layerwise EArly STopping (LEAST) for TTA to address this problem.
The key idea is to stop adapting individual layers during TTA if the features
being learned do not appear beneficial for the new domain. For that purpose, we
propose using a novel gradient-based metric to measure the relevance of the
current learnt features to the new domain without the need for supervised
labels. More specifically, we propose to use this metric to determine
dynamically when to stop updating each layer during TTA. This enables a more
balanced adaptation, restricted to layers benefiting from it, and only for a
certain number of steps. Such an approach also has the added effect of limiting
the forgetting of pretrained features useful for dealing with new domains.
Through extensive experiments, we demonstrate that Layerwise Early Stopping
improves the performance of existing TTA approaches across multiple datasets,
domain shifts, model architectures, and TTA losses.

通过引入 Layerwise EArly STopping (LEAST) 和使用一种新的基于梯度的度量来衡量当前学习特征对新域的相关性，我们提出了一种解决 Test Time Adaptation (TTA) 中保持学习新特征和保留预训练特征之间平衡的方法。这种方法可以在多个数据集、域转换、模型架构和 TTA 损失上提高现有 TTA 方法的性能。