Test-time adaptation (TTA) seeks to tackle potential distribution shifts
between training and test data by adapting a given model w.r.t. any test
sample. Although recent TTA has shown promising performance, we still face two
key challenges: 1) prior methods perform backpropagation for each test sample,
resulting in unbearable optimization costs to many applications; 2) while
existing TTA can significantly improve the test performance on
out-of-distribution data, they often suffer from severe performance degradation
on in-distribution data after TTA (known as forgetting). To this end, we have
proposed an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which
develops an active sample selection criterion to identify reliable and
non-redundant samples for test-time entropy minimization. To alleviate
forgetting, EATA introduces a Fisher regularizer estimated from test samples to
constrain important model parameters from drastic changes. However, in EATA,
the adopted entropy loss consistently assigns higher confidence to predictions
even for samples that are underlying uncertain, leading to overconfident
predictions. To tackle this, we further propose EATA with Calibration (EATA-C)
to separately exploit the reducible model uncertainty and the inherent data
uncertainty for calibrated TTA. Specifically, we measure the model uncertainty
by the divergence between predictions from the full network and its
sub-networks, on which we propose a divergence loss to encourage consistent
predictions instead of overconfident ones. To further recalibrate prediction
confidence, we utilize the disagreement among predicted labels as an indicator
of the data uncertainty, and then devise a min-max entropy regularizer to
selectively increase and decrease prediction confidence for different samples.
Experiments on image classification and semantic segmentation verify the
effectiveness of our methods.

测试时间适应 (Test-time adaptation，TTA) 在适应给定模型到任何测试样本，以解决训练与测试数据之间的潜在分布差异。然而，现有方法在每个测试样本上执行反向传播，导致对许多应用而言无法承受的优化成本，并且现有的 TTA 在无分布数据上可以显著提高测试性能，但在分布数据上经常遭受严重的性能退化（即遗忘）。为了解决这个问题，我们提出了一种高效的抗遗忘测试时间适应（Efficient Anti-Forgetting Test-Time Adaptation，EATA）方法，该方法通过开发主动样本选择准则来识别可靠且非冗余的测试样本，以实现测试时间熵的最小化。为了减轻遗忘，EATA 引入了一种从测试样本估计的 Fisher 正则化约束重要模型参数以防止剧烈变化的方法。然而，在 EATA 中，采用的熵损失通常将更高的置信度分配给不确定的样本，导致过度自信的预测。为了解决这个问题，我们进一步提出了用校准（Calibration）的 EATA（EATA-C）来分别利用可减小的模型不确定性和固有数据不确定性进行校准的 TTA。具体来说，我们通过完整网络和其子网络之间的预测差异来衡量模型不确定性，为此我们提出了一种差异损失来鼓励一致的预测而不是过度自信的预测。为了进一步重新校准预测置信度，我们利用预测标签之间的差异作为数据不确定性的指标，然后设计了一种最小最大熵正则化器来有选择地增加和减小不同样本的预测置信度。图像分类和语义分割的实验验证了我们方法的有效性。