Given the inevitability of domain shifts during inference in real-world
applications, test-time adaptation (TTA) is essential for model adaptation
after deployment. However, the real-world scenario of continuously changing
target distributions presents challenges including catastrophic forgetting and
error accumulation. Existing TTA methods for non-stationary domain shifts,
while effective, incur excessive computational load, making them impractical
for on-device settings. In this paper, we introduce a layer-wise auto-weighting
algorithm for continual and gradual TTA that autonomously identifies layers for
preservation or concentrated adaptation. By leveraging the Fisher Information
Matrix (FIM), we first design the learning weight to selectively focus on
layers associated with log-likelihood changes while preserving unrelated ones.
Then, we further propose an exponential min-max scaler to make certain layers
nearly frozen while mitigating outliers. This minimizes forgetting and error
accumulation, leading to efficient adaptation to non-stationary target
distribution. Experiments on CIFAR-10C, CIFAR-100C, and ImageNet-C show our
method outperforms conventional continual and gradual TTA approaches while
significantly reducing computational load, highlighting the importance of
FIM-based learning weight in adapting to continuously or gradually shifting
target domains.

采用 Fisher 信息矩阵为基础的算法，通过引入逐层自动加权的方法，实现在不断变化的目标分布环境中的模型自适应，以减少遗忘和错误积累，并大大降低计算负载。