Continual Test-Time Adaptation (CTTA), which aims to adapt the pre-trained
model to ever-evolving target domains, emerges as an important task for vision
models. As current vision models appear to be heavily biased towards texture,
continuously adapting the model from one domain distribution to another can
result in serious catastrophic forgetting. Drawing inspiration from the human
visual system's adeptness at processing both shape and texture according to the
famous Trichromatic Theory, we explore the integration of a
Mixture-of-Activation-Sparsity-Experts (MoASE) as an adapter for the CTTA task.
Given the distinct reaction of neurons with low/high activation to
domain-specific/agnostic features, MoASE decomposes the neural activation into
high-activation and low-activation components with a non-differentiable Spatial
Differentiate Dropout (SDD). Based on the decomposition, we devise a multi-gate
structure comprising a Domain-Aware Gate (DAG) that utilizes domain information
to adaptive combine experts that process the post-SDD sparse activations of
different strengths, and the Activation Sparsity Gate (ASG) that adaptively
assigned feature selection threshold of the SDD for different experts for more
precise feature decomposition. Finally, we introduce a Homeostatic-Proximal
(HP) loss to bypass the error accumulation problem when continuously adapting
the model. Extensive experiments on four prominent benchmarks substantiate that
our methodology achieves state-of-the-art performance in both classification
and segmentation CTTA tasks. Our code is now available at
this https URL

利用 Mixture-of-Activation-Sparsity-Experts 将神经激活分解为高激活和低激活成分，结合领域信息和自适应特征选择阈值，通过引入 Homeostatic-Proximal 损失解决了模型连续适应中的问题，从而在分类和分割任务中达到了最先进的性能。