Test-time adaptation has proven effective in adapting a given trained model
to unseen test samples with potential distribution shifts. However, in
real-world scenarios, models are usually deployed on resource-limited devices,
e.g., FPGAs, and are often quantized and hard-coded with non-modifiable
parameters for acceleration. In light of this, existing methods are often
infeasible since they heavily depend on computation-intensive backpropagation
for model updating that may be not supported. To address this, we propose a
test-time Forward-Only Adaptation (FOA) method. In FOA, we seek to solely learn
a newly added prompt (as model's input) via a derivative-free covariance matrix
adaptation evolution strategy. To make this strategy work stably under our
online unsupervised setting, we devise a novel fitness function by measuring
test-training statistic discrepancy and model prediction entropy. Moreover, we
design an activation shifting scheme that directly tunes the model activations
for shifted test samples, making them align with the source training domain,
thereby further enhancing adaptation performance. Without using any
backpropagation and altering model weights, FOA runs on quantized 8-bit ViT
outperforms gradient-based TENT on full-precision 32-bit ViT, while achieving
an up to 24-fold memory reduction on ImageNet-C. The source code will be
released.

测试时间自适应是一种有效的方法，用于将经过训练的模型适配于具有潜在分布偏移的未见测试样本。为了应对资源有限的设备上的实际场景，我们提出了一种新的前向适应方法，它通过衍生自由的协方差矩阵适应进化策略来学习新添加的提示（作为模型输入），并设计了一种激活迁移方案，通过调整模型激活使其与源域对齐，进一步增强适应性能。