Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation
(LoRA) can effectively adapt large pre-trained foundation models to downstream
tasks using only a small fraction (0.1%-10%) of the original trainable weights.
An under-explored question of PEFT is in extending the pre-training phase
without supervised labels; that is, can we adapt a pre-trained foundation model
to a new domain via efficient self-supervised pre-training on this new domain?
In this work, we introduce ExPLoRA, a highly effective technique to improve
transfer learning of pre-trained vision transformers (ViTs) under domain
shifts. Initializing a ViT with pre-trained weights on large, natural-image
datasets such as from DinoV2 or MAE, ExPLoRA continues the unsupervised
pre-training objective on a new domain. In this extended pre-training phase,
ExPLoRA only unfreezes 1-2 pre-trained ViT blocks and all normalization layers,
and then tunes all other layers with LoRA. Finally, we fine-tune the resulting
model only with LoRA on this new domain for supervised learning. Our
experiments demonstrate state-of-the-art results on satellite imagery, even
outperforming fully pre-training and fine-tuning ViTs. Using the DinoV2
training objective, we demonstrate up to 7% improvement in linear probing top-1
accuracy on downstream tasks while using <10% of the number of parameters that
are used in prior fully-tuned state-of-the art approaches. Our ablation studies
confirm the efficacy of our approach over other baselines, including PEFT and
simply unfreezing more transformer blocks.

通过使用 ExPLoRA 技术，在对卫星图像进行迁移学习时，只使用很少的参数数量，在无监督的预训练阶段只解冻 1-2 个预训练的 ViT 块和所有归一化层，然后通过 LoRA 微调所有其他层，最后只通过 LoRA 在新域上对模型进行微调，获得了基于先前完全训练和微调的 ViTs 的最先进结果。