For the first time, we explore few-shot tuning of vision foundation models
for class-incremental learning. Unlike existing few-shot class incremental
learning (FSCIL) methods, which train an encoder on a base session to ensure
forward compatibility for future continual learning, foundation models are
generally trained on large unlabelled data without such considerations. This
renders prior methods from traditional FSCIL incompatible for FSCIL with the
foundation model. To this end, we propose Consistency-guided Asynchronous
Contrastive Tuning (CoACT), a new approach to continually tune foundation
models for new classes in few-shot settings. CoACT comprises three components:
(i) asynchronous contrastive tuning, which learns new classes by including LoRA
modules in the pre-trained encoder, while enforcing consistency between two
asynchronous encoders; (ii) controlled fine-tuning, which facilitates effective
tuning of a subset of the foundation model; and (iii) consistency-guided
incremental tuning, which enforces additional regularization during later
sessions to reduce forgetting of the learned classes. We perform an extensive
study on 16 diverse datasets and demonstrate the effectiveness of CoACT,
outperforming the best baseline method by 2.47% on average and with up to
12.52% on individual datasets. Additionally, CoACT shows reduced forgetting and
robustness in low-shot experiments. As an added bonus, CoACT shows up to 13.5%
improvement in standard FSCIL over the current SOTA on benchmark evaluations.
We make our code publicly available at
this https URL

针对视觉基础模型的少样本调优方面的类增量学习，我们提出了 CoACT 方法，通过异步对比调优、控制微调和一致性引导增量调优三个组成部分，有效提升了模型性能和鲁棒性。