Few-Shot Class-Incremental Learning presents an extension of the Class
Incremental Learning problem where a model is faced with the problem of data
scarcity while addressing the catastrophic forgetting problem. This problem
remains an open problem because all recent works are built upon the
convolutional neural networks performing sub-optimally compared to the
transformer approaches. Our paper presents Robust Transformer Approach built
upon the Compact Convolution Transformer. The issue of overfitting due to few
samples is overcome with the notion of the stochastic classifier, where the
classifier's weights are sampled from a distribution with mean and variance
vectors, thus increasing the likelihood of correct classifications, and the
batch-norm layer to stabilize the training process. The issue of CF is dealt
with the idea of delta parameters, small task-specific trainable parameters
while keeping the backbone networks frozen. A non-parametric approach is
developed to infer the delta parameters for the model's predictions. The
prototype rectification approach is applied to avoid biased prototype
calculations due to the issue of data scarcity. The advantage of ROBUSTA is
demonstrated through a series of experiments in the benchmark problems where it
is capable of outperforming prior arts with big margins without any data
augmentation protocols.

在本研究中，提出了一种鲁棒的变换器方法（ROBUSTA）来解决少样本分类增量学习问题，其中利用随机分类器、稳定训练过程的批归一化层以及小任务特定可训练参数（delta parameters）来克服过拟合和遗忘问题，并通过非参数方法推断模型预测的 delta 参数。通过一系列实验验证了 ROBUSTA 在基准问题上的优势，且无需数据增强方法即可显著超越先前的方法。

鲁棒变换器方法的小样本增量学习

Few-Shot Class Incremental Learning via Robust Transformer Approach

In this paper, we uncover that Language Models (LMs), either encoder- or
decoder-based, can obtain new capabilities by assimilating the parameters of
homologous models without retraining or GPUs. Typically, new abilities of LMs
can be imparted by Supervised Fine-Tuning (SFT), reflected in the disparity
between fine-tuned and pre-trained parameters (i.e., delta parameters). We
initially observe that by introducing a novel operation called DARE (Drop And
REscale), most delta parameters can be directly set to zeros without affecting
the capabilities of SFT LMs and larger models can tolerate a higher proportion
of discarded parameters. Based on this observation, we further sparsify delta
parameters of multiple SFT homologous models with DARE and subsequently merge
them into a single model by parameter averaging. We conduct experiments on
eight datasets from the GLUE benchmark with BERT and RoBERTa. We also merge
WizardLM, WizardMath, and Code Alpaca based on Llama 2. Experimental results
show that: (1) The delta parameter value ranges for SFT models are typically
small, often within 0.005, and DARE can eliminate 99% of them effortlessly.
However, once the models are continuously pre-trained, the value ranges can
grow to around 0.03, making DARE impractical. We have also tried to remove
fine-tuned instead of delta parameters and find that a 10% reduction can lead
to drastically decreased performance (even to 0). This highlights that SFT
merely stimulates the abilities via delta parameters rather than injecting new
abilities into LMs; (2) DARE can merge multiple task-specific LMs into one LM
with diverse abilities. For instance, the merger of WizardLM and WizardMath
improves the GSM8K zero-shot accuracy of WizardLM from 2.2 to 66.3, retaining
its instruction-following ability while surpassing WizardMath's original 64.2
performance. Codes are available at this https URL

我们揭示了语言模型（LMs）可以通过吸收同类模型的参数而无需重新训练或使用图形处理器来获得新的能力。我们引入了一种名为 DARE（Drop And Rescale）的新操作，将绝大多数增量参数直接设为零，并可以将多个特定任务的 LM 合并为一个具有多样能力的 LM。