Transformer-based language models have achieved significant success in
various domains. However, the data-intensive nature of the transformer
architecture requires much labeled data, which is challenging in low-resource
scenarios (i.e., few-shot learning (FSL)). The main challenge of FSL is the
difficulty of training robust models on small amounts of samples, which
frequently leads to overfitting. Here we present Mask-BERT, a simple and
modular framework to help BERT-based architectures tackle FSL. The proposed
approach fundamentally differs from existing FSL strategies such as prompt
tuning and meta-learning. The core idea is to selectively apply masks on text
inputs and filter out irrelevant information, which guides the model to focus
on discriminative tokens that influence prediction results. In addition, to
make the text representations from different categories more separable and the
text representations from the same category more compact, we introduce a
contrastive learning loss function. Experimental results on public-domain
benchmark datasets demonstrate the effectiveness of Mask-BERT.

文章提出了一种名为 Mask-BERT 的模型，通过选择性地在文本输入上应用掩码来过滤掉不相关的信息，从而引导模型专注于有影响的标记，以帮助 BERT-based 架构解决低资源场景下（即 few-shot learning）训练稳健模型的困境，并引入对比学习损失函数，以使不同类别的文本表示更易区分，同一类别的文本表示更加紧凑。公共基准数据集上的实验结果证明了 Mask-BERT 模型的有效性。