Language models are pre-trained using large corpora of generic data like book
corpus, common crawl and Wikipedia, which is essential for the model to
understand the linguistic characteristics of the language. New studies suggest
using Domain Adaptive Pre-training (DAPT) and Task-Adaptive Pre-training (TAPT)
as an intermediate step before the final finetuning task. This step helps cover
the target domain vocabulary and improves the model performance on the
downstream task. In this work, we study the impact of training only the
embedding layer on the model's performance during TAPT and task-specific
finetuning. Based on our study, we propose a simple approach to make the
intermediate step of TAPT for BERT-based models more efficient by performing
selective pre-training of BERT layers. We show that training only the BERT
embedding layer during TAPT is sufficient to adapt to the vocabulary of the
target domain and achieve comparable performance. Our approach is
computationally efficient, with 78\% fewer parameters trained during TAPT. The
proposed embedding layer finetuning approach can also be an efficient domain
adaptation technique.

该文章研究了在 TAPT 过程中只训练 BERT 模型的嵌入层对模型性能的影响，并提出了一种简单的方法来使 BERT 模型在中间步骤更加高效，该方法是培训 BERT 模型的嵌入层，可以适应目标领域的词汇并达到可比较的性能。

文本分类任务自适应预训练的简单高效方法

Towards Simple and Efficient Task-Adaptive Pre-training for Text Classification

While Wav2Vec 2.0 has been proposed for speech recognition (ASR), it can also
be used for speech emotion recognition (SER); its performance can be
significantly improved using different fine-tuning strategies. Two baseline
methods, vanilla fine-tuning (V-FT) and task adaptive pretraining (TAPT) are
first presented. We show that V-FT is able to outperform state-of-the-art
models on the IEMOCAP dataset. TAPT, an existing NLP fine-tuning strategy,
further improves the performance on SER. We also introduce a novel fine-tuning
method termed P-TAPT, which modifies the TAPT objective to learn contextualized
emotion representations. Experiments show that P-TAPT performs better than
TAPT, especially under low-resource settings. Compared to prior works in this
literature, our top-line system achieved a 7.4\% absolute improvement in
unweighted accuracy (UA) over the state-of-the-art performance on IEMOCAP. Our
code is publicly available.

该研究介绍了使用不同的微调策略，将 Wav2Vec 2.0 应用在语音情感识别上，从而进一步提高其性能，最终成功达到了在 IEMOCAP 数据集上的最新状态下的表现，升高 7.4% 的绝对无权重准确率，同时提出了一种上下文化情感表示的微调方法 “P-TAPT”。