In the arena of language model fine-tuning, the traditional approaches, such
as Domain-Adaptive Pretraining (DAPT) and Task-Adaptive Pretraining (TAPT),
although effective, but computational intensive. This research introduces a
novel adaptation method utilizing the UniPELT framework as a base and added a
PromptTuning Layer, which significantly reduces the number of trainable
parameters while maintaining competitive performance across various benchmarks.
Our method employs adapters, which enable efficient transfer of pretrained
models to new tasks with minimal retraining of the base model parameters. We
evaluate our approach using three diverse datasets: the GLUE benchmark, a
domain-specific dataset comprising four distinct areas, and the Stanford
Question Answering Dataset 1.1 (SQuAD). Our results demonstrate that our
customized adapter-based method achieves performance comparable to full model
fine-tuning, DAPT+TAPT and UniPELT strategies while requiring fewer or
equivalent amount of parameters. This parameter efficiency not only alleviates
the computational burden but also expedites the adaptation process. The study
underlines the potential of adapters in achieving high performance with
significantly reduced resource consumption, suggesting a promising direction
for future research in parameter-efficient fine-tuning.

该研究介绍了一种新的适应方法，使用 UniPELT 框架作为基础，并添加了 PromptTuning 层，从而在保持竞争力的同时显著减少了可训练参数的数量。该方法利用适配器实现了预训练模型向新任务的有效转移，无需重新训练基础模型参数。通过对三个不同数据集进行评估，研究结果表明，该基于适配器的方法在性能上与全模型微调、DAPT+TAPT 和 UniPELT 策略相当，而需要更少或相同数量的参数。这种参数效率不仅减轻了计算负担，还加快了适应过程。该研究强调了适配器在实现高性能以及显著节约资源消耗方面的潜力，为参数高效微调的未来研究方向提供了有益的提示。