Parameter-efficient fine-tuning (PEFT) techniques, such as adapter tuning,
aim to fine-tune a pre-trained language model (PLM) using a minimal number of
parameters for a specific task or profile. Although adapter tuning provides
increased parameter efficiency compared to full-model fine-tuning, it
introduces a small set of additional parameters attached to a PLM for each
profile. This can become problematic in practical applications with multiple
profiles, particularly when a significant increase in the number of profiles
linearly boosts the total number of additional parameters. To mitigate this
issue, we introduce X-PEFT, a novel PEFT method that leverages a multitude of
given adapters by fine-tuning an extremely small set of compact tensors for a
new profile, which serve as binary masks to adaptively select the given
adapters. To efficiently validate our proposed method, we implement it using a
large number of trained or untrained (random) adapters. We evaluate the
performance of X-PEFT through LaMP and GLUE tasks and demonstrate that it
either matches or surpasses the effectiveness of conventional adapter tuning,
despite reducing the memory requirements per profile by a factor of 10,000
compared to it.

引入 X-PEFT，一种新的参数高效微调方法，通过微调极小的紧凑张量，作为二进制掩码来自适应地选择给定适配器，从而解决适配器数量线性增加的问题，相较于传统的适配器微调，在每个配置文件的内存需求减少了 10000 倍，而在 LaMP 和 GLUE 任务中表现出与传统适配器微调相当或超越的效果。