Standard fine-tuning of large pre-trained language models (PLMs) for
downstream tasks requires updating hundreds of millions to billions of
parameters, and storing a large copy of the PLM weights for every task
resulting in increased cost for storing, sharing and serving the models. To
address this, parameter-efficient fine-tuning (PEFT) techniques were introduced
where small trainable components are injected in the PLM and updated during
fine-tuning. We propose AdaMix as a general PEFT method that tunes a mixture of
adaptation modules -- given the underlying PEFT method of choice -- introduced
in each Transformer layer while keeping most of the PLM weights frozen. For
instance, AdaMix can leverage a mixture of adapters like Houlsby or a mixture
of low rank decomposition matrices like LoRA to improve downstream task
performance over the corresponding PEFT methods for fully supervised and
few-shot NLU and NLG tasks. Further, we design AdaMix such that it matches the
same computational cost and the number of tunable parameters as the underlying
PEFT method. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix
outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for
both NLU and NLG tasks.

提出 AdaMix，在保持大部分 PLM 权重不变的情况下，通过调整每个 Transformer 层中引入的适应模块的混合来优化下游任务的性能，仅调整 0.1-0.2％的 PLM 参数即可胜过 SOTA 参数高效调整和完整模型调整的 NLU 和 NLG 任务。

AdaMix: 适应混合策略用于参数高效调整的模型

AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning

本文提出 AdaMix 作为一种通用的参数高效微调方法，通过调整每个 Transformer 层中引入的自适应模块的混合来提高下游自然语言理解和生成任务的性能。这种方法只对 0.1-0.2％的 PLM 参数进行微调，可以超越目前的各种参数高效微调和完整模型微调方法。

AdaMix: 参数高效模型调整的自适应混合

We introduce AdaMix, an adaptive differentially private algorithm for
training deep neural network classifiers using both private and public image
data. While pre-training language models on large public datasets has enabled
strong differential privacy (DP) guarantees with minor loss of accuracy, a
similar practice yields punishing trade-offs in vision tasks. A few-shot or
even zero-shot learning baseline that ignores private data can outperform
fine-tuning on a large private dataset. AdaMix incorporates few-shot training,
or cross-modal zero-shot learning, on public data prior to private fine-tuning,
to improve the trade-off. AdaMix reduces the error increase from the
non-private upper bound from the 167-311\% of the baseline, on average across 6
datasets, to 68-92\% depending on the desired privacy level selected by the
user. AdaMix tackles the trade-off arising in visual classification, whereby
the most privacy sensitive data, corresponding to isolated points in
representation space, are also critical for high classification accuracy. In
addition, AdaMix comes with strong theoretical privacy guarantees and
convergence analysis.

AdaMix 是一种自适应差分隐私算法，用于使用私有和公共图像数据训练深度神经网络分类器。它采用 few-shot training 和 zero-shot learning 来解决视觉分类中的隐私和准确性之间的权衡问题，并提供强大的理论隐私保证和收敛分析。