Several parameter-efficient fine-tuning methods based on adapters have been proposed as a streamlined approach to incorporate not only a single specialized knowledge into existing Pre-Trained Language Models (PLMs) but also multiple of them at once. Recent works such as AdapterSoup propose to mix not all but only a selective sub-set of domain-specific adapters during inference via model weight averaging to optimize performance on novel, unseen domains with excellent computational efficiency. However, the essential generalizability of this emerging weight-space adapter mixing mechanism on unseen, in-domain examples remains unexplored. Thus, in this study, we conduct a comprehensive analysis to elucidate the generalizability of domain-specific adapter mixtures in in-domain evaluation. We also provide investigations into the inner workings of the mixture of domain-specific adapters by analyzing their weight signs, yielding critical analysis on the negative correlation between their fraction of weight sign difference and their mixtures' generalizability. All source code will be published.

通过细调参数的方法，本文提出了基于适配器的若干参数高效的Fine-Tuning方法，以简化将单个专业知识和多个专业知识合并到现有的预训练语言模型（PLMs）中的方法。最近的研究（例如AdapterSoup）提出了一种方法，通过模型权重平均化，在推理时只混合领域特定适配器的子集，以优化在未知领域上的性能并具有出色的计算效率。然而，这种在未知领域内示例中的权重空间适配器混合机制的普适性尚未得到探索。因此，在本研究中，我们进行了全面的分析以阐明领域特定适配器混合物在领域内评估中的普适性，并通过分析适配器的权重符号来探究领域特定适配器混合物的内在工作机制，从而得出它们之间的权重符号差异与混合物的普适性之间的负相关关系的关键分析。所有源代码将被发布。

基于有符号权重方向的特定领域适配器混合模型的普适性及其在模型精简中的应用