The knowledge encapsulated in a model is the core factor determining its
final performance on downstream tasks. Much research in NLP has focused on
efficient methods for storing and adapting different types of knowledge, e.g.,
in dedicated modularized structures, and on how to effectively combine these,
e.g., by learning additional parameters. However, given the many possible
options, a thorough understanding of the mechanisms involved in these
compositions is missing, and hence it remains unclear which strategies to
utilize. To address this research gap, we propose a novel framework for
zero-shot module composition, which encompasses existing and some novel
variations for selecting, weighting, and combining parameter modules under a
single unified notion. Focusing on the scenario of domain knowledge and adapter
layers, our framework provides a systematic unification of concepts, allowing
us to conduct the first comprehensive benchmarking study of various zero-shot
knowledge composition strategies. In particular, we test two module combination
methods and five selection and weighting strategies for their effectiveness and
efficiency in an extensive experimental setup. Our results highlight the
efficacy of ensembling but also hint at the power of simple though
often-ignored weighting methods. Further in-depth analyses allow us to
understand the role of weighting vs. top-k selection, and show that, to a
certain extent, the performance of adapter composition can even be predicted.

通过提出一个新的框架，对领域知识和适配器层进行零样本模块组合，我们进行了全面的基准研究，结果表明集成方法以及简单的加权方法在组合模块方面的有效性。

零样本知识组合的统一框架

What the Weight?! A Unified Framework for Zero-Shot Knowledge  Composition

Factual questions typically can be answered correctly at different levels of
granularity. For example, both ``August 4, 1961'' and ``1961'' are correct
answers to the question ``When was Barack Obama born?''. Standard question
answering (QA) evaluation protocols, however, do not explicitly take this into
account and compare a predicted answer against answers of a single granularity
level. In this work, we propose GRANOLA QA, a novel evaluation setting where a
predicted answer is evaluated in terms of accuracy and informativeness against
a set of multi-granularity answers. We present a simple methodology for
enriching existing datasets with multi-granularity answers, and create
GRANOLA-EQ, a multi-granularity version of the EntityQuestions dataset. We
evaluate a range of decoding methods on GRANOLA-EQ, including a new algorithm,
called Decoding with Response Aggregation (DRAG), that is geared towards
aligning the response granularity with the model's uncertainty. Our experiments
show that large language models with standard decoding tend to generate
specific answers, which are often incorrect. In contrast, when evaluated on
multi-granularity answers, DRAG yields a nearly 20 point increase in accuracy
on average, which further increases for rare entities. Overall, this reveals
that standard evaluation and decoding schemes may significantly underestimate
the knowledge encapsulated in LMs.

标准问答评估协议未考虑到答案的多粒度性质，与单一粒度的参考答案进行比较。本研究提出 GRANOLA QA 评估设置，用于多粒度答案的准确性和信息量评估。通过对现有数据集进行改进，建立了 GRANOLA-EQ 多粒度实体问题数据集，并在 GRANOLA-EQ 上评估了一系列解码方法，包括 DRAG 算法。实验证明，大型语言模型在标准解码下往往生成具体但错误答案，而 DRAG 算法在多粒度答案上平均提高了近 20 个百分点的准确性，对于罕见实体进一步提高。总体而言，这表明标准评估和解码方法可能严重低估了语言模型所包含的知识。