Many datasets have been created for training reading comprehension models,
and a natural question is whether we can combine them to build models that (1)
perform better on all of the training datasets and (2) generalize and transfer
better to new datasets. Prior work has addressed this goal by training one
network simultaneously on multiple datasets, which works well on average but is
prone to over- or under-fitting different sub-distributions and might transfer
worse compared to source models with more overlap with the target dataset. Our
approach is to model multi-dataset question answering with a collection of
single-dataset experts, by training a collection of lightweight,
dataset-specific adapter modules (Houlsby et al., 2019) that share an
underlying Transformer model. We find that these Multi-Adapter Dataset Experts
(MADE) outperform all our baselines in terms of in-distribution accuracy, and
simple methods based on parameter-averaging lead to better zero-shot
generalization and few-shot transfer performance, offering a strong and
versatile starting point for building new reading comprehension systems.

本研究通过训练轻量级、特定于数据集的适配器模块，并将其与基础转换模型共享，以建模多数据集问答，提出了 Multi-Adapter Dataset Experts（MADE）方法，其在数据集内的准确性优于基线模型，并且采用基于参数平均的简单方法可以实现更好的零样本泛化和少样本迁移性能，为构建新的阅读理解系统提供了一个强大而灵活的起点。