Multi-Task Learning (MTL) networks have emerged as a promising method for
transferring learned knowledge across different tasks. However, MTL must deal
with challenges such as: overfitting to low resource tasks, catastrophic
forgetting, and negative task transfer, or learning interference. Often, in
Natural Language Processing (NLP), a separate model per task is needed to
obtain the best performance. However, many fine-tuning approaches are both
parameter inefficient, i.e., potentially involving one new model per task, and
highly susceptible to losing knowledge acquired during pretraining. We propose
a novel Transformer architecture consisting of a new conditional attention
mechanism as well as a set of task-conditioned modules that facilitate weight
sharing. Through this construction (a hypernetwork adapter), we achieve more
efficient parameter sharing and mitigate forgetting by keeping half of the
weights of a pretrained model fixed. We also use a new multi-task data sampling
strategy to mitigate the negative effects of data imbalance across tasks. Using
this approach, we are able to surpass single task fine-tuning methods while
being parameter and data efficient (using around 66% of the data for weight
updates). Compared to other BERT Large methods on GLUE, our 8-task model
surpasses other Adapter methods by 2.8% and our 24-task model outperforms by
0.7-1.0% models that use MTL and single task fine-tuning. We show that a larger
variant of our single multi-task model approach performs competitively across
26 NLP tasks and yields state-of-the-art results on a number of test and
development sets. Our code is publicly available at
this https URL

通过使用新的 Transformer 架构，包括一种新的条件注意机制以及一组任务条件模块，旨在促进权重共享，我们实现了更有效的参数共享，并通过保持预训练模型的一半权重来缓解遗忘。同时，我们使用了新的多任务数据采样策略来减少任务之间数据不平衡的负面影响。与其他 BERT Large 方法在 GLUE 上相比，我们的八任务模型超过了其他 Adapter 方法 2.8％，而我们的 24 任务模型在使用 MTL 和单任务微调的模型上表现优异。我们还展示了我们的单个多任务模型方法的较大变体在 26 个 NLP 任务中竞争，并在一些测试和开发集上取得了最先进的结果。