Multi-task learning with transformer encoders (MTL) has emerged as a powerful technique to improve performance on closely-related tasks for both accuracy and efficiency while a question still remains whether or not it would perform as well on tasks that are distinct in nature. We first present MTL results on five NLP tasks, POS, NER, DEP, CON, and SRL, and depict its deficiency over single-task learning. We then conduct an extensive pruning analysis to show that a certain set of attention heads get claimed by most tasks during MTL, who interfere with one another to fine-tune those heads for their own objectives. Based on this finding, we propose the Stem Cell Hypothesis to reveal the existence of attention heads naturally talented for many tasks that cannot be jointly trained to create adequate embeddings for all of those tasks. Finally, we design novel parameter-free probes to justify our hypothesis and demonstrate how attention heads are transformed across the five tasks during MTL through label analysis.

通过多任务学习，结合transformer编码器，研究了多个NLP任务，发现多任务学习的注意力头之间相互干扰，提出干细胞假说解释在某些任务上具有天赋的注意力头不能被同时训练。同时，提出了新的无参考探针来验证该假说，通过标签分析展示了注意力头在五个任务之间如何被转换。

干细胞假说：利用Transformer编码器进行多任务学习面临的困境