Traditional multi-task learning (MTL) methods use dense networks that use the
same set of shared weights across several different tasks. This often creates
interference where two or more tasks compete to pull model parameters in
different directions. In this work, we study whether sparsely activated
Mixture-of-Experts (MoE) improve multi-task learning by specializing some
weights for learning shared representations and using the others for learning
task-specific information. To this end, we devise task-aware gating functions
to route examples from different tasks to specialized experts which share
subsets of network weights conditioned on the task. This results in a sparsely
activated multi-task model with a large number of parameters, but with the same
computational cost as that of a dense model. We demonstrate such sparse
networks to improve multi-task learning along three key dimensions: (i)
transfer to low-resource tasks from related tasks in the training mixture; (ii)
sample-efficient generalization to tasks not seen during training by making use
of task-aware routing from seen related tasks; (iii) robustness to the addition
of unrelated tasks by avoiding catastrophic forgetting of existing tasks.

使用稀疏激活的专家混合模型和任务感知门控函数结合的多任务学习方法，成功地提高了模型泛化能力、转移学习效果和鲁棒性。