Modular and composable transfer learning is an emerging direction in the
field of Parameter Efficient Fine-Tuning, as it enables neural networks to
better organize various aspects of knowledge, leading to improved cross-task
generalization. In this paper, we introduce a novel approach Customized
Polytropon C-Poly that combines task-common skills and task-specific skills,
while the skill parameters being highly parameterized using low-rank
techniques. Each task is associated with a customizable number of exclusive
specialized skills and also benefits from skills shared with peer tasks. A
skill assignment matrix is jointly learned. To evaluate our approach, we
conducted extensive experiments on the Super-NaturalInstructions and the
SuperGLUE benchmarks. Our findings demonstrate that C-Poly outperforms
fully-shared, task-specific, and skill-indistinguishable baselines,
significantly enhancing the sample efficiency in multi-task learning scenarios.

模块化和可组合的迁移学习是参数高效微调领域的新兴方向，它使神经网络能更好地组织各种知识方面，提高跨任务泛化能力；本文介绍了一种新的方法 Customized Polytropon C-Poly，它结合了任务共同技能和任务特定技能，并使用低秩技术对技能参数进行高度参数化；每个任务关联着可定制数量的专属专业技能，并从与同伴任务共享的技能中受益；共同学习了技能分配矩阵；通过在 Super-NaturalInstructions 和 SuperGLUE 基准上进行广泛实验评估，证明了 C-Poly 优于完全共享，任务特定和技能不可区分的基线方法，显著提高了多任务学习场景下的样本效率。