In multi-task learning, the conventional approach involves training a model
on multiple tasks simultaneously. However, the training signals from different
tasks can interfere with one another, potentially leading to \textit{negative
transfer}. To mitigate this, we investigate if modular language models can
facilitate positive transfer and systematic generalization. Specifically, we
propose a novel modular language model (\texttt{TensorPoly}), that balances
parameter efficiency with nuanced routing methods. For \textit{modules}, we
reparameterize Low-Rank Adaptation (\texttt{LoRA}) by employing an entangled
tensor through the use of tensor product operations and name the resulting
approach \texttt{TLoRA}. For \textit{routing function}, we tailor two
innovative routing functions according to the granularity:
\texttt{TensorPoly-I} which directs to each rank within the entangled tensor
while \texttt{TensorPoly-II} offers a finer-grained routing approach targeting
each order of the entangled tensor. The experimental results from the
multi-task T0-benchmark demonstrate that: 1) all modular LMs surpass the
corresponding dense approaches, highlighting the potential of modular language
models to mitigate negative inference in multi-task learning and deliver
superior outcomes. 2) \texttt{TensorPoly-I} achieves higher parameter
efficiency in adaptation and outperforms other modular LMs, which shows the
potential of our approach in multi-task transfer learning.

在多任务学习中，我们提出了一种新的模块化语言模型（TensorPoly），它通过精细的路由方法和参数效率平衡来减轻负面干扰，并通过高效的自适应方法在多任务迁移学习中取得更好的结果。