In multi-task learning, the conventional approach involves training a model
on multiple tasks simultaneously. However, the training signals from different
tasks can interfere with one another, potentially leading to \textit{negative
transfer}. To mitigate this, we investigate if modular language models can
facilitate positive transfer and systematic generalization. Specifically, we
propose a novel modular language model (\texttt{TensorPoly}), that balances
parameter efficiency with nuanced routing methods. For \textit{modules}, we
reparameterize Low-Rank Adaptation (\texttt{LoRA}) by employing an entangled
tensor through the use of tensor product operations and name the resulting
approach \texttt{TLoRA}. For \textit{routing function}, we tailor two
innovative routing functions according to the granularity:
\texttt{TensorPoly-I} which directs to each rank within the entangled tensor
while \texttt{TensorPoly-II} offers a finer-grained routing approach targeting
each order of the entangled tensor. The experimental results from the
multi-task T0-benchmark demonstrate that: 1) all modular LMs surpass the
corresponding dense approaches, highlighting the potential of modular language
models to mitigate negative inference in multi-task learning and deliver
superior outcomes. 2) \texttt{TensorPoly-I} achieves higher parameter
efficiency in adaptation and outperforms other modular LMs, which shows the
potential of our approach in multi-task transfer learning.

在多任务学习中，我们提出了一种新的模块化语言模型（TensorPoly），它通过精细的路由方法和参数效率平衡来减轻负面干扰，并通过高效的自适应方法在多任务迁移学习中取得更好的结果。

基于张量积的专家混合模型

Mixture of Experts Using Tensor Products

Mainstream parameter-efficient fine-tuning (PEFT) methods, such as LoRA or
Adapter, project a model's hidden states to a lower dimension, allowing
pre-trained models to adapt to new data through this low-rank bottleneck.
However, PEFT tasks involving multiple modalities, like vision-language (VL)
tasks, require not only adaptation to new data but also learning the
relationship between different modalities. Targeting at VL PEFT tasks, we
propose a family of operations, called routing functions, to enhance VL
alignment in the low-rank bottlenecks. The routing functions adopt linear
operations and do not introduce new trainable parameters. In-depth analyses are
conducted to study their behavior. In various VL PEFT settings, the routing
functions significantly improve performance of the original PEFT methods,
achieving over 20% improvement on VQAv2
($\text{RoBERTa}_{\text{large}}$+ViT-L/16) and 30% on COCO Captioning
(GPT2-medium+ViT-L/16). Also when fine-tuning a pre-trained multimodal model
such as CLIP-BART, we observe smaller but consistent improvements across a
range of VL PEFT tasks.

针对视觉 - 语言（VL）任务的参数高效微调方法，使用名为路由函数的操作在低秩瓶颈中增强了 VL 对齐，极大地改善了原始 PEFT 方法在各种 VL PEFT 设置中的表现，包括 20% 的提升（在 VQAv2 上）和 30% 的提升（在 COCO Captioning 上），同时也在多种 VL PEFT 任务中对预训练的多模态模型（如 CLIP-BART）进行微调时观察到了较小但一致的改进。

引入路由功能以低秩瓶颈优化视觉语言参数高效微调

Introducing Routing Functions to Vision-Language Parameter-Efficient  Fine-Tuning with Low-Rank Bottlenecks

Fine-grained visual categorization (FGVC) is an important but challenging
task due to high intra-class variances and low inter-class variances caused by
deformation, occlusion, illumination, etc. An attention convolutional binary
neural tree architecture is presented to address those problems for weakly
supervised FGVC. Specifically, we incorporate convolutional operations along
edges of the tree structure, and use the routing functions in each node to
determine the root-to-leaf computational paths within the tree. The final
decision is computed as the summation of the predictions from leaf nodes. The
deep convolutional operations learn to capture the representations of objects,
and the tree structure characterizes the coarse-to-fine hierarchical feature
learning process. In addition, we use the attention transformer module to
enforce the network to capture discriminative features. The negative
log-likelihood loss is used to train the entire network in an end-to-end
fashion by SGD with back-propagation. Several experiments on the CUB-200-2011,
Stanford Cars and Aircraft datasets demonstrate that the proposed method
performs favorably against the state-of-the-arts.

本文提出了一种基于关注的卷积二叉神经树框架，并使用路由功能、注意力变换器模块以及负对数似然损失函数等处理技术来解决细粒度视觉分类中遇到的问题，在多个标准数据集上的实验结果表明该方法性能优于先前提出的方法。