Efficient training and inference algorithms, such as low-rank adaption and
model pruning, have shown impressive performance for learning Transformer-based
large foundation models. However, due to the technical challenges of the
non-convex optimization caused by the complicated architecture of Transformers,
the theoretical study of why these methods can be applied to learn Transformers
is mostly elusive. To the best of our knowledge, this paper shows the first
theoretical analysis of the property of low-rank and sparsity of one-layer
Transformers by characterizing the trained model after convergence using
stochastic gradient descent. By focusing on a data model based on
label-relevant and label-irrelevant patterns, we quantify that the gradient
updates of trainable parameters are low-rank, which depends on the number of
label-relevant patterns. We also analyze how model pruning affects the
generalization while improving computation efficiency and conclude that proper
magnitude-based pruning has a slight effect on the testing performance. We
implement numerical experiments to support our findings.

这篇论文首次从理论上分析了低秩和稀疏性在一层 Transformer 中的特性，并通过数量化可训练参数的梯度更新得出了梯度具有低秩性的结论，同时论文还分析了模型剪枝对泛化能力的影响以及对计算效率的改善。

基于变形器的学习可证明具有低秩和稀疏性：一层分析

Learning on Transformers is Provable Low-Rank and Sparse: A One-layer  Analysis

Low-rank Adaption (LoRA) has been the de-facto parameter-efficient
fine-tuning technique for large language models. We present HeteroLoRA, a
light-weight search algorithm that leverages zero-cost proxies to allocate the
limited LoRA trainable parameters across the model for better fine-tuned
performance. In addition to the allocation for the standard LoRA-adapted
models, we also demonstrate the efficacy of HeteroLoRA by performing the
allocation in a more challenging search space that includes LoRA modules and
LoRA-adapted shortcut connections. Experiments show that HeteroLoRA enables
improvements in model performance given the same parameter budge. For example,
on MRPC, we see an improvement of 1.6% in accuracy with similar training
parameter budget. We will open-source our algorithm once the paper is accepted.

我们介绍了 HeteroLoRA，这是一种轻量级的搜索算法，通过利用零成本代理在模型中分配有限的 LoRA 可训练参数，以提高微调性能，并在更具挑战性的搜索空间中展示了 HeteroLoRA 的有效性。

解码低秩适配器中的全球协同效应

Unlocking the Global Synergies in Low-Rank Adapters

Low-rank adaption (LoRA) is a prominent method that adds a small number of
learnable parameters to the frozen pre-trained weights for parameter-efficient
fine-tuning. Prompted by the question, ``Can we make its representation enough
with LoRA weights solely at the final phase of finetuning without the
pre-trained weights?'' In this work, we introduce Progressive Compression
LoRA~(PC-LoRA), which utilizes low-rank adaptation (LoRA) to simultaneously
perform model compression and fine-tuning. The PC-LoRA method gradually removes
the pre-trained weights during the training process, eventually leaving only
the low-rank adapters in the end. Thus, these low-rank adapters replace the
whole pre-trained weights, achieving the goals of compression and fine-tuning
at the same time. Empirical analysis across various models demonstrates that
PC-LoRA achieves parameter and FLOPs compression rates of 94.36%/89.1% for
vision models, e.g., ViT-B, and 93.42%/84.2% parameters and FLOPs compressions
for language models, e.g., BERT.

逐步压缩低秩适应（PC-LoRA）方法通过低秩适应同时进行模型压缩和微调，最终仅保留低秩适配器以取代预训练权重，实现了参数和计算量的压缩。

PC-LoRA：基于知识蒸馏的逐步模型压缩的低秩适应

PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with  Knowledge Distillation

Multimodal Large Language Models (MLLMs) have demonstrated remarkable
proficiency in diverse tasks across different domains, with an increasing focus
on improving their zero-shot generalization capabilities for unseen multimodal
tasks. Multimodal instruction tuning has emerged as a successful strategy for
achieving zero-shot generalization by fine-tuning pre-trained models on diverse
multimodal tasks through instructions. As MLLMs grow in complexity and size,
the need for parameter-efficient fine-tuning methods like Low-Rank Adaption
(LoRA), which fine-tunes with a minimal set of parameters, becomes essential.
However, applying LoRA in multimodal instruction tuning presents the challenge
of task interference, which leads to performance degradation, especially when
dealing with a broad array of multimodal tasks. To address this, this paper
introduces a novel approach that integrates multimodal instruction tuning with
Conditional Mixture-of-LoRA (MixLoRA). It innovates upon LoRA by dynamically
constructing low-rank adaptation matrices tailored to the unique demands of
each input instance, aiming to mitigate task interference. Experimental results
on various multimodal evaluation datasets indicate that MixLoRA not only
outperforms the conventional LoRA with the same or even higher ranks,
demonstrating its efficacy and adaptability in diverse multimodal tasks.

通过与多模态指令调优相结合，引入了一种综合了条件的低秩适应矩阵 (MixLoRA) 的新方法，旨在减轻任务干扰，实现在多样的多模态任务中具有良好的适应性和灵活性。实验结果表明，MixLoRA 不仅在相同或更高秩的情况下胜过传统的低秩适应方法 LoRA，展现出其在各种多模态评估数据集上的有效性和适应性。