Large-scale pretraining followed by task-specific finetuning has achieved
great success in various NLP tasks. Since finetuning all parameters of large
pretrained models poses substantial computational and memory challenges,
several efficient finetuning methods have been developed. Among them, low-rank
adaptation (LoRA), which finetunes low-rank incremental update matrices on top
of frozen pretrained weights, has proven particularly effective. Nonetheless,
LoRA's uniform rank assignment across all layers, along with its reliance on an
exhaustive search to find the best rank, leads to high computation costs and
suboptimal finetuning performance. To address these limitations, we introduce
AutoLoRA, a meta learning based framework for automatically identifying the
optimal rank of each LoRA layer. AutoLoRA associates each rank-1 matrix in a
low-rank update matrix with a selection variable, which determines whether the
rank-1 matrix should be discarded. A meta learning based method is developed to
learn these selection variables. The optimal rank is determined by thresholding
the values of these variables. Our comprehensive experiments on natural
language understanding, generation, and sequence labeling demonstrate the
effectiveness of AutoLoRA.

AutoLoRA 是基于元学习的框架，用于自动识别每个 LoRA 层的最佳秩，通过与选择变量相关联的方法确定最佳秩，并在自然语言理解、生成和序列标记等任务上证明了其有效性。

AutoLoRA: 基于元学习的低秩适应中矩阵秩的自动调整

AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based  on Meta Learning

We present VILLA, the first known effort on large-scale adversarial training
for vision-and-language (V+L) representation learning. VILLA consists of two
training stages: (i) task-agnostic adversarial pre-training; followed by (ii)
task-specific adversarial finetuning. Instead of adding adversarial
perturbations on image pixels and textual tokens, we propose to perform
adversarial training in the embedding space of each modality. To enable
large-scale training, we adopt the "free" adversarial training strategy, and
combine it with KL-divergence-based regularization to promote higher invariance
in the embedding space. We apply VILLA to current best-performing V+L models,
and achieve new state of the art on a wide range of tasks, including Visual
Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval,
Referring Expression Comprehension, Visual Entailment, and NLVR2.

该研究提出了一种基于对抗训练的大规模视觉语言表示学习方法 VILLA，该方法在嵌入空间内进行对抗训练，取得了新的最优性能。