Multimodal Large Language Models (MLLMs) have demonstrated remarkable
proficiency in diverse tasks across different domains, with an increasing focus
on improving their zero-shot generalization capabilities for unseen multimodal
tasks. Multimodal instruction tuning has emerged as a successful strategy for
achieving zero-shot generalization by fine-tuning pre-trained models on diverse
multimodal tasks through instructions. As MLLMs grow in complexity and size,
the need for parameter-efficient fine-tuning methods like Low-Rank Adaption
(LoRA), which fine-tunes with a minimal set of parameters, becomes essential.
However, applying LoRA in multimodal instruction tuning presents the challenge
of task interference, which leads to performance degradation, especially when
dealing with a broad array of multimodal tasks. To address this, this paper
introduces a novel approach that integrates multimodal instruction tuning with
Conditional Mixture-of-LoRA (MixLoRA). It innovates upon LoRA by dynamically
constructing low-rank adaptation matrices tailored to the unique demands of
each input instance, aiming to mitigate task interference. Experimental results
on various multimodal evaluation datasets indicate that MixLoRA not only
outperforms the conventional LoRA with the same or even higher ranks,
demonstrating its efficacy and adaptability in diverse multimodal tasks.

通过与多模态指令调优相结合，引入了一种综合了条件的低秩适应矩阵 (MixLoRA) 的新方法，旨在减轻任务干扰，实现在多样的多模态任务中具有良好的适应性和灵活性。实验结果表明，MixLoRA 不仅在相同或更高秩的情况下胜过传统的低秩适应方法 LoRA，展现出其在各种多模态评估数据集上的有效性和适应性。

多模态指令调节与条件化 LoRA 混合

Multimodal Instruction Tuning with Conditional Mixture of LoRA

Autoregressive Visual Language Models (VLMs) showcase impressive few-shot
learning capabilities in a multimodal context. Recently, multimodal instruction
tuning has been proposed to further enhance instruction-following abilities.
However, we uncover the potential threat posed by backdoor attacks on
autoregressive VLMs during instruction tuning. Adversaries can implant a
backdoor by injecting poisoned samples with triggers embedded in instructions
or images, enabling malicious manipulation of the victim model's predictions
with predefined triggers. Nevertheless, the frozen visual encoder in
autoregressive VLMs imposes constraints on the learning of conventional image
triggers. Additionally, adversaries may encounter restrictions in accessing the
parameters and architectures of the victim model. To address these challenges,
we propose a multimodal instruction backdoor attack, namely VL-Trojan. Our
approach facilitates image trigger learning through an isolating and clustering
strategy and enhance black-box-attack efficacy via an iterative character-level
text trigger generation method. Our attack successfully induces target outputs
during inference, significantly surpassing baselines (+62.52\%) in ASR.
Moreover, it demonstrates robustness across various model scales and few-shot
in-context reasoning scenarios.

通过 VL-Trojan 攻击，我们成功诱导目标输出，在推理过程中明显超过了基线（+62.52%），并且在各种模型规模和少样本上下文推理场景中展示了鲁棒性。

VL-Trojan: 自回归视觉语言模型的多模态指令后门攻击

VL-Trojan: Multimodal Instruction Backdoor Attacks against  Autoregressive Visual Language Models

There is an emerging line of research on multimodal instruction tuning, and a
line of benchmarks have been proposed for evaluating these models recently.
Instead of evaluating the models directly, in this paper we try to evaluate the
Vision-Language Instruction-Tuning (VLIT) datasets themselves and further seek
the way of building a dataset for developing an all-powerful VLIT model, which
we believe could also be of utility for establishing a grounded protocol for
benchmarking VLIT models. For effective analysis of VLIT datasets that remains
an open question, we propose a tune-cross-evaluation paradigm: tuning on one
dataset and evaluating on the others in turn. For each single tune-evaluation
experiment set, we define the Meta Quality (MQ) as the mean score measured by a
series of caption metrics including BLEU, METEOR, and ROUGE-L to quantify the
quality of a certain dataset or a sample. On this basis, to evaluate the
comprehensiveness of a dataset, we develop the Dataset Quality (DQ) covering
all tune-evaluation sets. To lay the foundation for building a comprehensive
dataset and developing an all-powerful model for practical applications, we
further define the Sample Quality (SQ) to quantify the all-sided quality of
each sample. Extensive experiments validate the rationality of the proposed
evaluation paradigm. Based on the holistic evaluation, we build a new dataset,
REVO-LION (REfining VisiOn-Language InstructiOn tuNing), by collecting samples
with higher SQ from each dataset. With only half of the full data, the model
trained on REVO-LION can achieve performance comparable to simply adding all
VLIT datasets up. In addition to developing an all-powerful model, REVO-LION
also includes an evaluation set, which is expected to serve as a convenient
evaluation benchmark for future research.

本文针对多模态指导调优进行了研究，提出了一种新的评估范式并构建了一套新的数据集 REVO-LION，该数据集包括为发展全能 VLIT 模型提供基础的样本质量、综合性评估及性能可比性验证。