Visual instruction tuning is the key to building multimodal large language
models (MLLMs), which greatly improves the reasoning capabilities of large
language models (LLMs) in vision scenario. However, existing MLLMs mostly rely
on a mixture of multiple highly diverse visual instruction datasets for
training (even more than a million instructions), which may introduce data
redundancy. To investigate this issue, we conduct a series of empirical
studies, which reveal a significant redundancy within the visual instruction
datasets, and show that greatly reducing the amount of several instruction
dataset even do not affect the performance. Based on the findings, we propose a
new data selection approach TIVE, to eliminate redundancy within visual
instruction data. TIVE first estimates the task-level and instance-level value
of the visual instructions based on computed gradients. Then, according to the
estimated values, TIVE determines the task proportion within the visual
instructions, and selects representative instances to compose a smaller visual
instruction subset for training. Experiments on LLaVA-1.5 show that our
approach using only about 7.5% data can achieve comparable performance as the
full-data fine-tuned model across seven benchmarks, even surpassing it on four
of the benchmarks. Our code and data will be publicly released.

通过对视觉指令数据集的实证研究，发现其存在显著冗余性，因此提出了一种新的数据选择方法 TIVE，它基于计算梯度估计了视觉指令的任务级别和实例级别价值，并根据估计值确定了视觉指令中的任务比例，选择代表性实例组成较小的训练集，实验证明该方法只需使用约 7.5% 的数据即可在七个基准测试中达到与完整数据微调模型相当甚至超过的性能改进。

少即是多：用于视觉指令调优的数据值估计

Less is More: Data Value Estimation for Visual Instruction Tuning

Large language models (LLMs) have made impressive progress in natural
language processing. These models rely on proper human instructions (or
prompts) to generate suitable responses. However, the potential of LLMs are not
fully harnessed by commonly-used prompting methods: many human-in-the-loop
algorithms employ ad-hoc procedures for prompt selection; while auto prompt
generation approaches are essentially searching all possible prompts randomly
and inefficiently. We propose Evoke, an automatic prompt refinement framework.
In Evoke, there are two instances of a same LLM: one as a reviewer
(LLM-Reviewer), it scores the current prompt; the other as an author
(LLM-Author), it edits the prompt by considering the edit history and the
reviewer's feedback. Such an author-reviewer feedback loop ensures that the
prompt is refined in each iteration. We further aggregate a data selection
approach to Evoke, where only the hard samples are exposed to the LLM. The hard
samples are more important because the LLM can develop deeper understanding of
the tasks out of them, while the model may already know how to solve the easier
cases. Experimental results show that Evoke significantly outperforms existing
methods. For instance, in the challenging task of logical fallacy detection,
Evoke scores above 80, while all other baseline methods struggle to reach 20.

LLMs 的潜力并未被常用的提示方法充分利用，我们提出了 Evoke，一种自动提示改进框架，通过作者 - 评论者反馈循环来确保每次迭代中的提示被优化，在数据选择方面进行了进一步的汇总，实验证明 Evoke 显著优于现有方法。