Instruction tuning, or supervised finetuning on extensive task-specific data,
is necessary for Large Vision-Language Models (LVLMs) to generalize well across
a broad range of vision-language (VL) tasks. However, training on large VL
datasets can become prohibitively expensive. In this work, we introduce
COINCIDE, an effective and scalable data selection technique that uses a small
model as a reference model to select visual instruction tuning data for
efficient finetuning of a target LVLM, focusing on diversity and
transferability. Specifically, we cluster the training data using internal
activations from a small model, which identifies VL concept-skill compositions
needed by a target LVLM. We then sample data from these diverse clusters by
considering their density and transferability, or the ability to transfer well
to other concept-skill compositions. This approach ensures the diversity of
these compositions, which is vital for LVLM generalization. Extensive
experiments demonstrate that COINCIDE achieves superior performance and data
selection efficiency against 8 strong baselines on two distinct datasets:
LLaVA-1.5 and Vision-Flan. Using only 20% of the LLaVA-1.5 dataset, COINCIDE
achieves performance comparable to the LVLM finetuned on the whole dataset,
with 70% reduction of the wall-clock running time. On the Vision-Flan dataset,
our method achieves superior results with only 16.7% of the training data.

通过使用小模型作为参考模型，COINCIDE 是一种有效且可伸缩的数据选择技术，可用于选择用于目标 LVLM 的高效微调的视觉指令数据，重点关注多样性和可转移性。