Visual instruction tuning (VIT) for large vision-language models (LVLMs) requires training on expansive datasets of image-instruction pairs, which can be costly. Recent efforts in VIT data selection aim to select a small subset of high-quality image-instruction pairs, reducing VIT runtime while maintaining performance comparable to full-scale training. However, a major challenge often overlooked is that generating instructions from unlabeled images for VIT is highly expensive. Most existing VIT datasets rely heavily on human annotations or paid services like the GPT API, which limits users with constrained resources from creating VIT datasets for custom applications. To address this, we introduce Pre-Instruction Data Selection (PreSel), a more practical data selection paradigm that directly selects the most beneficial unlabeled images and generates instructions only for the selected images. PreSel first estimates the relative importance of each vision task within VIT datasets to derive task-wise sampling budgets. It then clusters image features within each task, selecting the most representative images with the budget. This approach reduces computational overhead for both instruction generation during VIT data formation and LVLM fine-tuning. By generating instructions for only 15% of the images, PreSel achieves performance comparable to full-data VIT on the LLaVA-1.5 and Vision-Flan datasets. The link to our project page: https://bardisafa.github.io/PreSel

本研究解决了视觉指令调优（VIT）中生成指令的高成本问题，提出了一种新的数据选择方法——预指令数据选择（PreSel）。通过优先选择最具价值的未标记图像并仅对这些图像生成指令，PreSel在大幅减少计算开销的同时，能够在仅对15%的图像生成指令的情况下，实现与全数据VIT相当的性能。这一方法使得资源有限的用户能够有效创建定制应用的VIT数据集。

先过滤图像，再生成指令：用于视觉指令调优的预指令数据选择