Vision-Language Models have made significant progress on many perception-focused tasks, however, their progress on reasoning-focused tasks seem to be limited due to the lack of high-quality and diverse training data. In this work, we aim to address the scarcity issue of reasoning-focused multimodal datasets. We propose VisualWebInstruct - a novel approach that leverages search engine to create a diverse, and high-quality dataset spanning multiple disciplines like math, physics, finance, chemistry, etc. Starting with meticulously selected 30,000 seed images, we employ Google Image search to identify websites containing similar images. We collect and process the HTMLs from over 700K unique URL sources. Through a pipeline of content extraction, filtering and synthesis, we build a dataset of approximately 900K question-answer pairs, with 40% being visual QA pairs and the rest as text QA pairs. Models fine-tuned on VisualWebInstruct demonstrate significant performance gains: (1) training from Llava-OV-mid shows 10-20% absolute point gains across benchmarks, (2) training from MAmmoTH-VL shows 5% absoluate gain. Our best model MAmmoTH-VL2 shows state-of-the-art performance within the 10B parameter class on MMMU-Pro-std (40.7%), MathVerse (42.6%), and DynaMath (55.7%). These remarkable results highlight the effectiveness of our dataset in enhancing VLMs' reasoning capabilities for complex multimodal tasks.

本研究解决了多模态推理数据集稀缺的问题，提出了一种新颖的方法VisualWebInstruct，利用搜索引擎创建覆盖多个学科（如数学、物理、金融等）的多样化高质量数据集。通过从30,000张精选的种子图像出发，最终构建了约90万对问答对，其中40%为视觉问答对。经过在VisualWebInstruct上微调的模型取得了显著的性能提升，尤其是在复杂多模态推理任务中的表现，显示出该数据集在提升视觉语言模型推理能力方面的有效性。

VisualWebInstruct：通过网络搜索扩大多模态指令数据的规模