Instruction tuning unlocks the superior capability of Large Language Models
(LLM) to interact with humans. Furthermore, recent instruction-following
datasets include images as visual inputs, collecting responses for image-based
instructions. However, visual instruction-tuned models cannot comprehend
textual details within images well. This work enhances the current visual
instruction tuning pipeline with text-rich images (e.g., movie posters, book
covers, etc.). Specifically, we first use publicly available OCR tools to
collect results on 422K text-rich images from the LAION dataset. Moreover, we
prompt text-only GPT-4 with recognized texts and image captions to generate 16K
conversations, each containing question-answer pairs for text-rich images. By
combining our collected data with previous multi-modal instruction-following
data, our model, LLaVAR, substantially improves the LLaVA model's capability on
text-based VQA datasets (up to 20% accuracy improvement) while achieving an
accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following
evaluation also demonstrates the improvement of our model on both natural
images and text-rich images. Through qualitative analysis, LLaVAR shows
promising interaction (e.g., reasoning, writing, and elaboration) skills with
humans based on the latest real-world online content that combines text and
images. We make our code/data/models publicly available at
this https URL

本研究提出了 LLaVAR 模型，是通过用包含文字的图像文本对模型进行训练来增强当前视觉指令调优流程的，该模型显着提高了模型在基于文本的 VQA 数据集上的性能，并通过定性分析展示了与人类交互的潜力。