Text-rich images have significant and extensive value, deeply integrated into
various aspects of human life. Notably, both visual cues and linguistic symbols
in text-rich images play crucial roles in information transmission but are
accompanied by diverse challenges. Therefore, the efficient and effective
understanding of text-rich images is a crucial litmus test for the capability
of Vision-Language Models. We have crafted an efficient vision-language model,
StrucTexTv3, tailored to tackle various intelligent tasks for text-rich images.
The significant design of StrucTexTv3 is presented in the following aspects:
Firstly, we adopt a combination of an effective multi-scale reduced visual
transformer and a multi-granularity token sampler (MG-Sampler) as a visual
token generator, successfully solving the challenges of high-resolution input
and complex representation learning for text-rich images. Secondly, we enhance
the perception and comprehension abilities of StrucTexTv3 through instruction
learning, seamlessly integrating various text-oriented tasks into a unified
framework. Thirdly, we have curated a comprehensive collection of high-quality
text-rich images, abbreviated as TIM-30M, encompassing diverse scenarios like
incidental scenes, office documents, web pages, and screenshots, thereby
improving the robustness of our model. Our method achieved SOTA results in
text-rich image perception tasks, and significantly improved performance in
comprehension tasks. Among multimodal models with LLM decoder of approximately
1.8B parameters, it stands out as a leader, which also makes the deployment of
edge devices feasible. In summary, the StrucTexTv3 model, featuring efficient
structural design, outstanding performance, and broad adaptability, offers
robust support for diverse intelligent application tasks involving text-rich
images, thus exhibiting immense potential for widespread application.

StrucTexTv3 是一种高效的视觉语言模型，通过采用多尺度减少可视转换器和多粒度令牌采样器作为视觉令牌生成器，结合指令学习提高感知与理解能力，并使用 TIM-30M 来提升模型的鲁棒性，实现了领先的文本丰富图像感知任务和理解任务的结果，展现了广泛应用的巨大潜力。

StrucTexTv3：一个高效的视觉语言模型，适用于文本丰富的图像感知、理解和更多

StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image  Perception, Comprehension, and Beyond

Instruction tuning unlocks the superior capability of Large Language Models
(LLM) to interact with humans. Furthermore, recent instruction-following
datasets include images as visual inputs, collecting responses for image-based
instructions. However, visual instruction-tuned models cannot comprehend
textual details within images well. This work enhances the current visual
instruction tuning pipeline with text-rich images (e.g., movie posters, book
covers, etc.). Specifically, we first use publicly available OCR tools to
collect results on 422K text-rich images from the LAION dataset. Moreover, we
prompt text-only GPT-4 with recognized texts and image captions to generate 16K
conversations, each containing question-answer pairs for text-rich images. By
combining our collected data with previous multi-modal instruction-following
data, our model, LLaVAR, substantially improves the LLaVA model's capability on
text-based VQA datasets (up to 20% accuracy improvement) while achieving an
accuracy of 91.42% on ScienceQA. The GPT-4-based instruction-following
evaluation also demonstrates the improvement of our model on both natural
images and text-rich images. Through qualitative analysis, LLaVAR shows
promising interaction (e.g., reasoning, writing, and elaboration) skills with
humans based on the latest real-world online content that combines text and
images. We make our code/data/models publicly available at
this https URL

本研究提出了 LLaVAR 模型，是通过用包含文字的图像文本对模型进行训练来增强当前视觉指令调优流程的，该模型显着提高了模型在基于文本的 VQA 数据集上的性能，并通过定性分析展示了与人类交互的潜力。