The advent of Large Multimodal Models (LMMs) has sparked a surge in research
aimed at harnessing their remarkable reasoning abilities. However, for
understanding text-rich images, challenges persist in fully leveraging the
potential of LMMs, and existing methods struggle with effectively processing
high-resolution images. In this work, we propose TextCoT, a novel
Chain-of-Thought framework for text-rich image understanding. TextCoT utilizes
the captioning ability of LMMs to grasp the global context of the image and the
grounding capability to examine local textual regions. This allows for the
extraction of both global and local visual information, facilitating more
accurate question-answering. Technically, TextCoT consists of three stages,
including image overview, coarse localization, and fine-grained observation.
The image overview stage provides a comprehensive understanding of the global
scene information, and the coarse localization stage approximates the image
area containing the answer based on the question asked. Then, integrating the
obtained global image descriptions, the final stage further examines specific
regions to provide accurate answers. Our method is free of extra training,
offering immediate plug-and-play functionality. Extensive experiments are
conducted on a series of text-rich image question-answering benchmark datasets
based on several advanced LMMs, and the results demonstrate the effectiveness
and strong generalization ability of our method. Code is available at
this https URL

提出了一种名为 TextCoT 的 Chain-of-Thought 框架，用于理解富文本图像，利用 LMMs 的字幕能力来把握图像的全局背景和细节地区，从而提供准确的问题回答，方法经过了广泛实验验证了其有效性和强大的普适能力。