Multimodal Large Language Models (MLLMs) have shown impressive results on
various multimodal tasks. However, most existing MLLMs are not well suited for
document-oriented tasks, which require fine-grained image perception and
information compression. In this paper, we present TextHawk, a MLLM that is
specifically designed for document-oriented tasks, while preserving the general
capabilities of MLLMs. TextHawk is aimed to explore efficient fine-grained
perception by designing four dedicated components. Firstly, a ReSampling and
ReArrangement (ReSA) module is proposed to reduce the redundancy in the
document texts and lower the computational cost of the MLLM. We explore
encoding the positions of each local feature by presenting Scalable Positional
Embeddings (SPEs), which can preserve the scalability of various image sizes. A
Query Proposal Network (QPN) is then adopted to initialize the queries
dynamically among different sub-images. To further enhance the fine-grained
visual perceptual ability of the MLLM, we design a Multi-Level Cross-Attention
(MLCA) mechanism that captures the hierarchical structure and semantic
relations of document images. Furthermore, we create a new instruction-tuning
dataset for document-oriented tasks by enriching the multimodal document data
with Gemini Pro. We conduct extensive experiments on both general and
document-oriented MLLM benchmarks, and show that TextHawk outperforms the
state-of-the-art methods, demonstrating its effectiveness and superiority in
fine-grained document perception and general abilities.

TextHawk 是一种为面向文档任务设计的多模态大型语言模型，通过引入四个专门的组件实现高效的细粒度视觉感知，捕捉文档图像的层级结构和语义关系，表现出在细粒度文档感知和一般能力上的有效性和优越性。

TextHawk：探索多模态大型语言模型的高效细粒度感知

TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal  Large Language Models

When exploring the development of Artificial General Intelligence (AGI), a
critical task for these models involves interpreting and processing information
from multiple image inputs. However, Large Multimodal Models (LMMs) encounter
two issues in such scenarios: (1) a lack of fine-grained perception, and (2) a
tendency to blend information across multiple images. We first extensively
investigate the capability of LMMs to perceive fine-grained visual details when
dealing with multiple input images. The research focuses on two aspects: first,
image-to-image matching (to evaluate whether LMMs can effectively reason and
pair relevant images), and second, multi-image-to-text matching (to assess
whether LMMs can accurately capture and summarize detailed image information).
We conduct evaluations on a range of both open-source and closed-source large
models, including GPT-4V, Gemini, OpenFlamingo, and MMICL. To enhance model
performance, we further develop a Contrastive Chain-of-Thought (CoCoT)
prompting approach based on multi-input multimodal models. This method requires
LMMs to compare the similarities and differences among multiple image inputs,
and then guide the models to answer detailed questions about multi-image inputs
based on the identified similarities and differences. Our experimental results
showcase CoCoT's proficiency in enhancing the multi-image comprehension
capabilities of large multimodal models.

当探索人工通用智能（AGI）的发展时，大型多模态模型（LMMs）在处理多个图像输入的信息时面临两个问题：细粒度感知的缺乏和融合多个图像的倾向。我们首先广泛研究了 LMMs 在处理多个输入图像时感知细粒度视觉细节的能力。研究集中在两个方面：首先，图像与图像匹配（评估 LMMs 是否能够有效推理和配对相关图像），其次，多图像与文本匹配（评估 LMMs 是否能够准确捕捉和总结详细的图像信息）。我们对一系列开源和闭源的大型模型进行评估，包括 GPT-4V，Gemini，OpenFlamingo 和 MMICL。为了增强模型性能，我们还基于多输入多模态模型开发了一种对比的思维链（CoCoT）启发方法。该方法要求 LMMs 比较多个图像输入的相似性和差异性，然后根据确定的相似性和差异性指导模型回答关于多图像输入的详细问题。我们的实验结果展示了 CoCoT 在增强大型多模态模型的多图像理解能力方面的熟练度。

CoCoT: 大型多模型多图像输入的对比思维链接引导

CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal  Models with Multiple Image Inputs

We empirically investigate proper pre-training methods to build good visual
tokenizers, making Large Language Models (LLMs) powerful Multimodal Large
Language Models (MLLMs). In our benchmark, which is curated to evaluate MLLMs
visual semantic understanding and fine-grained perception capabilities, we
discussed different visual tokenizers pre-trained with dominant methods (i.e.,
DeiT, CLIP, MAE, DINO), and observe that: i) Fully/weakly supervised models
capture more semantics than self-supervised models, but the gap is narrowed by
scaling up the pre-training dataset. ii) Self-supervised models are better at
fine-grained perception, where patch-level supervision is particularly
effective. iii) Tuning the visual tokenizer leads to the loss of semantics
obtained from large-scale pretraining, which is unfavorable with relatively
small-scale instruction-tuning dataset. Given the findings, we reviewed methods
that attempted to unify semantics and fine-grained visual understanding, e.g.,
patch-level feature distillation with semantically-rich targets. We obtain an
intriguing insight mask-based strategies that were once all the rage may not be
applicable for obtaining good visual tokenizers. Based on this critical
observation, we obtain a new MLLM equipped with a tailored Good Visual
Tokenizer (GVT), which exhibits strong visual comprehension capability at
multiple scales. In particular, without introducing extra parameters and
task-specific fine-tuning, GVT achieves superior performance on visual question
answering, image captioning, and other fine-grained visual understanding tasks
such as object counting and multi-class identification.

通过实证研究，我们讨论了将主流的预训练方法应用于视觉编码器以构建良好的视觉分词器，以使大型语言模型（LLM）成为强大的多模态大语言模型（MLLM）。我们发现，GVT 在不引入额外的参数和任务特定的微调的情况下，展现出了在多个尺度上的强大视觉理解能力，特别是在视觉问答、图像字幕、对象计数和多类别识别等细粒度视觉理解任务上具有优异的性能。