We present TextMonkey, a large multimodal model (LMM) tailored for
text-centric tasks, including document question answering (DocVQA) and scene
text analysis. Our approach introduces enhancement across several dimensions:
by adopting Shifted Window Attention with zero-initialization, we achieve
cross-window connectivity at higher input resolutions and stabilize early
training; We hypothesize that images may contain redundant tokens, and by using
similarity to filter out significant tokens, we can not only streamline the
token length but also enhance the model's performance. Moreover, by expanding
our model's capabilities to encompass text spotting and grounding, and
incorporating positional information into responses, we enhance
interpretability and minimize hallucinations. Additionally, TextMonkey can be
finetuned to gain the ability to comprehend commands for clicking screenshots.
Overall, our method notably boosts performance across various benchmark
datasets, achieving increases of 5.2%, 6.9%, and 2.8% in Scene Text-Centric
VQA, Document Oriented VQA, and KIE, respectively, especially with a score of
561 on OCRBench, surpassing prior open-sourced large multimodal models for
document understanding. Code will be released at
this https URL

我们介绍了 TextMonkey，一个大型多模态模型（LMM），该模型针对以文本为中心的任务，包括文档问题回答（DocVQA）和场景文本分析。通过采用零初始化的 Shifted Window Attention，我们在更高的输入分辨率下实现了跨窗口连接，并稳定了早期训练；我们假设图像可能包含冗余的标记，通过使用相似性过滤显著的标记，我们不仅可以简化标记长度，还可以提高模型的性能。此外，通过扩展模型的能力，包括文本定位和基准化，并将位置信息融入响应中，我们增强了可解释性并减少了产生虚幻结果的情况。另外，TextMonkey 可以进行微调，以具备理解点击截图命令的能力。总体而言，我们的方法显著提升了在各种基准数据集上的性能，尤其在 OCRBench 上获得了 561 分的分数，超过了先前用于文档理解的开源大型多模态模型。代码将在此 URL 发布。