This work presents DocPedia, a novel large multimodal model (LMM) for
versatile OCR-free document understanding, capable of parsing images up to
2,560$\times$2,560 resolution. Unlike existing work either struggle with
high-resolution documents or give up the large language model thus vision or
language ability constrained, our DocPedia directly processes visual input in
the frequency domain rather than the pixel space. The unique characteristic
enables DocPedia to capture a greater amount of visual and textual information
using a limited number of visual tokens. To consistently enhance both
perception and comprehension abilities of our model, we develop a dual-stage
training strategy and enrich instructions/annotations of all training tasks
covering multiple document types. Extensive quantitative and qualitative
experiments conducted on various publicly available benchmarks confirm the
mutual benefits of jointly learning perception and comprehension tasks. The
results provide further evidence of the effectiveness and superior performance
of our DocPedia over other methods.

DocPedia 是一种新型的大型多模态模型，用于 OCR-free 文档理解，能够处理高分辨率图像。通过在频域而不是像素空间中直接处理视觉输入，DocPedia 能够利用有限数量的视觉标记捕捉更多的视觉和文本信息。通过开发双阶段训练策略和丰富的训练任务指导 / 注释，我们不断增强模型的感知和理解能力，实验证实了联合学习感知和理解任务的互利益。结果进一步证明了我们的 DocPedia 相比其他方法的有效性和优越性能。