In this study, we formulate an OCR-free sequence generation model for visual
document understanding (VDU). Our model not only parses text from document
images but also extracts the spatial coordinates of the text based on the
multi-head architecture. Named as Coordinate-aware End-to-end Document Parser
(CREPE), our method uniquely integrates these capabilities by introducing a
special token for OCR text, and token-triggered coordinate decoding. We also
proposed a weakly-supervised framework for cost-efficient training, requiring
only parsing annotations without high-cost coordinate annotations. Our
experimental evaluations demonstrate CREPE's state-of-the-art performances on
document parsing tasks. Beyond that, CREPE's adaptability is further
highlighted by its successful usage in other document understanding tasks such
as layout analysis, document visual question answering, and so one. CREPE's
abilities including OCR and semantic parsing not only mitigate error
propagation issues in existing OCR-dependent methods, it also significantly
enhance the functionality of sequence generation models, ushering in a new era
for document understanding studies.

通过 OCR-free 序列生成模型，本研究提出了 Coordinate-aware End-to-end Document Parser (CREPE) 模型，用于图像文档理解。该模型不仅可以从文档图像中解析文本，还可以提取文本的空间坐标，并通过引入特殊标记和基于标记的坐标解码实现这些功能。实验结果表明 CREPE 在文档解析任务上取得了最新的性能，其适应性还体现在布局分析、文档视觉问答等其他文档理解任务中。该模型不仅减少了现有依赖 OCR 方法的错误传播问题，还显著增强了序列生成模型的功能，引领了文档理解研究的新时代。