Scientific documents record research findings and valuable human knowledge,
comprising a vast corpus of high-quality data. Leveraging multi-modality data
extracted from these documents and assessing large models' abilities to handle
scientific document-oriented tasks is therefore meaningful. Despite promising
advancements, large models still perform poorly on multi-page scientific
document extraction and understanding tasks, and their capacity to process
within-document data formats such as charts and equations remains
under-explored. To address these issues, we present DocGenome, a structured
document benchmark constructed by annotating 500K scientific documents from 153
disciplines in the arXiv open-access community, using our custom auto-labeling
pipeline. DocGenome features four key characteristics: 1) Completeness: It is
the first dataset to structure data from all modalities including 13 layout
attributes along with their LaTeX source codes. 2) Logicality: It provides 6
logical relationships between different entities within each scientific
document. 3) Diversity: It covers various document-oriented tasks, including
document classification, visual grounding, document layout detection, document
transformation, open-ended single-page QA and multi-page QA. 4) Correctness: It
undergoes rigorous quality control checks conducted by a specialized team. We
conduct extensive experiments to demonstrate the advantages of DocGenome and
objectively evaluate the performance of large models on our benchmark.

通过提取自科学文档中的多模态数据，并评估大型模型处理科学文档任务的能力，对科学文档记录的研究发现和有价值的人类知识进行综合性的数据分析是有意义的。我们介绍了 DocGenome，这是一个结构化文档基准，通过我们的自定义自动标注流水线对 arXiv 开放访问社区中的 153 个学科的 50 万份科学文档进行注释，它具有完整性、逻辑性、多样性和正确性等四个关键特征。我们进行广泛的实验证明了 DocGenome 的优势，并客观评估了大型模型在我们的基准上的性能。