Information Extraction processes in handwritten documents tend to rely on
obtaining an automatic transcription and performing Named Entity Recognition
(NER) over such transcription. For this reason, in publicly available datasets,
the performance of the systems is usually evaluated with metrics particular to
each dataset. Moreover, most of the metrics employed are sensitive to reading
order errors. Therefore, they do not reflect the expected final application of
the system and introduce biases in more complex documents. In this paper, we
propose and publicly release a set of reading order independent metrics
tailored to Information Extraction evaluation in handwritten documents. In our
experimentation, we perform an in-depth analysis of the behavior of the metrics
to recommend what we consider to be the minimal set of metrics to evaluate a
task correctly.

手写文件中的信息提取过程往往依赖于获得自动转录并对该转录执行命名实体识别（NER）；出于这个原因，在公开可用的数据集中，系统的性能通常使用适用于每个数据集的特定指标进行评估。而且，大多数使用的指标对阅读顺序错误非常敏感，因此不能反映系统的预期最终应用，并且在更复杂的文件中引入了偏见。在本文中，我们提出并公开发布了一套针对手写文件中信息提取评估的与阅读顺序无关的度量标准。在实验中，我们对度量标准的行为进行了深入分析，并推荐了我们认为的正确评估任务所需的最小度量标准。

手写文件中信息提取的读取顺序无关度量

Reading Order Independent Metrics for Information Extraction in  Handwritten Documents

We propose multiple techniques for automatic document order generation for
(1) curriculum development and for (2) creation of optimal reading order for
use in learning, training, and other content-sequencing applications. Such
techniques could potentially be used to improve comprehension, identify areas
that need expounding, generate curricula, and improve search engine results. We
advance two main techniques: The first uses document similarities through
various methods. The second uses entropy against the backdrop of topics
generated through Latent Dirichlet Allocation (LDA). In addition, we try the
same methods on the summarized documents and compare them against the results
obtained using the complete documents. Our results showed that while the
document orders for our control document sets (biographies, novels, and
Wikipedia articles) could not be predicted using our methods, our test
documents (textbooks, courses, journal papers, dissertations) provided more
reliability. We also demonstrated that summarized documents were good stand-ins
for the complete documents for the purposes of ordering.

我们提出了多种技术，用于自动文档排序生成，用于课程开发和在学习、培训和其他内容排序应用中创建最佳阅读顺序。我们通过使用不同方法的文档相似度和潜在狄利克雷分配（LDA）生成的主题熵，推进了两个主要技术。我们的结果表明，通过我们的方法无法预测我们的控制文档集（传记、小说和维基百科文章）的文档顺序，但我们的测试文档（教科书、课程、期刊论文、学位论文）提供了更可靠的排列。我们还证明了摘要文档可以很好地代替完整文档进行排序。