Concerns about reproducibility in artificial intelligence (AI) have emerged, as researchers have reported unsuccessful attempts to directly reproduce published findings in the field. Replicability, the ability to affirm a finding using the same procedures on new data, has not been well studied. In this paper, we examine both reproducibility and replicability of a corpus of 16 papers on table structure recognition (TSR), an AI task aimed at identifying cell locations of tables in digital documents. We attempt to reproduce published results using codes and datasets provided by the original authors. We then examine replicability using a dataset similar to the original as well as a new dataset, GenTSR, consisting of 386 annotated tables extracted from scientific papers. Out of 16 papers studied, we reproduce results consistent with the original in only four. Two of the four papers are identified as replicable using the similar dataset under certain IoU values. No paper is identified as replicable using the new dataset. We offer observations on the causes of irreproducibility and irreplicability. All code and data are available on Codeocean at https://codeocean.com/capsule/6680116/tree.

本文考察了人工智能表格结构识别领域中16篇已发表论文的可重现性和可复制性，通过使用原始作者提供的代码和数据集尝试复现其结果，并采用类似数据集和新数据集进行复制性检查，检验结果表明只有其中4篇论文的结果与原始相符，其中2篇论文的可复制性可在一定IoU值下得到证实，新数据集没有证实任何论文的可复制性。

表格结构识别方法的可重现性和可复制性研究