Visually Rich Form Understanding (VRFU) poses a complex research problem due
to the documents' highly structured nature and yet highly variable style and
content. Current annotation schemes decompose form understanding and omit key
hierarchical structure, making development and evaluation of end-to-end models
difficult. In this paper, we propose a novel F1 metric to evaluate form parsers
and describe a new content-agnostic, tree-based annotation scheme for VRFU:
TreeForm. We provide methods to convert previous annotation schemes into
TreeForm structures and evaluate TreeForm predictions using a modified version
of the normalized tree-edit distance. We present initial baselines for our
end-to-end performance metric and the TreeForm edit distance, averaged over the
FUNSD and XFUND datasets, of 61.5 and 26.4 respectively. We hope that TreeForm
encourages deeper research in annotating, modeling, and evaluating the
complexities of form-like documents.

我们提出了一种新的 F1 度量标准来评估表单解析器，并描述了一种适用于 Visually Rich Form Understanding 的内容不可知的基于树的注释方案：TreeForm。我们提供了将以前的注释方案转换为 TreeForm 结构并使用归一化树编辑距离的修改版本来评估 TreeForm 预测的方法。我们在 FUNSD 和 XFUND 数据集上分别得到了 61.5 和 26.4 的 end-to-end 性能评估和 TreeForm 编辑距离的初始基线。我们希望 TreeForm 可以促进对表单类文档注释、建模和评估复杂性的深入研究。

TreeForm: 表格文档解析的端到端标注与评估

TreeForm: End-to-end Annotation and Evaluation for Form Document Parsing

Scientific Information Extraction (ScientificIE) is a critical task that
involves the identification of scientific entities and their relationships. The
complexity of this task is compounded by the necessity for domain-specific
knowledge and the limited availability of annotated data. Two of the most
popular datasets for ScientificIE are SemEval-2018 Task-7 and SciERC. They have
overlapping samples and differ in their annotation schemes, which leads to
conflicts. In this study, we first introduced a novel approach based on
multi-task learning to address label variations. We then proposed a soft
labeling technique that converts inconsistent labels into probabilistic
distributions. The experimental results demonstrated that the proposed method
can enhance the model robustness to label noise and improve the end-to-end
performance in both ScientificIE tasks. The analysis revealed that label
variations can be particularly effective in handling ambiguous instances.
Furthermore, the richness of the information captured by label variations can
potentially reduce data size requirements. The findings highlight the
importance of releasing variation labels and promote future research on other
tasks in other domains. Overall, this study demonstrates the effectiveness of
multi-task learning and the potential of label variations to enhance the
performance of ScientificIE.

本研究通过多任务学习和标签变异的方法，提高科学信息抽取的性能，并展示了标签变异在处理模糊实例和减少数据需求方面的潜力。

通过多任务学习解决科学信息提取中的标签变异问题

Solving Label Variation in Scientific Information Extraction via  Multi-Task Learning

The diverse relationships among real-world events, including coreference,
temporal, causal, and subevent relations, are fundamental to understanding
natural languages. However, two drawbacks of existing datasets limit event
relation extraction (ERE) tasks: (1) Small scale. Due to the annotation
complexity, the data scale of existing datasets is limited, which cannot well
train and evaluate data-hungry models. (2) Absence of unified annotation.
Different types of event relations naturally interact with each other, but
existing datasets only cover limited relation types at once, which prevents
models from taking full advantage of relation interactions. To address these
issues, we construct a unified large-scale human-annotated ERE dataset
MAVEN-ERE with improved annotation schemes. It contains 103,193 event
coreference chains, 1,216,217 temporal relations, 57,992 causal relations, and
15,841 subevent relations, which is larger than existing datasets of all the
ERE tasks by at least an order of magnitude. Experiments show that ERE on
MAVEN-ERE is quite challenging, and considering relation interactions with
joint learning can improve performances. The dataset and source codes can be
obtained from this https URL.

本研究构建了一个统一的大规模人工标注事件关系提取数据集 MAVEN-ERE，并改进了标注方案。该数据集包含 103,193 个事件共指链，1,216,217 个时间关系，57,992 个因果关系和 15,841 个子事件关系，是现有所有事件关系提取任务数据集的十倍以上。实验表明，MAVEN-ERE 上的事件关系提取具有相当高的挑战性，联合学习可以提高性能。

MAVEN-ERE: 一个用于事件指代消解、时间、因果和子事件关系抽取的统一大规模数据集

MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction

Although, recent advances in neural network models for coreference resolution
have led to substantial improvements on benchmark datasets, it remains a
challenge to successfully transfer those models to new target domains
containing many out-of-vocabulary spans and requiring differing annotation
schemes. Typical approaches for domain adaptation involve continued training on
coreference annotations in the target domain, but obtaining those annotations
is costly and time-consuming. In this work, we show that adapting mention
detection is the key component to successful domain adaptation of coreference
models, rather than antecedent linking. Through timed annotation experiments,
we also show annotating mentions alone is nearly twice as fast as annotating
full coreference chains. Based on these insights, we propose a method for
effectively adapting coreference models that requires only mention annotations
in the target domain. We use an auxiliary mention detection objective trained
with mention examples in the target domain resulting in higher mention
precision. We demonstrate that our approach facilitates sample- and
time-efficient transfer to new annotation schemes and lexicons in extensive
evaluation across three English coreference datasets: CoNLL-2012
(news/conversation), i2b2/VA (medical case notes), and a dataset of child
welfare case notes. We show that annotating mentions results in 7-14%
improvement in average F1 over annotating coreference over an equivalent amount
of time.

本文探讨了利用提及检测在目标域中进行兼容性适应的方法来提高神经网络核心指称分辨率的效率，取得了 7-14％的通过提及注释来提高核心指称分辨率的平均 F1 值的改进。