Relation Extraction (RE) has attracted increasing attention, but current RE
evaluation is limited to in-domain evaluation setups. Little is known on how
well a RE system fares in challenging, but realistic out-of-distribution
evaluation setups. To address this gap, we propose CrossRE, a new,
freely-available cross-domain benchmark for RE, which comprises six distinct
text domains and includes multi-label annotations. An additional innovation is
that we release meta-data collected during annotation, to include explanations
and flags of difficult instances. We provide an empirical evaluation with a
state-of-the-art model for relation classification. As the meta-data enables us
to shed new light on the state-of-the-art model, we provide a comprehensive
analysis on the impact of difficult cases and find correlations between model
and human annotations. Overall, our empirical investigation highlights the
difficulty of cross-domain RE. We release our dataset, to spur more research in
this direction.

本文提出了一种新的跨领域关系抽取基准，并提供了元数据和多标签注释来帮助理解困难实例。作者通过在六个不同的文本领域上进行评估，评估了最先进的关系分类模型，发现跨领域关系抽取的困难性，为该方向的研究提供了新的数据集和启示。

CrossRE：一个跨领域的关系抽取数据集

CrossRE: A Cross-Domain Dataset for Relation Extraction

Epistemic voting interprets votes as noisy signals about a ground truth. We
consider contexts where the truth consists of a set of objective winners,
knowing a lower and upper bound on its cardinality. A prototypical problem for
this setting is the aggre-gation of multi-label annotations with prior
knowledge on the size of the ground truth. We posit noisemodels, for which we
define rules that output an optimal set of winners. We report on experiments on
multi-label annotations (which we collected).

本文研究了基于不同噪声模型的 epistemic voting 方法在多标签注释聚合方面的应用，探讨在已知胜者数量范围的情况下，如何确定最佳胜者组合，并给出了相关实验结果。