TACRED is one of the largest and most widely used sentence-level relation
extraction datasets. Proposed models that are evaluated using this dataset
consistently set new state-of-the-art performance. However, they still exhibit
large error rates despite leveraging external knowledge and unsupervised
pretraining on large text corpora. A recent study suggested that this may be
due to poor dataset quality. The study observed that over 50% of the most
challenging sentences from the development and test sets are incorrectly
labeled and account for an average drop of 8% f1-score in model performance.
However, this study was limited to a small biased sample of 5k (out of a total
of 106k) sentences, substantially restricting the generalizability and broader
implications of its findings. In this paper, we address these shortcomings by:
(i) performing a comprehensive study over the whole TACRED dataset, (ii)
proposing an improved crowdsourcing strategy and deploying it to re-annotate
the whole dataset, and (iii) performing a thorough analysis to understand how
correcting the TACRED annotations affects previously published results. After
verification, we observed that 23.9% of TACRED labels are incorrect. Moreover,
evaluating several models on our revised dataset yields an average f1-score
improvement of 14.3% and helps uncover significant relationships between the
different models (rather than simply offsetting or scaling their scores by a
constant factor). Finally, aside from our analysis we also release Re-TACRED, a
new completely re-annotated version of the TACRED dataset that can be used to
perform reliable evaluation of relation extraction models.

本文针对 TACRED 数据集进行了全面研究和改进，提出了改进的众包策略，通过重新注释整个数据集，发现 TACRED 标签中有 23.9% 是错误的，用改进后的数据集评估模型，平均 f1 得分提高了 14.3％，并揭示了不同模型之间的显著关系。最终，我们发布了 Re-TACRED，这是一个全新完全重新注释的 TACRED 数据集版本，可以用于可靠地评估关系提取模型。

Re-TACRED: 解决 TACRED 数据集的不足

Re-TACRED: Addressing Shortcomings of the TACRED Dataset

The current supervised relation classification (RC) task uses a single
embedding to represent the relation between a pair of entities. We argue that a
better approach is to treat the RC task as span-prediction (SP) problem,
similar to Question answering (QA). We present a span-prediction based system
for RC and evaluate its performance compared to the embedding based system. We
demonstrate that the supervised SP objective works significantly better then
the standard classification based objective. We achieve state-of-the-art
results on the TACRED and SemEval task 8 datasets.

通过将关系分类任务作为一个区间预测问题，我们提出了一个基于区间预测的关系分类系统，使用监督式的目标函数明显更优于标准的分类目标函数，并在 TACRED 和 SemEval 任务 8 的数据集上实现了最优结果。

关系分类作为双向跨度预测

Relation Classification as Two-way Span-Prediction

TACRED (Zhang et al., 2017) is one of the largest, most widely used
crowdsourced datasets in Relation Extraction (RE). But, even with recent
advances in unsupervised pre-training and knowledge enhanced neural RE, models
still show a high error rate. In this paper, we investigate the questions: Have
we reached a performance ceiling or is there still room for improvement? And
how do crowd annotations, dataset, and models contribute to this error rate? To
answer these questions, we first validate the most challenging 5K examples in
the development and test sets using trained annotators. We find that label
errors account for 8% absolute F1 test error, and that more than 50% of the
examples need to be relabeled. On the relabeled test set the average F1 score
of a large baseline model set improves from 62.1 to 70.1. After validation, we
analyze misclassifications on the challenging instances, categorize them into
linguistically motivated error groups, and verify the resulting error
hypotheses on three state-of-the-art RE models. We show that two groups of
ambiguous relations are responsible for most of the remaining errors and that
models may adopt shallow heuristics on the dataset when entities are not
masked.

本文研究了 TACRED 数据集在关系抽取中高错误率的问题，通过训练标注者重新标注，了解了错误来源，发现部分标注错误和数据集中存在的歧义关系导致错误率高，对此提出对应解决方案，并验证了三种最先进的 RE 模型的错误假设。