Entity names play an effective role in relation extraction (RE) and often
influence model performance. As a result, the entity names in the benchmarks'
test sets significantly influence the evaluation of RE models. In this work, we
find that the standard RE benchmarks' datasets have a large portion of
incorrect entity annotations, low entity name diversity, and are prone to have
shortcuts from entity names to ground-truth relations. These issues make the
standard benchmarks far from reflecting the real-world scenarios. Hence, in
this work, we present EntRED, a challenging RE benchmark with reduced shortcuts
and higher diversity of entities. To build EntRED, we propose an end-to-end
entity replacement pipeline based on causal inference (CI): ERIC. ERIC performs
type-constrained replacements on entities to reduce the shortcuts from entity
bias to ground-truth relations. ERIC applies CI in two aspects: 1) targeting
the instances that need entity replacements, and 2) determining the candidate
entities for replacements. We apply ERIC on TACRED to produce EntRED. Our
EntRED evaluates whether the RE model can correctly extract the relations from
the text instead of relying on entity bias. Empirical results reveal that even
the strong RE model has a significant performance drop on EntRED, which
memorizes entity name patterns instead of reasoning from the textual context.
We release ERIC's source code and the EntRED benchmark at
this https URL

本研究发现，在现有的关系抽取标准基准数据集中，数据集中的实体注释存在大量错误的情况，实体命名多样性较低，并且易于从实体名字的快捷方式到基本事实关系。为此，我们提出了一个新的具有挑战性的关系抽取数据集 EntRED，提高了实体多样性，并通过引入基于因果推理（CI）的端到端实体替换管道 ERIC。我们将 ERIC 应用于 TACRED 以生成 EntRED。实验表明，即使针对强关系模型，其性能在 EntRED 上也会显着下降，该模型仅仅记忆实体名字的模式而不是从文本上下文推理。