Natural language understanding tasks such as open-domain question answering
often require retrieving and assimilating factual information from multiple
sources. We propose to address this problem by integrating a semi-parametric
representation of a large text corpus into a Transformer model as a source of
factual knowledge. Specifically, our method represents knowledge with `mention
memory', a table of dense vector representations of every entity mention in a
corpus. The proposed model - TOME - is a Transformer that accesses the
information through internal memory layers in which each entity mention in the
input passage attends to the mention memory. This approach enables synthesis of
and reasoning over many disparate sources of information within a single
Transformer model. In experiments using a memory of 150 million Wikipedia
mentions, TOME achieves strong performance on several open-domain
knowledge-intensive tasks, including the claim verification benchmarks HoVer
and FEVER and several entity-based QA benchmarks. We also show that the model
learns to attend to informative mentions without any direct supervision.
Finally we demonstrate that the model can generalize to new unseen entities by
updating the memory without retraining.

该论文提出了一种利用 Transformer 模型中的半参数表示以及文本语料库的 “指向记忆” 来提取多个文本来源的事实信息的方法，称为 TOME，并在多个实验中证明其在自然语言理解任务中的优异表现。

提及记忆：通过实体提及注意力将文本知识融入 Transformers

Mention Memory: incorporating textual knowledge into Transformers  through entity mention attention

Fine-Grained Named Entity Typing (FG-NET) is a key component in Natural
Language Processing (NLP). It aims at classifying an entity mention into a wide
range of entity types. Due to a large number of entity types, distant
supervision is used to collect training data for this task, which noisily
assigns type labels to entity mentions irrespective of the context. In order to
alleviate the noisy labels, existing approaches on FGNET analyze the entity
mentions entirely independent of each other and assign type labels solely based
on mention sentence-specific context. This is inadequate for highly overlapping
and noisy type labels as it hinders information passing across sentence
boundaries. For this, we propose an edge-weighted attentive graph convolution
network that refines the noisy mention representations by attending over
corpus-level contextual clues prior to the end classification. Experimental
evaluation shows that the proposed model outperforms the existing research by a
relative score of upto 10.2% and 8.3% for macro f1 and micro f1 respectively.

本文提出了一种基于图卷积网络的方法，通过关注语料库级别的上下文提示来提炼含噪声的命名实体类型，实验证明该方法相对于现有研究提高了 10.2％和 8.3％的总体宏 / 微 F1 得分。