This paper considers the problem of zero-shot entity linking, in which a link
in the test time may not present in training. Following the prevailing
BERT-based research efforts, we find a simple yet effective way is to expand
the long-range sequence modeling. Unlike many previous methods, our method does
not require expensive pre-training of BERT with long position embedding.
Instead, we propose an efficient position embeddings initialization method
called Embedding-repeat, which initializes larger position embeddings based on
BERT-Base. On Wikia's zero-shot EL dataset, our method improves the SOTA from
76.06% to 79.08%, and for its long data, the corresponding improvement is from
74.57% to 82.14%. Our experiments suggest the effectiveness of long-range
sequence modeling without retraining the BERT model.

本文提出了一种不需要用长位置嵌入来预训练 BERT 的方法，通过称为 Embedding-repeat 的有效位置嵌入初始化方法扩展长程序列建模来解决零样本实体链接问题，并在 Wikia 的零样本 EL 数据集上将 SOTA 从 76.06％提高到了 79.08％，在长数据上的相应改进从 74.57％到了 82.14％。实验证明了长程序列建模的有效性。