In this work, we present an entity linking model which combines a Transformer
architecture with large scale pretraining from Wikipedia links. Our model
achieves the state-of-the-art on two commonly used entity linking datasets:
96.7% on CoNLL and 94.9% on TAC-KBP. We present detailed analyses to understand
what design choices are important for entity linking, including choices of
negative entity candidates, Transformer architecture, and input perturbations.
Lastly, we present promising results on more challenging settings such as
end-to-end entity linking and entity linking without in-domain training data.

结合 Transformer 架构和维基百科链接的大规模预训练，我们提出了一个实体链接模型，在 CoNLL 和 TAC-KBP 两个常用实体链接数据集上实现了最先进的 96.7% 和 94.9% 的准确率。通过负实体候选、Transformer 架构选择和数据扰动等方面进行了详细分析，最后在没有领域内训练数据的情况下，给出了更具挑战性设置方面的有希望的结果。

监督实体链接预训练策略的经验评估

Empirical Evaluation of Pretraining Strategies for Supervised Entity  Linking

Semantic Role Labeling (SRL) is believed to be a crucial step towards natural
language understanding and has been widely studied. Recent years, end-to-end
SRL with recurrent neural networks (RNN) has gained increasing attention.
However, it remains a major challenge for RNNs to handle structural information
and long range dependencies. In this paper, we present a simple and effective
architecture for SRL which aims to address these problems. Our model is based
on self-attention which can directly capture the relationships between two
tokens regardless of their distance. Our single model achieves F$_1=83.4$ on
the CoNLL-2005 shared task dataset and F$_1=82.7$ on the CoNLL-2012 shared task
dataset, which outperforms the previous state-of-the-art results by $1.8$ and
$1.0$ F$_1$ score respectively. Besides, our model is computationally
efficient, and the parsing speed is 50K tokens per second on a single Titan X
GPU.

本文提出了一种基于自我注意力机制的新型神经网络架构以实现语义角色标注，能够解决循环神经网络在处理结构信息和长距离依赖时面临的主要挑战，其在 CoNLL 2005 和 CoNLL 2012 语义角色标注竞赛中的 F1 得分分别超过了之前的最佳成绩。此外，我们的模型计算效率高，单个 Titan X GPU 可以处理 50000 个标记每秒。

带自注意力的深度语义角色标注

Deep Semantic Role Labeling with Self-Attention

Restricted non-monotonicity has been shown beneficial for the projective
arc-eager dependency parser in previous research, as posterior decisions can
repair mistakes made in previous states due to the lack of information. In this
paper, we propose a novel, fully non-monotonic transition system based on the
non-projective Covington algorithm. As a non-monotonic system requires
exploration of erroneous actions during the training process, we develop
several non-monotonic variants of the recently defined dynamic oracle for the
Covington parser, based on tight approximations of the loss. Experiments on
datasets from the CoNLL-X and CoNLL-XI shared tasks show that a non-monotonic
dynamic oracle outperforms the monotonic version in the majority of languages.

本文提出了一种基于 Covington 算法的全非单调转移系统，并开发了几种非单调变体的动态预测器，从而探索训练过程中的错误操作。在 CoNLL-X 和 CoNLL-XI 共享任务的数据集上进行实验，实验结果显示，非单调的动态预测器在大部分语言中表现优于单调版本。