Span-based models are one of the most straightforward methods for named
entity recognition (NER). Existing span-based NER systems shallowly aggregate
the token representations to span representations. However, this typically
results in significant ineffectiveness for long-span entities, a coupling
between the representations of overlapping spans, and ultimately a performance
degradation. In this study, we propose DSpERT (Deep Span Encoder
Representations from Transformers), which comprises a standard Transformer and
a span Transformer. The latter uses low-layered span representations as
queries, and aggregates the token representations as keys and values, layer by
layer from bottom to top. Thus, DSpERT produces span representations of deep
semantics.
With weight initialization from pretrained language models, DSpERT achieves
performance higher than or competitive with recent state-of-the-art systems on
eight NER benchmarks. Experimental results verify the importance of the depth
for span representations, and show that DSpERT performs particularly well on
long-span entities and nested structures. Further, the deep span
representations are well structured and easily separable in the feature space.

本研究提出 DSpERT（Deep Span Encoder Representations from Transformers），使用预训练语言模型的权重初始化，利用 Transformer 架构和检索式编码在命名实体识别任务上实现了高性能，特别是在长跨度实体和嵌套结构方面表现出色。