Pre-trained language models (PrLM) have to carefully manage input units when
training on a very large text with a vocabulary consisting of millions of
words. Previous works have shown that incorporating span-level information over
consecutive words in pre-training could further improve the performance of
PrLMs. However, given that span-level clues are introduced and fixed in
pre-training, previous methods are time-consuming and lack of flexibility. To
alleviate the inconvenience, this paper presents a novel span fine-tuning
method for PrLMs, which facilitates the span setting to be adaptively
determined by specific downstream tasks during the fine-tuning phase. In
detail, any sentences processed by the PrLM will be segmented into multiple
spans according to a pre-sampled dictionary. Then the segmentation information
will be sent through a hierarchical CNN module together with the representation
outputs of the PrLM and ultimately generate a span-enhanced representation.
Experiments on GLUE benchmark show that the proposed span fine-tuning method
significantly enhances the PrLM, and at the same time, offer more flexibility
in an efficient way.

本文提出了一种新颖的跨度微调算法，利用基于卷积神经网络的层次结构增强了预训练语言模型，并在 GLUE 基准下显著提升了其性能。