Position embeddings, encoding the positional relationships among tokens in
text sequences, make great contributions to modeling local context features in
Transformer-based pre-trained language models. However, in Extractive Question
Answering, position embeddings trained with instances of varied context lengths
may not perform well as we expect. Since the embeddings of rear positions are
updated fewer times than the front position embeddings, the rear ones may not
be properly trained. In this paper, we propose a simple but effective strategy,
Random Padding, without any modifications to architectures of existing
pre-trained language models. We adjust the token order of input sequences when
fine-tuning, to balance the number of updating times of every position
embedding. Experiments show that Random Padding can significantly improve model
performance on the instances whose answers are located at rear positions,
especially when models are trained on short contexts but evaluated on long
contexts. Our code and data will be released for future research.

本文提出了一种简单而有效的策略 Random Padding 来平衡位置嵌入的更新次数，从而在提取式问答的模型性能上获得显著提升。在训练模型对长文本进行评估时，特别是当模型在短文本上进行训练时，Random Padding 会表现得更加优异。