Most language models (LMs) are trained and applied in an autoregressive
left-to-right fashion, assuming that the next token only depends on the
preceding ones. However, this assumption ignores the potential benefits of
using the full sequence information during training, and the possibility of
having context from both sides during inference. In this paper, we propose a
new pre-training paradigm with techniques that jointly improve the training
data efficiency and the capabilities of the LMs in the infilling task. The
first is a training objective that aligns the predictions of a left-to-right LM
with those of a right-to-left LM, trained on the same data but in reverse
order. The second is a bidirectional inference procedure that enables both LMs
to meet in the middle. We show the effectiveness of our pre-training paradigm
with extensive experiments on both programming and natural language models,
outperforming strong baselines.

本文提出了一种新的预训练范例，采用双向推理和训练效率优化技术，以提高语言模型在数据填充任务中的能力并展开大量编程和自然语言实验取得优异成果。

中途相遇：一种新的预训练范式

Meet in the Middle: A New Pre-training Paradigm

This paper describes an efficient approach to improve the accuracy of a named
entity recognition system for Vietnamese. The approach combines regular
expressions over tokens and a bidirectional inference method in a sequence
labelling model. The proposed method achieves an overall $F_1$ score of 89.66%
on a test set of an evaluation campaign, organized in late 2016 by the
Vietnamese Language and Speech Processing (VLSP) community.

该论文提出了一种有效的方法，通过在标记序列模型中应用标记规则和双向推理方法来提高越南语命名实体识别系统的准确性，该方法在 VLSP 社区于 2016 年底组织的一次评估活动中的测试集上取得了 89.66％的总体 F1 得分。