While humans sometimes do show the capability of correcting their own
erroneous guesses with self-critiquing, there seems to be no basis for that
assumption in the case of LLMs.

人类具备自我批评能力，但似乎在语言模型的情况下缺乏基础。

大型语言模型是否能够推理和规划？

Can Large Language Models Reason and Plan?

We present a self-supervised learning framework, COCO-LM, that pretrains
Language Models by COrrecting and COntrasting corrupted text sequences.
Following ELECTRA-style pretraining, COCO-LM employs an auxiliary language
model to corrupt text sequences, upon which it constructs two new tasks for
pretraining the main model. The first token-level task, Corrective Language
Modeling, is to detect and correct tokens replaced by the auxiliary model, in
order to better capture token-level semantics. The second sequence-level task,
Sequence Contrastive Learning, is to align text sequences originated from the
same source input while ensuring uniformity in the representation space.
Experiments on GLUE and SQuAD demonstrate that COCO-LM not only outperforms
recent state-of-the-art pretrained models in accuracy, but also improves
pretraining efficiency. It achieves the MNLI accuracy of ELECTRA with 50% of
its pretraining GPU hours. With the same pretraining steps of standard
base/large-sized models, COCO-LM outperforms the previous best models by 1+
GLUE average points.

本文提出一种自监督学习框架 COCO-LM，通过纠错和对比损坏的文本序列，对语言模型进行预训练，并在 GLUE 和 SQuAD 上的实验表明，其在准确性和预训练效率方面均优于最近的最先进预训练模型。