Unsupervised learning objectives like language modeling and de-noising
constitute a significant part in producing pre-trained models that perform
various downstream applications from natural language understanding to
conversational tasks. However, despite impressive conversational capabilities
of recent large language model, their abilities to capture syntactic or
semantic structure within text lag behind. We hypothesize that the mismatch
between linguistic performance and competence in machines is attributable to
insufficient transfer of linguistic structure knowledge to computational
systems with currently popular pre-training objectives. We show that
punctuation restoration transfers to improvements in in- and
out-of-distribution performance on structure-related tasks like named entity
recognition, open information extraction, chunking, and part-of-speech tagging.
Punctuation restoration is an effective learning objective that can improve
structure understanding and yield a more robust structure-aware representations
of natural language.

非监督学习目标如语言建模和去噪在产生预训练模型方面扮演重要角色，然而，最近的大语言模型的对话能力令人印象深刻，但它们在捕捉文本内的句法或语义结构方面滞后。我们假设这种语言性能和机器能力之间的差异是由于目前流行的预训练目标对语言结构知识的转移不足引起的。我们表明，标点恢复可以提高与结构相关的任务的内、外分布性能，如命名实体识别、开放信息提取、块划分和词性标注。标点恢复是一种有效的学习目标，可以改善结构理解并产生更健壮的结构感知的自然语言表示。