A syntactic language model (SLM) incrementally generates a sentence with its
syntactic tree in a left-to-right manner. We present Generative Pretrained
Structured Transformers (GPST), an unsupervised SLM at scale capable of being
pre-trained from scratch on raw texts with high parallelism. GPST circumvents
the limitations of previous SLMs such as relying on gold trees and sequential
training. It consists of two components, a usual SLM supervised by a
uni-directional language modeling loss, and an additional composition model,
which induces syntactic parse trees and computes constituent representations,
supervised by a bi-directional language modeling loss. We propose a
representation surrogate to enable joint parallel training of the two models in
a hard-EM fashion. We pre-train GPST on OpenWebText, a corpus with $9$ billion
tokens, and demonstrate the superiority of GPST over GPT-2 with a comparable
size in numerous tasks covering both language understanding and language
generation. Meanwhile, GPST also significantly outperforms existing
unsupervised SLMs on left-to-right grammar induction, while holding a
substantial acceleration on training.

使用大规模预训练结构转换器的句法语言模型（SLM），通过从头开始在原始文本上进行高并行度的无监督训练，弥补了以前的 SLM 存在的限制，如依赖于黄金树和顺序训练。同时，我们介绍了一个表示替代方法，以在硬 EM 方式下联合并行训练模型，证明了 Generative Pretrained Structured Transformers（GPST）在语言理解和语言生成等多个任务中相比 GPT-2 具有可比大小时的优越性，同时在左到右语法归纳方面也显著优于现有的无监督 SLM，并具有训练加速的优势。