Large Language Models (LLMs) have achieved remarkable success thanks to
scalability on large text corpora, but have some drawback in training
efficiency. In contrast, Syntactic Language Models (SLMs) can be trained
efficiently to reach relatively high performance thanks to syntactic
supervision, but have trouble with scalability. Thus, given these complementary
advantages of LLMs and SLMs, it is necessary to develop an architecture that
integrates the scalability of LLMs with the training efficiency of SLMs, namely
Syntactic Large Language Models (SLLM). In this paper, we propose a novel
method dubbed tree-planting: implicitly "plant" trees into attention weights of
Transformer LMs to reflect syntactic structures of natural language.
Specifically, Transformer LMs trained with tree-planting will be called
Tree-Planted Transformers (TPT), which learn syntax on small treebanks via
tree-planting and then scale on large text corpora via continual learning with
syntactic scaffolding. Targeted syntactic evaluations on the SyntaxGym
benchmark demonstrated that TPTs, despite the lack of explicit syntactic
supervision, significantly outperformed various SLMs with explicit syntactic
supervision that generate hundreds of syntactic structures in parallel,
suggesting that tree-planting and TPTs are the promising foundation for SLLMs.

提出了一种名为 Tree-Planted Transformers（TPT）的基于树状平衡方法的革新性模型，能够在没有显式句法监督的情况下通过树状结构反映自然语言的句法结构，并在句法评估中表现出色，为 Syntactic Large Language Models（SLLMs）奠定了有前景的基础。