By training to predict the next token in an unlabeled corpus, large language
models learn to perform many tasks without any labeled data. However, their
next-token-prediction objective arguably limits their performance in scenarios
that require planning, such as writing a coherent article. In this paper, we
train a module for planning the future writing process via a self-supervised
learning objective. By conditioning on generated latent plans, our model
extends the successful language model formula to more abstract planning in an
unsupervised way. Empirically, we demonstrate that our method improves language
modeling performance in general, particularly with respect to the text
structure. Because our framework uses a planner module that is unsupervised and
external to the language model, new planner modules can be trained at large
scale and easily be shared with the community.

通过自我监督学习目标，我们训练了一个用于规划未来写作流程的模块，通过生成的潜在计划的条件，在无监督的方式下扩展了成功的语言模型公式，提升了语言建模性能，特别是在文本结构方面。

从无标签数据中学习语言建模的规划

Learning to Plan for Language Modeling from Unlabeled Data

In this study, Nobel Laureate Orhan Pamuk's works are chosen as examples of
Turkish literature. By counting the number of letters and words in his texts,
we find it possible to study his works statistically. It has been known that
there is a geometrical order in text structures. Here the method based on the
basic assumption of fractal geometry is introduced for calculating the fractal
dimensions of Pamuk's texts. The results are compared with the applications of
Zipf's law, which is successfully applied for letters and words, where two
concepts, namely Zipf's dimension and Zipf's order, are introduced. The Zipf
dimension of the novel My Name is Red is found to be much different than his
other novels. However, it is linguistically observed that there is no
fundamental difference between his corpora. The results are interpreted in
terms of fractal dimensions and the Turkish language.

通过统计诺贝尔文学奖得主奥尔罕・帕慕克作品的字母和单词数，应用分形几何学方法计算他的文本的分形维度，与 Zipf 定律的应用进行了比较，并用统计分析方法研究其文本结构和土耳其语言的关系。

奥尔罕・帕慕克的诺贝尔作品的数学和语言特征描述

Mathematical and Linguistic Characterization of Orhan Pamuk's Nobel  Works

Today the pre-trained language models achieve great success for question
generation (QG) task and significantly outperform traditional
sequence-to-sequence approaches. However, the pre-trained models treat the
input passage as a flat sequence and are thus not aware of the text structure
of input passage. For QG task, we model text structure as answer position and
syntactic dependency, and propose answer localness modeling and syntactic mask
attention to address these limitations. Specially, we present localness
modeling with a Gaussian bias to enable the model to focus on answer-surrounded
context, and propose a mask attention mechanism to make the syntactic structure
of input passage accessible in question generation process. Experiments on
SQuAD dataset show that our proposed two modules improve performance over the
strong pre-trained model ProphetNet, and combing them together achieves very
competitive results with the state-of-the-art pre-trained model.

研究表明，采用预训练模型生成问题 (QG) 任务可以显著优于传统的序列对序列方法，但是这些模型对于输入的文章缺乏文本结构的认识。为了克服这个问题，我们将文本结构模拟为回答位置和句法依赖性，提出回答局部性建模和句法遮盖自注意力机制。实验表明，这两个模块可以改进预训练模型 ProphetNet 的性能，并将它们结合在一起可以达到现有最先进预训练模型的竞争性结果。