Next-token prediction (NTP) over large text corpora has become the go-to paradigm to train large language models. Yet, it remains unclear how NTP influences the mapping of linguistic patterns to geometric properties of the resulting model representations. We frame training of large language models as soft-label classification over sparse probabilistic label vectors, coupled with an analytical approximation that allows unrestricted generation of context embeddings. This approach links NTP training to rank-constrained, nuclear-norm regularized optimization in the logit domain, offering a framework for analyzing the geometry of word and context embeddings. In large embedding spaces, we find that NTP implicitly favors learning logits with a sparse plus low-rank structure. While the sparse component captures the co-occurrence frequency of context-word pairs, the orthogonal low-rank component, which becomes dominant as training progresses, depends solely on the sparsity pattern of the co-occurrence matrix. Consequently, when projected onto an appropriate subspace, representations of contexts that are followed by the same set of next-tokens collapse, a phenomenon we term subspace-collapse. We validate our findings on synthetic and small-scale real language datasets. Finally, we outline potential research directions aimed at deepening the understanding of NTP's influence on the learning of linguistic patterns and regularities.

本研究解决了下一标记预测（NTP）如何影响语言模式与模型表示几何属性之间映射的未知问题。通过将大型语言模型的训练框架视为稀疏概率标签向量上的软标签分类，提出了一种新颖的方法来分析单词和上下文嵌入的几何特征。研究发现，NTP隐式地促进了稀疏加低秩结构的学习，可能导致相同下一标记上下文的表示在适当子空间中聚集，这一现象称为子空间崩溃。

下一个标记预测的隐式几何：从语言稀疏模式到模型表示