The structure of causal language model training assumes that each token can be accurately predicted from the previous context. This contrasts with humans' natural writing and reasoning process, where goals are typically known before the exact argument or phrasings. While this mismatch has been well studied in the literature, the working assumption has been that architectural changes are needed to address this mismatch. We argue that rearranging and processing the training data sequences can allow models to more accurately imitate the true data-generating process, and does not require any other changes to the architecture or training infrastructure. We demonstrate that this technique, Trelawney, and the inference algorithms derived from it allow us to improve performance on several key benchmarks that span planning, algorithmic reasoning, and story generation tasks. Finally, our method naturally enables the generation of long-term goals at no additional cost. We investigate how using the model's goal-generation capability can further improve planning and reasoning. Additionally, we believe Trelawney could potentially open doors to new capabilities beyond the current language modeling paradigm.

本研究解决了当前因果语言模型训练中，标记预测与人类写作和推理过程之间的差异问题。论文提出了一种名为Trelawney的新方法，通过重新排列和处理训练数据序列，无需改动模型架构即可更准确地模拟真实数据生成过程。研究显示，该方法在多个关键基准上表现出提升，尤其在规划、算法推理和故事生成任务上，并能有效生成长期目标，从而显著增强模型的规划和推理能力。

超越下一个标记的视角