Traditional language models operate autoregressively, i.e., they predict one
token at a time. Rapid explosion in model sizes has resulted in high inference
times. In this work, we propose DynaMo, a suite of multi-token prediction
language models that reduce net inference times. Our models
$\textit{dynamically}$ predict multiple tokens based on their confidence in the
predicted joint probability distribution. We propose a lightweight technique to
train these models, leveraging the weights of traditional autoregressive
counterparts. Moreover, we propose novel ways to enhance the estimated joint
probability to improve text generation quality, namely co-occurrence weighted
masking and adaptive thresholding. We also propose systematic qualitative and
quantitative methods to rigorously test the quality of generated text for
non-autoregressive generation. One of the models in our suite, DynaMo-7.3B-T3,
achieves same-quality generated text as the baseline (Pythia-6.9B) while
achieving 2.57$\times$ speed-up with only 5.87% and 2.67% parameter and
training time overheads, respectively.

DynaMo 是一个多令牌预测语言模型套件，通过在预测联合概率分布的基础上动态预测多个令牌来减少净推理时间，实现了与基准 (Pythia-6.9B) 相同质量的生成文本，并获得 2.57 倍的加速，仅有 5.87% 和 2.67% 的参数和训练时间开销。

DynaMo: 动态多词采样加速语言模型推断

DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token  Sampling

Large language models such as GPT and Llama are trained with a next-token
prediction loss. In this work, we suggest that training language models to
predict multiple future tokens at once results in higher sample efficiency.
More specifically, at each position in the training corpus, we ask the model to
predict the following n tokens using n independent output heads, operating on
top of a shared model trunk. Considering multi-token prediction as an auxiliary
training task, we measure improved downstream capabilities with no overhead in
training time for both code and natural language models. The method is
increasingly useful for larger model sizes, and keeps its appeal when training
for multiple epochs. Gains are especially pronounced on generative benchmarks
like coding, where our models consistently outperform strong baselines by
several percentage points. Our 13B parameter models solves 12 % more problems
on HumanEval and 17 % more on MBPP than comparable next-token models.
Experiments on small algorithmic tasks demonstrate that multi-token prediction
is favorable for the development of induction heads and algorithmic reasoning
capabilities. As an additional benefit, models trained with 4-token prediction
are up to 3 times faster at inference, even with large batch sizes.

我们建议通过训练语言模型来预测多个未来标记，以提高样本利用效率，并对其下游能力进行改进，特别是在多词预测作为辅助训练任务时，在代码和自然语言生成模型方面获得了显著的改善。