Autoregressive models are typically applied to sequences of discrete tokens, but recent research indicates that generating sequences of continuous embeddings in an autoregressive manner is also feasible. However, such Continuous Autoregressive Models (CAMs) can suffer from a decline in generation quality over extended sequences due to error accumulation during inference. We introduce a novel method to address this issue by injecting random noise into the input embeddings during training. This procedure makes the model robust against varying error levels at inference. We further reduce error accumulation through an inference procedure that introduces low-level noise. Experiments on musical audio generation show that CAM substantially outperforms existing autoregressive and non-autoregressive approaches while preserving audio quality over extended sequences. This work paves the way for generating continuous embeddings in a purely autoregressive setting, opening new possibilities for real-time and interactive generative applications.

本研究解决了连续自回归模型（CAMs）在生成长序列时因推断过程中的误差积累而导致生成质量下降的问题。通过在训练过程中向输入嵌入注入随机噪声，提出了一种新的方法，使模型在推断时能够适应不同的误差水平，并在音乐音频生成实验中显示出显著的性能提升。此工作为在纯自回归环境中生成连续嵌入铺平了道路，为实时和交互式生成应用打开了新的可能性。

带有噪声增强的连续自回归模型避免误差积累