Recent advances in large language models have shown that autoregressive
modeling can generate complex and novel sequences that have many real-world
applications. However, these models must generate outputs autoregressively,
which becomes time-consuming when dealing with long sequences. Hierarchical
autoregressive approaches that compress data have been proposed as a solution,
but these methods still generate outputs at the original data frequency,
resulting in slow and memory-intensive models. In this paper, we propose a
model based on the Hierarchical Recurrent Encoder Decoder (HRED) architecture.
This model independently encodes input sub-sequences without global context,
processes these sequences using a lower-frequency model, and decodes outputs at
the original data frequency. By interpreting the encoder as an implicitly
defined embedding matrix and using sampled softmax estimation, we develop a
training algorithm that can train the entire model without a high-frequency
decoder, which is the most memory and compute-intensive part of hierarchical
approaches. In a final, brief phase, we train the decoder to generate data at
the original granularity. Our algorithm significantly reduces memory
requirements for training autoregressive models and it also improves the total
training wall-clock time.

本文提出了一种基于分层循环编码器解码器结构的模型，通过采样 softmax 估计开发了一种训练算法，可以在不需要高频解码器的情况下训练整个模型，显著降低了自回归模型训练的存储需求并改进了总的训练时间。