Autoregressive transformers are spectacular models for short sequences but
scale poorly to long sequences such as high-resolution images, podcasts, code,
or books. We proposed Megabyte, a multi-scale decoder architecture that enables
end-to-end differentiable modeling of sequences of over one million bytes.
Megabyte segments sequences into patches and uses a local submodel within
patches and a global model between patches. This enables sub-quadratic
self-attention, much larger feedforward layers for the same compute, and
improved parallelism during decoding -- unlocking better performance at reduced
cost for both training and generation. Extensive experiments show that Megabyte
allows byte-level models to perform competitively with subword models on long
context language modeling, achieve state-of-the-art density estimation on
ImageNet, and model audio from raw files. Together, these results establish the
viability of tokenization-free autoregressive sequence modeling at scale.

我们提出了 Megabyte 模型，它是一种多尺度解码器结构，可以对超过一百万字节的序列进行端到端的微分建模，从而使自我注意力的时间复杂度进一步缩小，大量扩展了前馈层，实现了更好的并行处理能力，更高的性能和更低的成本，证明了无标记符自回归序列建模的可行性。