Token-free language models learn directly from raw bytes and remove the bias
of subword tokenization. Operating on bytes, however, results in significantly
longer sequences, and standard autoregressive Transformers scale poorly in such
settings. We experiment with MambaByte, a token-free adaptation of the Mamba
state space model, trained autoregressively on byte sequences. Our experiments
indicate the computational efficiency of MambaByte compared to other byte-level
models. We also find MambaByte to be competitive with and even outperform
state-of-the-art subword Transformers. Furthermore, owing to linear scaling in
length, MambaByte benefits from fast inference compared to Transformers. Our
findings establish the viability of MambaByte in enabling token-free language
modeling.

基于字节的 MambaByte 模型是一种无需标记的语言模型，它在计算效率方面表现出色，优于其他基于字节的模型，并且可以超过现有的基于子词的模型，因此可用于无需标记的语言建模。

MambaByte: 无需令牌的选择性状态空间模型

MambaByte: Token-free Selective State Space Model

Most widely-used pre-trained language models operate on sequences of tokens
corresponding to word or subword units. By comparison, token-free models that
operate directly on raw text (bytes or characters) have many benefits: they can
process text in any language out of the box, they are more robust to noise, and
they minimize technical debt by removing complex and error-prone text
preprocessing pipelines. Since byte or character sequences are longer than
token sequences, past work on token-free models has often introduced new model
architectures designed to amortize the cost of operating directly on raw text.
In this paper, we show that a standard Transformer architecture can be used
with minimal modifications to process byte sequences. We characterize the
trade-offs in terms of parameter count, training FLOPs, and inference speed,
and show that byte-level models are competitive with their token-level
counterparts. We also demonstrate that byte-level models are significantly more
robust to noise and perform better on tasks that are sensitive to spelling and
pronunciation. As part of our contribution, we release a new set of pre-trained
byte-level Transformer models based on the T5 architecture, as well as all code
and data used in our experiments.

本文研究了基于字节级别文本的 Transformer 架构，相比于基于词或子词的标准模型，该模型可处理任何语言的文本，更加稳健且精度更高，并开源了基于 T5 架构的预训练模型及所有实验所用代码和数据。