Tokenization is widely used in large language models because it significantly
improves performance. However, tokenization imposes several disadvantages, such
as performance biases, increased adversarial vulnerability, decreased
character-level modeling performance, and increased modeling complexity. To
address these disadvantages without sacrificing performance, we propose
SpaceByte, a novel byte-level decoder architecture that closes the performance
gap between byte-level and subword autoregressive language modeling. SpaceByte
consists of a byte-level Transformer model, but with extra larger transformer
blocks inserted in the middle of the layers. We find that performance is
significantly improved by applying these larger blocks only after certain
bytes, such as space characters, which typically denote word boundaries. Our
experiments show that for a fixed training and inference compute budget,
SpaceByte outperforms other byte-level architectures and roughly matches the
performance of tokenized Transformer architectures.

提出了一种新颖的字节级解码器架构 SpaceByte，通过在层次结构中插入更大的 Transformer 块对字节级别和子词级别语言模型建模的性能差距进行优化，通过在特定的字节后插入这些更大块，如空格字符，来提高性能。实验结果显示，在固定的计算资源下，SpaceByte 的性能优于其他字节级架构，大致与分词的 Transformer 架构相匹配。