Tokenization is widely used in large language models because it significantly
improves performance. However, tokenization imposes several disadvantages, such
as performance biases, increased adversarial vulnerability, decreased
character-level modeling performance, and increased modeling complexity. To
address these disadvantages without sacrificing performance, we propose
SpaceByte, a novel byte-level decoder architecture that closes the performance
gap between byte-level and subword autoregressive language modeling. SpaceByte
consists of a byte-level Transformer model, but with extra larger transformer
blocks inserted in the middle of the layers. We find that performance is
significantly improved by applying these larger blocks only after certain
bytes, such as space characters, which typically denote word boundaries. Our
experiments show that for a fixed training and inference compute budget,
SpaceByte outperforms other byte-level architectures and roughly matches the
performance of tokenized Transformer architectures.

提出了一种新颖的字节级解码器架构 SpaceByte，通过在层次结构中插入更大的 Transformer 块对字节级别和子词级别语言模型建模的性能差距进行优化，通过在特定的字节后插入这些更大块，如空格字符，来提高性能。实验结果显示，在固定的计算资源下，SpaceByte 的性能优于其他字节级架构，大致与分词的 Transformer 架构相匹配。

SpaceByte：大规模语言模型中消除分词的研究

SpaceByte: Towards Deleting Tokenization from Large Language Modeling

Encoder-decoder architecture is widely adopted for sequence-to-sequence
modeling tasks. For machine translation, despite the evolution from long
short-term memory networks to Transformer networks, plus the introduction and
development of attention mechanism, encoder-decoder is still the de facto
neural network architecture for state-of-the-art models. While the motivation
for decoding information from some hidden space is straightforward, the strict
separation of the encoding and decoding steps into an encoder and a decoder in
the model architecture is not necessarily a must. Compared to the task of
autoregressive language modeling in the target language, machine translation
simply has an additional source sentence as context. Given the fact that neural
language models nowadays can already handle rather long contexts in the target
language, it is natural to ask whether simply concatenating the source and
target sentences and training a language model to do translation would work. In
this work, we investigate the aforementioned concept for machine translation.
Specifically, we experiment with bilingual translation, translation with
additional target monolingual data, and multilingual translation. In all cases,
this alternative approach performs on par with the baseline encoder-decoder
Transformer, suggesting that an encoder-decoder architecture might be redundant
for neural machine translation.

本研究探讨了将源文本和目标文本直接拼接并训练语言模型进行翻译的想法，通过对双语翻译、额外目标语单语数据翻译和多语言翻译的实验，结果表明这种替代方法与基线中的编码器 - 解码器 Transformer 模型表现相当，表明编码器 - 解码器架构对于神经机器翻译可能是多余的。