Pixel-based language models process text rendered as images, which allows
them to handle any script, making them a promising approach to open vocabulary
language modelling. However, recent approaches use text renderers that produce
a large set of almost-equivalent input patches, which may prove sub-optimal for
downstream tasks, due to redundancy in the input representations. In this
paper, we investigate four approaches to rendering text in the PIXEL model
(Rust et al., 2023), and find that simple character bigram rendering brings
improved performance on sentence-level tasks without compromising performance
on token-level or multilingual tasks. This new rendering strategy also makes it
possible to train a more compact model with only 22M parameters that performs
on par with the original 86M parameter model. Our analyses show that character
bigram rendering leads to a consistently better model but with an anisotropic
patch embedding space, driven by a patch frequency bias, highlighting the
connections between image patch- and tokenization-based language models.

像素模型的渲染策略对语言模型的性能有重要影响，使用字符二元组渲染能够提高模型的性能，同时在句子级任务上不会降低处理标记和多语言任务的性能，还能以只有 2200 万参数的更小模型达到与原始 8600 万参数模型相当的性能，分析表明字符二元组渲染产生了更好的模型，但由于补丁频率偏差而导致异性的补丁嵌入空间，突显了图像补丁和记号化语言模型之间的联系。