Pixel-based language models process text rendered as images, which allows
them to handle any script, making them a promising approach to open vocabulary
language modelling. However, recent approaches use text renderers that produce
a large set of almost-equivalent input patches, which may prove sub-optimal for
downstream tasks, due to redundancy in the input representations. In this
paper, we investigate four approaches to rendering text in the PIXEL model
(Rust et al., 2023), and find that simple character bigram rendering brings
improved performance on sentence-level tasks without compromising performance
on token-level or multilingual tasks. This new rendering strategy also makes it
possible to train a more compact model with only 22M parameters that performs
on par with the original 86M parameter model. Our analyses show that character
bigram rendering leads to a consistently better model but with an anisotropic
patch embedding space, driven by a patch frequency bias, highlighting the
connections between image patch- and tokenization-based language models.

像素模型的渲染策略对语言模型的性能有重要影响，使用字符二元组渲染能够提高模型的性能，同时在句子级任务上不会降低处理标记和多语言任务的性能，还能以只有 2200 万参数的更小模型达到与原始 8600 万参数模型相当的性能，分析表明字符二元组渲染产生了更好的模型，但由于补丁频率偏差而导致异性的补丁嵌入空间，突显了图像补丁和记号化语言模型之间的联系。

像素语言模型的文本呈现策略

Text Rendering Strategies for Pixel Language Models

The digitisation of historical documents has provided historians with
unprecedented research opportunities. Yet, the conventional approach to
analysing historical documents involves converting them from images to text
using OCR, a process that overlooks the potential benefits of treating them as
images and introduces high levels of noise. To bridge this gap, we take
advantage of recent advancements in pixel-based language models trained to
reconstruct masked patches of pixels instead of predicting token distributions.
Due to the scarcity of real historical scans, we propose a novel method for
generating synthetic scans to resemble real historical documents. We then
pre-train our model, PHD, on a combination of synthetic scans and real
historical newspapers from the 1700-1900 period. Through our experiments, we
demonstrate that PHD exhibits high proficiency in reconstructing masked image
patches and provide evidence of our model's noteworthy language understanding
capabilities. Notably, we successfully apply our model to a historical QA task,
highlighting its usefulness in this domain.

利用基于像素的语言模型对历史文档进行像素遮罩重建，通过生成合成扫描以模拟真实历史文档，展示模型在像素遮罩重建和语言理解能力上的卓越效果，并成功应用于历史问答任务。