The digitisation of historical documents has provided historians with
unprecedented research opportunities. Yet, the conventional approach to
analysing historical documents involves converting them from images to text
using OCR, a process that overlooks the potential benefits of treating them as
images and introduces high levels of noise. To bridge this gap, we take
advantage of recent advancements in pixel-based language models trained to
reconstruct masked patches of pixels instead of predicting token distributions.
Due to the scarcity of real historical scans, we propose a novel method for
generating synthetic scans to resemble real historical documents. We then
pre-train our model, PHD, on a combination of synthetic scans and real
historical newspapers from the 1700-1900 period. Through our experiments, we
demonstrate that PHD exhibits high proficiency in reconstructing masked image
patches and provide evidence of our model's noteworthy language understanding
capabilities. Notably, we successfully apply our model to a historical QA task,
highlighting its usefulness in this domain.

利用基于像素的语言模型对历史文档进行像素遮罩重建，通过生成合成扫描以模拟真实历史文档，展示模型在像素遮罩重建和语言理解能力上的卓越效果，并成功应用于历史问答任务。