An emerging family of language models (LMs), capable of processing both text
and images within a single visual view, has the promise to unlock complex tasks
such as chart understanding and UI navigation. We refer to these models as
screenshot language models. Despite their appeal, existing screenshot LMs
substantially lag behind text-only models on language understanding tasks. To
close this gap, we adopt a simplified setting where the model inputs are
plain-text-rendered screenshots, and we focus on improving the text ability of
screenshot LMs. We propose a novel Patch-and-Text Prediction (PTP) objective,
which masks and recovers both image patches of screenshots and text within
screenshots. We also conduct extensive ablation studies on masking rates and
patch sizes, as well as designs for improving training stability. Our
pre-trained model, while solely taking visual inputs, achieves comparable
performance with BERT on 6 out of 8 GLUE tasks (within 2%) and improves up to
8% over prior work. Additionally, we extend PTP to train autoregressive
screenshot LMs and demonstrate its effectiveness--our models can significantly
reduce perplexity by utilizing the screenshot context. Together, we hope our
findings can inspire future research on developing powerful screenshot LMs and
extending their reach to broader applications.

我们提出了一种新颖的 Patch-and-Text Prediction (PTP) 目标函数，通过掩盖和恢复截图中的图像块和文本，改善了截图语言模型的文本能力，并通过在 GLUE 任务上获得与 BERT 相当的性能（在 2% 之内）以及高达 8% 的改进，证明了所提出模型的有效性。