We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel
pre-training paradigm for Vision-Language Models using data from large-scale
web screenshot rendering. Using web screenshots unlocks a treasure trove of
visual and textual cues that are not present in using image-text pairs. In S4,
we leverage the inherent tree-structured hierarchy of HTML elements and the
spatial localization to carefully design 10 pre-training tasks with large scale
annotated data. These tasks resemble downstream tasks across different domains
and the annotations are cheap to obtain. We demonstrate that, compared to
current screenshot pre-training objectives, our innovative pre-training method
significantly enhances performance of image-to-text model in nine varied and
popular downstream tasks - up to 76.1% improvements on Table Detection, and at
least 1% on Widget Captioning.

提出了一种新的预训练范式 —— 基于网络截图的强监督预训练（S4），利用大规模网络截图渲染的数据进行视觉语言模型的预训练。通过使用网络截图，可以获取在图像 - 文本对中不存在的丰富的视觉和文本线索。在 S4 中，利用 HTML 元素的树状层次结构和空间定位，精心设计了 10 个具有大规模注释数据的预训练任务。这些任务类似于不同领域的下游任务，而且注释成本较低。实验证明，与当前的截图预训练目标相比，我们的创新预训练方法显著提高了图像 - 文本模型在九个多样化和热门的下游任务上的性能 —— 在表格检测上提高了 76.1%，在小部件字幕上提高了至少 1%。