We introduce Visual Caption Restoration (VCR), a novel vision-language task
that challenges models to accurately restore partially obscured texts using
pixel-level hints within images. This task stems from the observation that text
embedded in images is intrinsically different from common visual elements and
natural language due to the need to align the modalities of vision, text, and
text embedded in images. While numerous works have integrated text embedded in
images into visual question-answering tasks, approaches to these tasks
generally rely on optical character recognition or masked language modeling,
thus reducing the task to mainly text-based processing. However, text-based
processing becomes ineffective in VCR as accurate text restoration depends on
the combined information from provided images, context, and subtle cues from
the tiny exposed areas of masked texts. We develop a pipeline to generate
synthetic images for the VCR task using image-caption pairs, with adjustable
caption visibility to control the task difficulty. With this pipeline, we
construct a dataset for VCR called VCR-Wiki using images with captions from
Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and
hard split variants. Our results reveal that current vision language models
significantly lag behind human performance in the VCR task, and merely
fine-tuning the models on our dataset does not lead to notable improvements. We
release VCR-Wiki and the data construction code to facilitate future research.

我们介绍了一种名为 Visual Caption Restoration（VCR）的新颖视觉 - 语言任务，该任务要求模型使用图像中的像素级提示准确恢复部分被遮挡的文本。我们开发了一个流程来生成用于 VCR 任务的合成图像，并构建了一个名为 VCR-Wiki 的数据集，该数据集包含来自维基百科的图像标题对，包括在易和难两个变体中的 211 万英文实体和 34.6 万中文实体。我们的结果表明，当前的视觉语言模型在 VCR 任务中明显落后于人类表现，仅对我们的数据集进行微调并没有显著改进。我们提供了 VCR-Wiki 数据集和数据构建代码，以便促进未来的研究。