We introduce Visual Caption Restoration (VCR), a novel vision-language task
that challenges models to accurately restore partially obscured texts using
pixel-level hints within images. This task stems from the observation that text
embedded in images is intrinsically different from common visual elements and
natural language due to the need to align the modalities of vision, text, and
text embedded in images. While numerous works have integrated text embedded in
images into visual question-answering tasks, approaches to these tasks
generally rely on optical character recognition or masked language modeling,
thus reducing the task to mainly text-based processing. However, text-based
processing becomes ineffective in VCR as accurate text restoration depends on
the combined information from provided images, context, and subtle cues from
the tiny exposed areas of masked texts. We develop a pipeline to generate
synthetic images for the VCR task using image-caption pairs, with adjustable
caption visibility to control the task difficulty. With this pipeline, we
construct a dataset for VCR called VCR-Wiki using images with captions from
Wikipedia, comprising 2.11M English and 346K Chinese entities in both easy and
hard split variants. Our results reveal that current vision language models
significantly lag behind human performance in the VCR task, and merely
fine-tuning the models on our dataset does not lead to notable improvements. We
release VCR-Wiki and the data construction code to facilitate future research.

我们介绍了一种名为 Visual Caption Restoration（VCR）的新颖视觉 - 语言任务，该任务要求模型使用图像中的像素级提示准确恢复部分被遮挡的文本。我们开发了一个流程来生成用于 VCR 任务的合成图像，并构建了一个名为 VCR-Wiki 的数据集，该数据集包含来自维基百科的图像标题对，包括在易和难两个变体中的 211 万英文实体和 34.6 万中文实体。我们的结果表明，当前的视觉语言模型在 VCR 任务中明显落后于人类表现，仅对我们的数据集进行微调并没有显著改进。我们提供了 VCR-Wiki 数据集和数据构建代码，以便促进未来的研究。

视觉字幕恢复

VCR: Visual Caption Restoration

Ancient history relies on disciplines such as epigraphy, the study of ancient
inscribed texts, for evidence of the recorded past. However, these texts,
"inscriptions", are often damaged over the centuries, and illegible parts of
the text must be restored by specialists, known as epigraphists. This work
presents Pythia, the first ancient text restoration model that recovers missing
characters from a damaged text input using deep neural networks. Its
architecture is carefully designed to handle long-term context information, and
deal efficiently with missing or corrupted character and word representations.
To train it, we wrote a non-trivial pipeline to convert PHI, the largest
digital corpus of ancient Greek inscriptions, to machine actionable text, which
we call PHI-ML. On PHI-ML, Pythia's predictions achieve a 30.1% character error
rate, compared to the 57.3% of human epigraphists. Moreover, in 73.5% of cases
the ground-truth sequence was among the Top-20 hypotheses of Pythia, which
effectively demonstrates the impact of this assistive method on the field of
digital epigraphy, and sets the state-of-the-art in ancient text restoration.

这篇论文介绍了 Pythia，它是第一个使用深度神经网络从损坏的文本中恢复缺失字符的古代文本修复模型，在 PHI-ML 上实现了 30.1% 的字符错误率并在古代文字修复领域达到了最新水平。