Fine-grained supervision based on object annotations has been widely used for
vision and language pre-training (VLP). However, in real-world application
scenarios, aligned multi-modal data is usually in the image-caption format,
which only provides coarse-grained supervision. It is cost-expensive to collect
object annotations and build object annotation pre-extractor for different
scenarios. In this paper, we propose a fine-grained self-supervision signal
without object annotations from a replacement perspective. First, we propose a
homonym sentence rewriting (HSR) algorithm to provide token-level supervision.
The algorithm replaces a verb/noun/adjective/quantifier word of the caption
with its homonyms from WordNet. Correspondingly, we propose a replacement
vision-language modeling (RVLM) framework to exploit the token-level
supervision. Two replaced modeling tasks, i.e., replaced language contrastive
(RLC) and replaced language modeling (RLM), are proposed to learn the
fine-grained alignment. Extensive experiments on several downstream tasks
demonstrate the superior performance of the proposed method.

本文提出了一种无需对象注释的细粒度自我监督信号，其基于同义词句子改写（HSR）算法提供令牌级别的监督，并使用置换视觉语言建模（RVLM）框架，分别提供被替换语言对比（RLC）和被替换语言模型（RLM）两种方法来学习细粒度对齐，通过多项下游任务的广泛实验，证明了所提出方法的卓越性能。