We propose a visual-linguistic representation learning approach within a
self-supervised learning framework by introducing a new operation, loss, and
data augmentation strategy. First, we generate diverse features for the
image-text matching (ITM) task via soft-masking the regions in an image, which
are most relevant to a certain word in the corresponding caption, instead of
completely removing them. Since our framework relies only on image-caption
pairs with no fine-grained annotations, we identify the relevant regions to
each word by computing the word-conditional visual attention using multi-modal
encoder. Second, we encourage the model to focus more on hard but diverse
examples by proposing a focal loss for the image-text contrastive learning
(ITC) objective, which alleviates the inherent limitations of overfitting and
bias issues. Last, we perform multi-modal data augmentations for
self-supervised learning via mining various examples by masking texts and
rendering distortions on images. We show that the combination of these three
innovations is effective for learning a pretrained model, leading to
outstanding performance on multiple vision-language downstream tasks.

提出了一个自我监督学习框架中的视觉语言表示学习方法，引入了一种新的操作、损失和数据增强策略，其中将图像中最相关于对应的标题中某个单词的区域进行软掩蔽以生成多样的图像特征，然后通过多模态编码器计算出每个单词的条件视觉注意力来确定与其相关的区域，提出了一个用于图像文本对比学习（ITC）目标的焦点损失，并进行多模态数据增强以进行自我监督学习。