Recent self-supervised approaches have used large-scale image-text datasets
to learn powerful representations that transfer to many tasks without
finetuning. These methods often assume that there is one-to-one correspondence
between its images and their (short) captions. However, many tasks require
reasoning about multiple images and long text narratives, such as describing
news articles with visual summaries. Thus, we explore a novel setting where the
goal is to learn a self-supervised visual-language representation that is
robust to varying text length and the number of images. In addition, unlike
prior work which assumed captions have a literal relation to the image, we
assume images only contain loose illustrative correspondence with the text. To
explore this problem, we introduce a large-scale multimodal dataset containing
over 31M articles, 22M images and 1M videos. We show that state-of-the-art
image-text alignment methods are not robust to longer narratives with multiple
images. Finally, we introduce an intuitive baseline that outperforms these
methods on zero-shot image-set retrieval by 10% on the GoodNews dataset.

该研究探索了一个新的问题，即学习对不同长度和数量的图像文本具有强鲁棒性的自我监督视觉语言表示法，其介绍了一个包括超过 31M 篇文章，22M 张图片和 1M 个视频的大规模多模态数据集，并表明最先进的图像文本对齐方法不能很好地处理带有多张图像和更长篇幅的叙述，且还提出了一种直观的基线方法，在 GoodNews 数据集上零样本图像集检索表现比这些方法高出 10%。