Large pre-trained multimodal models have demonstrated significant success in
a range of downstream tasks, including image captioning, image-text retrieval,
visual question answering (VQA), etc. However, many of these methods rely on
image-text pairs collected from the web as pre-training data and unfortunately
overlook the need for fine-grained feature alignment between vision and
language modalities, which requires detailed understanding of images and
language expressions. While integrating VQA and dense captioning (DC) into
pre-training can address this issue, acquiring image-question-answer as well as
image-location-caption triplets is challenging and time-consuming.
Additionally, publicly available datasets for VQA and dense captioning are
typically limited in scale due to manual data collection and labeling efforts.
In this paper, we propose a novel method called Joint QA and DC GEneration
(JADE), which utilizes a pre-trained multimodal model and easily-crawled
image-text pairs to automatically generate and filter large-scale VQA and dense
captioning datasets. We apply this method to the Conceptual Caption (CC3M)
dataset to generate a new dataset called CC3M-QA-DC. Experiments show that when
used for pre-training in a multi-task manner, CC3M-QA-DC can improve the
performance with various backbones on various downstream tasks. Furthermore,
our generated CC3M-QA-DC can be combined with larger image-text datasets (e.g.,
CC15M) and achieve competitive results compared with models using much more
data. Code and dataset will be released.

本文提出一种名为 Joint QA and DC Generation (JADE) 的新方法，利用预训练的多模态模型及易于爬取的图像 - 文本对生成和过滤大规模的视觉问答和密集字幕数据集。我们将该方法应用于概念字幕（CC3M）数据集，生成了一个名为 CC3M-QA-DC 的新的数据集，在多任务方式预训练时，CC3M-QA-DC 可以改善各种骨干网络在各种下游任务中的性能，并与更多数据使用模型相比，我们生成的 CC3M-QA-DC 和更大的图像 - 文本数据集（例如 CC15M）相结合，在相同的计算条件下达到了有竞争力的结果。