In this paper, we propose an Omni-perception Pre-Trainer (OPT) for
cross-modal understanding and generation, by jointly modeling visual, text and
audio resources. OPT is constructed in an encoder-decoder framework, including
three single-modal encoders to generate token-based embeddings for each
modality, a cross-modal encoder to encode the correlations among the three
modalities, and two cross-modal decoders to generate text and image
respectively. For the OPT's pre-training, we design a multi-task pretext
learning scheme to model multi-modal resources from three different data
granularities, \ie, token-, modality-, and sample-level modeling, through which
OPT learns to align and translate among different modalities. The pre-training
task is carried out on a large amount of image-text-audio triplets from Open
Images. Experimental results show that OPT can learn strong image-text-audio
multi-modal representations and achieve promising results on a variety of
cross-modal understanding and generation tasks.

本文提出了一种跨模态的全视觉感知预训练器，其采用了多任务预训练策略从不同数据粒度学习了对图片、文字和音频的跨模态理解与生成。