Despite the potential of multi-modal pre-training to learn highly
discriminative feature representations from complementary data modalities,
current progress is being slowed by the lack of large-scale modality-diverse
datasets. By leveraging the natural suitability of E-commerce, where different
modalities capture complementary semantic information, we contribute a
large-scale multi-modal pre-training dataset M5Product. The dataset comprises 5
modalities (image, text, table, video, and audio), covers over 6,000 categories
and 5,000 attributes, and is 500 larger than the largest publicly available
dataset with a similar number of modalities. Furthermore, M5Product contains
incomplete modality pairs and noise while also having a long-tailed
distribution, resembling most real-world problems. We further propose
Self-harmonized ContrAstive LEarning (SCALE), a novel pretraining framework
that integrates the different modalities into a unified model through an
adaptive feature fusion mechanism, where the importance of each modality is
learned directly from the modality embeddings and impacts the inter-modality
contrastive learning and masked tasks within a multi-modal transformer model.
We evaluate the current multi-modal pre-training state-of-the-art approaches
and benchmark their ability to learn from unlabeled data when faced with the
large number of modalities in the M5Product dataset. We conduct extensive
experiments on four downstream tasks and demonstrate the superiority of our
SCALE model, providing insights into the importance of dataset scale and
diversity.

该论文提出了一种基于 E-commerce 的多模态预训练数据集 M5Product 和一种名为 SCALE 的预训练框架，实现了不同模态特征的融合和学习，并在四个下游任务中展示了其优越性，该方法具有重要的数据集规模和多样性。