Nowadays, customer's demands for E-commerce are more diversified, which
introduces more complications to the product retrieval industry. Previous
methods are either subject to single-modal input or perform supervised
image-level product retrieval, thus fail to accommodate real-life scenarios
where enormous weakly annotated multi-modal data are present. In this paper, we
investigate a more realistic setting that aims to perform weakly-supervised
multi-modal instance-level product retrieval among fine-grained product
categories. To promote the study of this challenging task, we contribute
Product1M, one of the largest multi-modal cosmetic datasets for real-world
instance-level retrieval. Notably, Product1M contains over 1 million
image-caption pairs and consists of two sample types, i.e., single-product and
multi-product samples, which encompass a wide variety of cosmetics brands. In
addition to the great diversity, Product1M enjoys several appealing
characteristics including fine-grained categories, complex combinations, and
fuzzy correspondence that well mimic the real-world scenes. Moreover, we
propose a novel model named Cross-modal contrAstive Product Transformer for
instance-level prodUct REtrieval (CAPTURE), that excels in capturing the
potential synergy between multi-modal inputs via a hybrid-stream transformer in
a self-supervised manner.CAPTURE generates discriminative instance features via
masked multi-modal learning as well as cross-modal contrastive pretraining and
it outperforms several SOTA cross-modal baselines. Extensive ablation studies
well demonstrate the effectiveness and the generalization capacity of our
model. Dataset and codes are available at https:
//github.com/zhanxlin/Product1M.

这篇论文针对电子商务中存在的多样化需求和多模态数据问题，提出了一种基于实例级的、弱监督、跨模态的商品检索方法，在构建一个包含千万张图像 - 标题对的大型数据集 Product1M 的同时，提出了一个用于商品实例级检索的新型模型 ——CAPTURE，该模型通过多模态学习和交叉模态对比预训练来捕获多模态输入的潜在协同作用，并生成判别性特征，比多个基线模型表现更佳。