Large-scale Vision-and-Language (V+L) pre-training for representation
learning has proven to be effective in boosting various downstream V+L tasks.
However, when it comes to the fashion domain, existing V+L methods are
inadequate as they overlook the unique characteristics of both the fashion V+L
data and downstream tasks. In this work, we propose a novel fashion-focused V+L
representation learning framework, dubbed as FashionViL. It contains two novel
fashion-specific pre-training tasks designed particularly to exploit two
intrinsic attributes with fashion V+L data. First, in contrast to other domains
where a V+L data point contains only a single image-text pair, there could be
multiple images in the fashion domain. We thus propose a Multi-View Contrastive
Learning task for pulling closer the visual representation of one image to the
compositional multimodal representation of another image+text. Second, fashion
text (e.g., product description) often contains rich fine-grained concepts
(attributes/noun phrases). To exploit this, a Pseudo-Attributes Classification
task is introduced to encourage the learned unimodal (visual/textual)
representations of the same concept to be adjacent. Further, fashion V+L tasks
uniquely include ones that do not conform to the common one-stream or
two-stream architectures (e.g., text-guided image retrieval). We thus propose a
flexible, versatile V+L model architecture consisting of a modality-agnostic
Transformer so that it can be flexibly adapted to any downstream tasks.
Extensive experiments show that our FashionViL achieves a new state of the art
across five downstream tasks. Code is available at
this https URL

本文提出了 FashionViL，一个针对时尚领域的视觉语言（V+L）表征学习框架，包含两个周到设计的预训练任务：多视角对比学习和伪属性分类学习，以及一个基于 Transformer 的灵活多用途模型架构，将其广泛适用于各种 V+L 任务，并在 5 个下游任务上取得了最佳成果。