In this paper, we address the text and image matching in cross-modal
retrieval of the fashion industry. Different from the matching in the general
domain, the fashion matching is required to pay much more attention to the
fine-grained information in the fashion images and texts. Pioneer approaches
detect the region of interests (i.e., RoIs) from images and use the RoI
embeddings as image representations. In general, RoIs tend to represent the
"object-level" information in the fashion images, while fashion texts are prone
to describe more detailed information, e.g. styles, attributes. RoIs are thus
not fine-grained enough for fashion text and image matching. To this end, we
propose FashionBERT, which leverages patches as image features. With the
pre-trained BERT model as the backbone network, FashionBERT learns high level
representations of texts and images. Meanwhile, we propose an adaptive loss to
trade off multitask learning in the FashionBERT modeling. Two tasks (i.e., text
and image matching and cross-modal retrieval) are incorporated to evaluate
FashionBERT. On the public dataset, experiments demonstrate FashionBERT
achieves significant improvements in performances than the baseline and
state-of-the-art approaches. In practice, FashionBERT is applied in a concrete
cross-modal retrieval application. We provide the detailed matching performance
and inference efficiency analysis.

本研究提出了一种 FashionBERT 模型，利用预训练的 BERT 模型作为骨干网络，学习文字和图像的高级表示，在时尚文本和图像匹配中取得了显著的性能提升。