Vision-language models (VLMs) have made significant strides in cross-modal
understanding through large-scale paired datasets. However, in fashion domain,
datasets often exhibit a disparity between the information conveyed in image
and text. This issue stems from datasets containing multiple images of a single
fashion item all paired with one text, leading to cases where some textual
details are not visible in individual images. This mismatch, particularly when
non-co-occurring elements are masked, undermines the training of conventional
VLM objectives like Masked Language Modeling and Masked Image Modeling, thereby
hindering the model's ability to accurately align fine-grained visual and
textual features. Addressing this problem, we propose Synchronized attentional
Masking (SyncMask), which generate masks that pinpoint the image patches and
word tokens where the information co-occur in both image and text. This
synchronization is accomplished by harnessing cross-attentional features
obtained from a momentum model, ensuring a precise alignment between the two
modalities. Additionally, we enhance grouped batch sampling with semi-hard
negatives, effectively mitigating false negative issues in Image-Text Matching
and Image-Text Contrastive learning objectives within fashion datasets. Our
experiments demonstrate the effectiveness of the proposed approach,
outperforming existing methods in three downstream tasks.

通过生成准确定位信息在图像和文本中同时出现的图像块和单词标记的掩码，SyncMask 解决了时尚数据集中图像和文本之间信息不匹配的问题，并在时尚数据集中的三个下游任务中表现出优秀的性能。