Image-text retrieval, as a fundamental and important branch of information
retrieval, has attracted extensive research attentions. The main challenge of
this task is cross-modal semantic understanding and matching. Some recent works
focus more on fine-grained cross-modal semantic matching. With the prevalence
of large scale multimodal pretraining models, several state-of-the-art models
(e.g. X-VLM) have achieved near-perfect performance on widely-used image-text
retrieval benchmarks, i.e. MSCOCO-Test-5K and Flickr30K-Test-1K. In this paper,
we review the two common benchmarks and observe that they are insufficient to
assess the true capability of models on fine-grained cross-modal semantic
matching. The reason is that a large amount of images and texts in the
benchmarks are coarse-grained. Based on the observation, we renovate the
coarse-grained images and texts in the old benchmarks and establish the
improved benchmarks called MSCOCO-FG and Flickr30K-FG. Specifically, on the
image side, we enlarge the original image pool by adopting more similar images.
On the text side, we propose a novel semi-automatic renovation approach to
refine coarse-grained sentences into finer-grained ones with little human
effort. Furthermore, we evaluate representative image-text retrieval models on
our new benchmarks to demonstrate the effectiveness of our method. We also
analyze the capability of models on fine-grained semantic comprehension through
extensive experiments. The results show that even the state-of-the-art models
have much room for improvement in fine-grained semantic understanding,
especially in distinguishing attributes of close objects in images. Our code
and improved benchmark datasets are publicly available at:
this https URL, which we hope will inspire
further in-depth research on cross-modal retrieval.

本文针对图文检索中的细粒度语义匹配问题，以 MSCOCO-Test-5K 和 Flickr30K-Test-1K 数据集不足的情况为背景，提出了将其重建为 MSCOCO-FG 和 Flickr30K-FG 等数据集的方法，并通过模型评估和实验指出了模型在细粒度语义理解方面的不足之处和提升空间。