Large-scale vision-language pre-training has shown impressive advances in a
wide range of downstream tasks. Existing methods mainly model the cross-modal
alignment by the similarity of the global representations of images and texts,
or advanced cross-modal attention upon image and text features. However, they
fail to explicitly learn the fine-grained semantic alignment between visual
regions and textual phrases, as only global image-text alignment information is
available. In this paper, we introduce LOUPE, a fine-grained semantically
aLigned visiOn-langUage PrE-training framework, which learns fine-grained
semantic alignment from the novel perspective of game-theoretic interactions.
To efficiently compute the game-theoretic interactions, we further propose an
uncertainty-aware neural Shapley interaction learning module. Experiments show
that LOUPE achieves state-of-the-art performance on a variety of
vision-language tasks. Furthermore, without any object-level human annotations
and fine-tuning, LOUPE achieves competitive performance on object detection and
visual grounding. More importantly, LOUPE opens a new promising direction of
learning fine-grained semantics from large-scale raw image-text pairs. The
repository of this work is at this https URL

LOUPE 是一个 fine-grained semantically aligned vision-language pre-training framework，通过博弈论交互的新视角学习到精细的语义对齐，并进一步提出了一个基于 Uncertainty-Aware Shapley interaction 的神经网络模块来高效地计算博弈论交互。实验表明， LOUPE 在各种视觉 - 语言任务上均达到了最先进的性能。此外， LOUPE 实现了无需任何对象级人员注释和微调的目标检测和视觉定位的竞争性能，同时也开启了一个新的方向，即从大规模原始图像文本数据中学习精细语义。