Large-scale vision-language pre-training has shown impressive advances in a
wide range of downstream tasks. Existing methods mainly model the cross-modal
alignment by the similarity of the global representations of images and texts,
or advanced cross-modal attention upon image and text features. However, they
fail to explicitly learn the fine-grained semantic alignment between visual
regions and textual phrases, as only global image-text alignment information is
available. In this paper, we introduce LOUPE, a fine-grained semantically
aLigned visiOn-langUage PrE-training framework, which learns fine-grained
semantic alignment from the novel perspective of game-theoretic interactions.
To efficiently compute the game-theoretic interactions, we further propose an
uncertainty-aware neural Shapley interaction learning module. Experiments show
that LOUPE achieves state-of-the-art performance on a variety of
vision-language tasks. Furthermore, without any object-level human annotations
and fine-tuning, LOUPE achieves competitive performance on object detection and
visual grounding. More importantly, LOUPE opens a new promising direction of
learning fine-grained semantics from large-scale raw image-text pairs. The
repository of this work is at this https URL

LOUPE 是一个 fine-grained semantically aligned vision-language pre-training framework，通过博弈论交互的新视角学习到精细的语义对齐，并进一步提出了一个基于 Uncertainty-Aware Shapley interaction 的神经网络模块来高效地计算博弈论交互。实验表明， LOUPE 在各种视觉 - 语言任务上均达到了最先进的性能。此外， LOUPE 实现了无需任何对象级人员注释和微调的目标检测和视觉定位的竞争性能，同时也开启了一个新的方向，即从大规模原始图像文本数据中学习精细语义。

细粒度语义对齐视觉 - 语言预训练

Fine-Grained Semantically Aligned Vision-Language Pre-Training

Although many methods have been proposed to enhance the transferability of
adversarial perturbations, these methods are designed in a heuristic manner,
and the essential mechanism for improving adversarial transferability is still
unclear. This paper summarizes the common mechanism shared by twelve previous
transferability-boosting methods in a unified view, i.e., these methods all
reduce game-theoretic interactions between regional adversarial perturbations.
To this end, we focus on the attacking utility of all interactions between
regional adversarial perturbations, and we first discover and prove the
negative correlation between the adversarial transferability and the attacking
utility of interactions. Based on this discovery, we theoretically prove and
empirically verify that twelve previous transferability-boosting methods all
reduce interactions between regional adversarial perturbations. More crucially,
we consider the reduction of interactions as the essential reason for the
enhancement of adversarial transferability. Furthermore, we design the
interaction loss to directly penalize interactions between regional adversarial
perturbations during attacking. Experimental results show that the interaction
loss significantly improves the transferability of adversarial perturbations.

本文通过总结 12 种先前的可传递性提高方法的共同机制，即减少区域对抗扰动之间的博弈论交互，发现其与对抗传递性的负相关关系，并证明这些方法都减少了区域对抗扰动之间的交互，设计了交互损失，通过实验结果验证其显著提高了对抗扰动的传递性。

证明十二种提升对抗性可迁移性方法共享共同机制

Proving Common Mechanisms Shared by Twelve Methods of Boosting Adversarial Transferability

This paper aims to understand and improve the utility of the dropout
operation from the perspective of game-theoretic interactions. We prove that
dropout can suppress the strength of interactions between input variables of
deep neural networks (DNNs). The theoretic proof is also verified by various
experiments. Furthermore, we find that such interactions were strongly related
to the over-fitting problem in deep learning. Thus, the utility of dropout can
be regarded as decreasing interactions to alleviate the significance of
over-fitting. Based on this understanding, we propose an interaction loss to
further improve the utility of dropout. Experimental results have shown that
the interaction loss can effectively improve the utility of dropout and boost
the performance of DNNs.

本文从博弈论的角度探讨了 Dropout 操作的效用，证明 Dropout 可以抑制深度神经网络输入变量之间的相互作用，并找到这种相互作用与过拟合问题之间的强相关性。在此基础上，提出了一种交互损失来进一步提高 Dropout 的效用，实验证明交互损失可以有效提高 Dropout 的效用并提高深度神经网络的性能。