Supervised or weakly supervised methods for phrase localization (textual
grounding) either rely on human annotations or some other supervised models,
e.g., object detectors. Obtaining these annotations is labor-intensive and may
be difficult to scale in practice. We propose to leverage recent advances in
contrastive language-vision models, CLIP, pre-trained on image and caption
pairs collected from the internet. In its original form, CLIP only outputs an
image-level embedding without any spatial resolution. We adapt CLIP to generate
high-resolution spatial feature maps. Importantly, we can extract feature maps
from both ViT and ResNet CLIP model while maintaining the semantic properties
of an image embedding. This provides a natural framework for phrase
localization. Our method for phrase localization requires no human annotations
or additional training. Extensive experiments show that our method outperforms
existing no-training methods in zero-shot phrase localization, and in some
cases, it even outperforms supervised methods. Code is available at
this https URL .

利用对比语言 - 视觉模型 CLIP，我们可以实现无需人工注释或额外训练的短语定位方法，其零样本短语定位性能优于现有无训练方法，并在某些情况下甚至超过了有监督的方法。

无需进一步训练即可将 CLIP 用于短语定位

Adapting CLIP For Phrase Localization Without Further Training

Textual grounding is an important but challenging task for human-computer
interaction, robotics and knowledge mining. Existing algorithms generally
formulate the task as selection from a set of bounding box proposals obtained
from deep net based systems. In this work, we demonstrate that we can cast the
problem of textual grounding into a unified framework that permits efficient
search over all possible bounding boxes. Hence, the method is able to consider
significantly more proposals and doesn't rely on a successful first stage
hypothesizing bounding box proposals. Beyond, we demonstrate that the trained
parameters of our model can be used as word-embeddings which capture
spatial-image relationships and provide interpretability. Lastly, at the time
of submission, our approach outperformed the current state-of-the-art methods
on the Flickr 30k Entities and the ReferItGame dataset by 3.08% and 7.77%
respectively.

本文研究了文本 grounding 的问题，提出了一种统一的框架来有效地搜索所有可能的 bounding box 提议，通过训练的模型参数作为 word-embedding，实现了空间 - 图像关系的捕捉和解释性，实验表明本方法在 Flickr 30k Entities 和 ReferItGame 数据集上的表现要好于当前最先进的方法。

可解释的、全局最优的基于图像概念的文本 grounding 预测

Interpretable and Globally Optimal Prediction for Textual Grounding  using Image Concepts

Textual grounding, i.e., linking words to objects in images, is a challenging
but important task for robotics and human-computer interaction. Existing
techniques benefit from recent progress in deep learning and generally
formulate the task as a supervised learning problem, selecting a bounding box
from a set of possible options. To train these deep net based approaches,
access to a large-scale datasets is required, however, constructing such a
dataset is time-consuming and expensive. Therefore, we develop a completely
unsupervised mechanism for textual grounding using hypothesis testing as a
mechanism to link words to detected image concepts. We demonstrate our approach
on the ReferIt Game dataset and the Flickr30k data, outperforming baselines by
7.98% and 6.96% respectively.

本研究提出了一种基于深度学习的无监督的文本 grounding（文本与图像上物体的对应关系）方法，并在 ReferIt Game 数据集与 Flickr30k 数据集上分别超过了基线 7.98% 和 6.96%。