Vision-and-language models trained to match images with text can be combined
with visual explanation methods to point to the locations of specific objects
in an image. Our work shows that the localization --"grounding"-- abilities of
these models can be further improved by finetuning for self-consistent visual
explanations. We propose a strategy for augmenting existing text-image datasets
with paraphrases using a large language model, and SelfEQ, a weakly-supervised
strategy on visual explanation maps for paraphrases that encourages
self-consistency. Specifically, for an input textual phrase, we attempt to
generate a paraphrase and finetune the model so that the phrase and paraphrase
map to the same region in the image. We posit that this both expands the
vocabulary that the model is able to handle, and improves the quality of the
object locations highlighted by gradient-based visual explanation methods (e.g.
GradCAM). We demonstrate that SelfEQ improves performance on Flickr30k,
ReferIt, and RefCOCO+ over a strong baseline method and several prior works.
Particularly, comparing to other methods that do not use any type of box
annotations, we obtain 84.07% on Flickr30k (an absolute improvement of 4.69%),
67.40% on ReferIt (an absolute improvement of 7.68%), and 75.10%, 55.49% on
RefCOCO+ test sets A and B respectively (an absolute improvement of 3.74% on
average).

使用视觉与语言模型、视觉解释方法和近义词进行微调，目标是提高定位能力和对象高亮质量。在多个数据集中，通过该方法相较于基线方法和之前的工作获得了显著的改进。