We tackle the problem of understanding visual ads where given an ad image,
our goal is to rank appropriate human generated statements describing the
purpose of the ad. This problem is generally addressed by jointly embedding
images and candidate statements to establish correspondence. Decoding a visual
ad requires inference of both semantic and symbolic nuances referenced in an
image and prior methods may fail to capture such associations especially with
weakly annotated symbols. In order to create better embeddings, we leverage an
attention mechanism to associate image proposals with symbols and thus
effectively aggregate information from aligned multimodal representations. We
propose a multihop co-attention mechanism that iteratively refines the
attention map to ensure accurate attention estimation. Our attention based
embedding model is learned end-to-end guided by a max-margin loss function. We
show that our model outperforms other baselines on the benchmark Ad dataset and
also show qualitative results to highlight the advantages of using multihop
co-attention.

本文提出了一种注意力机制下的嵌入模型，通过利用多模态表示来关联图像提议与符号并从中有效地汇集信息。我们应用多次多模态注意力机制来迭代改进注意力映射，从而提高精确度，并采用最大化边缘损失函数进行端到端学习，证明该模型在广告数据集上的效果优于其他基线模型。