Textual-visual matching aims at measuring similarities between sentence
descriptions and images. Most existing methods tackle this problem without
effectively utilizing identity-level annotations. In this paper, we propose an
identity-aware two-stage framework for the textual-visual matching problem. Our
stage-1 CNN-LSTM network learns to embed cross-modal features with a novel
Cross-Modal Cross-Entropy (CMCE) loss. The stage-1 network is able to
efficiently screen easy incorrect matchings and also provide initial training
point for the stage-2 training. The stage-2 CNN-LSTM network refines the
matching results with a latent co-attention mechanism. The spatial attention
relates each word with corresponding image regions while the latent semantic
attention aligns different sentence structures to make the matching results
more robust to sentence structure variations. Extensive experiments on three
datasets with identity-level annotations show that our framework outperforms
state-of-the-art approaches by large margins.

本文提出了一个基于身份感知的两阶段框架，通过新颖的 CMCE 损失函数和潜在的共同注意机制来提高文本 - 视觉匹配的准确性，实验证明其相对于先前的最先进方法取得了显著的性能改进。