The current state-of-the-art image-sentence retrieval methods implicitly
align the visual-textual fragments, like regions in images and words in
sentences, and adopt attention modules to highlight the relevance of
cross-modal semantic correspondences. However, the retrieval performance
remains unsatisfactory due to a lack of consistent representation in both
semantics and structural spaces. In this work, we propose to address the above
issue from two aspects: (i) constructing intrinsic structure (along with
relations) among the fragments of respective modalities, e.g., "dog $\to$ play
$\to$ ball" in semantic structure for an image, and (ii) seeking explicit
inter-modal structural and semantic correspondence between the visual and
textual modalities. In this paper, we propose a novel Structured Multi-modal
Feature Embedding and Alignment (SMFEA) model for image-sentence retrieval. In
order to jointly and explicitly learn the visual-textual embedding and the
cross-modal alignment, SMFEA creates a novel multi-modal structured module with
a shared context-aware referral tree. In particular, the relations of the
visual and textual fragments are modeled by constructing Visual Context-aware
Structured Tree encoder (VCS-Tree) and Textual Context-aware Structured Tree
encoder (TCS-Tree) with shared labels, from which visual and textual features
can be jointly learned and optimized. We utilize the multi-modal tree structure
to explicitly align the heterogeneous image-sentence data by maximizing the
semantic and structural similarity between corresponding inter-modal tree
nodes. Extensive experiments on Microsoft COCO and Flickr30K benchmarks
demonstrate the superiority of the proposed model in comparison to the
state-of-the-art methods.

本文提出了一种新的结构化多模式特征嵌入与对齐模型（SMFEA）来增强图像 - 句子检索，通过构建视觉和文本片段之间的内在结构和外在交叉模态结构和语义对应关系，使用共享的上下文感知式引用树的新型多模式结构模块联合并显性地学习视觉 - 文本嵌入和跨模态对齐，将各自模态的片段处理为结构化树编码，通过最大化相应跨模态树节点之间的语义和结构相似性来首次实现了对齐方法的交叉模态结构和语义对应关系.