A major challenge in matching images and text is that they have intrinsically
different data distributions and feature representations. Most existing
approaches are based either on embedding or classification, the first one
mapping image and text instances into a common embedding space for distance
measuring, and the second one regarding image-text matching as a binary
classification problem. Neither of these approaches can, however, balance the
matching accuracy and model complexity well. We propose a novel framework that
achieves remarkable matching performance with acceptable model complexity.
Specifically, in the training stage, we propose a novel Multi-modal Tensor
Fusion Network (MTFN) to explicitly learn an accurate image-text similarity
function with rank-based tensor fusion rather than seeking a common embedding
space for each image-text instance. Then, during testing, we deploy a generic
Cross-modal Re-ranking (RR) scheme for refinement without requiring additional
training procedure. Extensive experiments on two datasets demonstrate that our
MTFN-RR consistently achieves the state-of-the-art matching performance with
much less time complexity. The implementation code is available at
this https URL

我们提出了一个新的框架，使用基于排名的张量融合来显式学习准确的图像文本相似度函数，通过通用的跨模态重新排序方案实现了模型效果和模型复杂性之间的平衡，从而达到了优秀的匹配性能。