Image-text matching is an interesting and fascinating task in modern AI
research. Despite the evolution of deep-learning-based image and text
processing systems, multi-modal matching remains a challenging problem. In this
work, we consider the problem of accurate image-text matching for the task of
multi-modal large-scale information retrieval. State-of-the-art results in
image-text matching are achieved by inter-playing image and text features from
the two different processing pipelines, usually using mutual attention
mechanisms. However, this invalidates any chance to extract separate visual and
textual features needed for later indexing steps in large-scale retrieval
systems. In this regard, we introduce the Transformer Encoder Reasoning Network
(TERN), an architecture built upon one of the modern relationship-aware
self-attentive architectures, the Transformer Encoder (TE). This architecture
is able to separately reason on the two different modalities and to enforce a
final common abstract concept space by sharing the weights of the deeper
transformer layers. Thanks to this design, the implemented network is able to
produce compact and very rich visual and textual features available for the
successive indexing step. Experiments are conducted on the MS-COCO dataset, and
we evaluate the results using a discounted cumulative gain metric with
relevance computed exploiting caption similarities, in order to assess possibly
non-exact but relevant search results. We demonstrate that on this metric we
are able to achieve state-of-the-art results in the image retrieval task. Our
code is freely available at this https URL

本文介绍了使用 Transformer Encoder Reasoning Network 架构来进行图像与文本的自动匹配，以便于实现大规模信息检索。试验结果表明，本架构成功实现了该任务且获得了创新性的成果。

图像文本匹配和检索的 Transformer 推理网络

Transformer Reasoning Network for Image-Text Matching and Retrieval

Hashing is promising for large-scale information retrieval tasks thanks to
the efficiency of distance evaluation between binary codes. Generative hashing
is often used to generate hashing codes in an unsupervised way. However,
existing generative hashing methods only considered the use of simple priors,
like Gaussian and Bernoulli priors, which limits these methods to further
improve their performance. In this paper, two mixture-prior generative models
are proposed, under the objective to produce high-quality hashing codes for
documents. Specifically, a Gaussian mixture prior is first imposed onto the
variational auto-encoder (VAE), followed by a separate step to cast the
continuous latent representation of VAE into binary code. To avoid the
performance loss caused by the separate casting, a model using a Bernoulli
mixture prior is further developed, in which an end-to-end training is admitted
by resorting to the straight-through (ST) discrete gradient estimator.
Experimental results on several benchmark datasets demonstrate that the
proposed methods, especially the one using Bernoulli mixture priors,
consistently outperform existing ones by a substantial margin.

本文提出了两种混合先验生成模型以产生高质量文件哈希码，实验结果显示这些方法在几个基准数据集上比现有方法表现出更好的性能，特别是使用 Bernoulli 混合先验的方法。