Cross-modal hashing is usually regarded as an effective technique for
large-scale textual-visual cross retrieval, where data from different
modalities are mapped into a shared Hamming space for matching. Most of the
traditional textual-visual binary encoding methods only consider holistic image
representations and fail to model descriptive sentences. This renders existing
methods inappropriate to handle the rich semantics of informative cross-modal
data for quality textual-visual search tasks. To address the problem of hashing
cross-modal data with semantic-rich cues, in this paper, a novel integrated
deep architecture is developed to effectively encode the detailed semantics of
informative images and long descriptive sentences, named as Textual-Visual Deep
Binaries (TVDB). In particular, region-based convolutional networks with long
short-term memory units are introduced to fully explore image regional details
while semantic cues of sentences are modeled by a text convolutional network.
Additionally, we propose a stochastic batch-wise training routine, where
high-quality binary codes and deep encoding functions are efficiently optimized
in an alternating manner. Experiments are conducted on three multimedia
datasets, i.e. Microsoft COCO, IAPR TC-12, and INRIA Web Queries, where the
proposed TVDB model significantly outperforms state-of-the-art binary coding
methods in the task of cross-modal retrieval.

本文提出了一种名为文本视觉深度二进制编码（TVDB）的方法，该方法利用区域卷积网络和文本卷积网络来分别探索图像的区域细节和句子的语义线索，通过交替优化来高效地优化二进制编码和深层编码函数，实验证明此方法能够显著提高跨模态检索的性能。