We present an approach named JSFusion (Joint Sequence Fusion) that can
measure semantic similarity between any pairs of multimodal sequence data (e.g.
a video clip and a language sentence). Our multimodal matching network consists
of two key components. First, the Joint Semantic Tensor composes a dense
pairwise representation of two sequence data into a 3D tensor. Then, the
Convolutional Hierarchical Decoder computes their similarity score by
discovering hidden hierarchical matches between the two sequence modalities.
Both modules leverage hierarchical attention mechanisms that learn to promote
well-matched representation patterns while prune out misaligned ones in a
bottom-up manner. Although the JSFusion is a universal model to be applicable
to any multimodal sequence data, this work focuses on video-language tasks
including multimodal retrieval and video QA. We evaluate the JSFusion model in
three retrieval and VQA tasks in LSMDC, for which our model achieves the best
performance reported so far. We also perform multiple-choice and movie
retrieval tasks for the MSR-VTT dataset, on which our approach outperforms many
state-of-the-art methods.

本文提出了一种名为 JSFusion（联合序列融合）的方法，可以测量多模式序列数据（例如视频剪辑和语言句子）之间的语义相似性，并通过发现两个序列模态之间的隐藏分层匹配来计算它们的相似度得分，该模型适用于多模式检索和视频 QA 等任务，并在多个数据集上超过了现有的许多最先进的方法。