In this paper we introduce ViSiL, a Video Similarity Learning architecture
that considers fine-grained Spatio-Temporal relations between pairs of videos
-- such relations are typically lost in previous video retrieval approaches
that embed the whole frame or even the whole video into a vector descriptor
before the similarity estimation. By contrast, our Convolutional Neural Network
(CNN)-based approach is trained to calculate video-to-video similarity from
refined frame-to-frame similarity matrices, so as to consider both intra- and
inter-frame relations. In the proposed method, pairwise frame similarity is
estimated by applying Tensor Dot (TD) followed by Chamfer Similarity (CS) on
regional CNN frame features - this avoids feature aggregation before the
similarity calculation between frames. Subsequently, the similarity matrix
between all video frames is fed to a four-layer CNN, and then summarized using
Chamfer Similarity (CS) into a video-to-video similarity score -- this avoids
feature aggregation before the similarity calculation between videos and
captures the temporal similarity patterns between matching frame sequences. We
train the proposed network using a triplet loss scheme and evaluate it on five
public benchmark datasets on four different video retrieval problems where we
demonstrate large improvements in comparison to the state of the art. The
implementation of ViSiL is publicly available.

本文介绍了 ViSiL，一种视频相似性学习架构，它考虑了视频对之间的细粒度时空关系，包括了相似性评估之前的整个帧或整个视频的向量描述。通过卷积神经网络，我们从提炼的帧间相似性矩阵中计算视频间相似度，以同时考虑帧内和帧间的相似性。我们的方法避免了在帧之间的相似度计算之前进行特征聚合。我们使用三元组损失方案训练提出的网络，并在五个公共基准数据集上评估它，其中涵盖了四种不同的视频检索问题，结果表明，我们的方法比现有技术取得了显着的进步。ViSiL 的实现公开可用。

ViSiL: 细粒度时空视频相似性学习

ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning

Food computing is playing an increasingly important role in human daily life,
and has found tremendous applications in guiding human behavior towards smart
food consumption and healthy lifestyle. An important task under the
food-computing umbrella is retrieval, which is particularly helpful for health
related applications, where we are interested in retrieving important
information about food (e.g., ingredients, nutrition, etc.). In this paper, we
investigate an open research task of cross-modal retrieval between cooking
recipes and food images, and propose a novel framework Adversarial Cross-Modal
Embedding (ACME) to resolve the cross-modal retrieval task in food domains.
Specifically, the goal is to learn a common embedding feature space between the
two modalities, in which our approach consists of several novel ideas: (i)
learning by using a new triplet loss scheme together with an effective sampling
strategy, (ii) imposing modality alignment using an adversarial learning
strategy, and (iii) imposing cross-modal translation consistency such that the
embedding of one modality is able to recover some important information of
corresponding instances in the other modality. ACME achieves the
state-of-the-art performance on the benchmark Recipe1M dataset, validating the
efficacy of the proposed technique.

研究了食品领域跨模态检索的开放性研究任务，提出了一种新的框架 Adversarial Cross-Modal Embedding（ACME）来解决这个问题，通过三元组损失和有效的采样策略学习公共嵌入特征空间，并实现模态对齐、跨模态翻译一致性等多种模型，取得了基准 Recipe1M 数据集的最先进性能。