In this paper we introduce ViSiL, a Video Similarity Learning architecture
that considers fine-grained Spatio-Temporal relations between pairs of videos
-- such relations are typically lost in previous video retrieval approaches
that embed the whole frame or even the whole video into a vector descriptor
before the similarity estimation. By contrast, our Convolutional Neural Network
(CNN)-based approach is trained to calculate video-to-video similarity from
refined frame-to-frame similarity matrices, so as to consider both intra- and
inter-frame relations. In the proposed method, pairwise frame similarity is
estimated by applying Tensor Dot (TD) followed by Chamfer Similarity (CS) on
regional CNN frame features - this avoids feature aggregation before the
similarity calculation between frames. Subsequently, the similarity matrix
between all video frames is fed to a four-layer CNN, and then summarized using
Chamfer Similarity (CS) into a video-to-video similarity score -- this avoids
feature aggregation before the similarity calculation between videos and
captures the temporal similarity patterns between matching frame sequences. We
train the proposed network using a triplet loss scheme and evaluate it on five
public benchmark datasets on four different video retrieval problems where we
demonstrate large improvements in comparison to the state of the art. The
implementation of ViSiL is publicly available.

本文介绍了 ViSiL，一种视频相似性学习架构，它考虑了视频对之间的细粒度时空关系，包括了相似性评估之前的整个帧或整个视频的向量描述。通过卷积神经网络，我们从提炼的帧间相似性矩阵中计算视频间相似度，以同时考虑帧内和帧间的相似性。我们的方法避免了在帧之间的相似度计算之前进行特征聚合。我们使用三元组损失方案训练提出的网络，并在五个公共基准数据集上评估它，其中涵盖了四种不同的视频检索问题，结果表明，我们的方法比现有技术取得了显着的进步。ViSiL 的实现公开可用。