The task of retrieving video content relevant to natural language queries
plays a critical role in effectively handling internet-scale datasets. Most of
the existing methods for this caption-to-video retrieval problem do not fully
exploit cross-modal cues present in video. Furthermore, they aggregate
per-frame visual features with limited or no temporal information. In this
paper, we present a multi-modal transformer to jointly encode the different
modalities in video, which allows each of them to attend to the others. The
transformer architecture is also leveraged to encode and model the temporal
information. On the natural language side, we investigate the best practices to
jointly optimize the language embedding together with the multi-modal
transformer. This novel framework allows us to establish state-of-the-art
results for video retrieval on three datasets. More details are available at
this http URL

本文提出了一种基于多模态 transformer 架构的视频检索方法，该方法能够充分利用视频中的跨模态线索，并融合先前的时间信息。我们还研究了联合优化语言嵌入和多模态 transformer 的最佳实践。该方法在三个数据集上取得了最新的视频检索结果。