Currently, in the field of video-text retrieval, there are many
transformer-based methods. Most of them usually stack frame features and
regrade frames as tokens, then use transformers for video temporal modeling.
However, they commonly neglect the inferior ability of the transformer modeling
local temporal information. To tackle this problem, we propose a transformer
variant named Multi-Scale Temporal Difference Transformer (MSTDT). MSTDT mainly
addresses the defects of the traditional transformer which has limited ability
to capture local temporal information. Besides, in order to better model the
detailed dynamic information, we make use of the difference feature between
frames, which practically reflects the dynamic movement of a video. We extract
the inter-frame difference feature and integrate the difference and frame
feature by the multi-scale temporal transformer. In general, our proposed MSTDT
consists of a short-term multi-scale temporal difference transformer and a
long-term temporal transformer. The former focuses on modeling local temporal
information, the latter aims at modeling global temporal information. At last,
we propose a new loss to narrow the distance of similar samples. Extensive
experiments show that backbone, such as CLIP, with MSTDT has attained a new
state-of-the-art result.

在视频文本检索领域，我们提出了一种名为 MSTDT 的变种 Transformer，主要解决了传统 Transformer 对于捕捉本地时间信息能力有限的问题，并通过多尺度时间 Transformer 来提取帧差异特征和整合差异和帧特征来更好地建模动态信息，实现了与 CLIP 等主干网络的全新最先进结果。