Recently, masked video modeling has been widely explored and significantly
improved the model's understanding ability of visual regions at a local level.
However, existing methods usually adopt random masking and follow the same
reconstruction paradigm to complete the masked regions, which do not leverage
the correlations between cross-modal content. In this paper, we present Mask
for Semantics Completion (MASCOT) based on semantic-based masked modeling.
Specifically, after applying attention-based video masking to generate
high-informed and low-informed masks, we propose Informed Semantics Completion
to recover masked semantics information. The recovery mechanism is achieved by
aligning the masked content with the unmasked visual regions and corresponding
textual context, which makes the model capture more text-related details at a
patch level. Additionally, we shift the emphasis of reconstruction from
irrelevant backgrounds to discriminative parts to ignore regions with
low-informed masks. Furthermore, we design dual-mask co-learning to incorporate
video cues under different masks and learn more aligned video representation.
Our MASCOT performs state-of-the-art performance on four major text-video
retrieval benchmarks, including MSR-VTT, LSMDC, ActivityNet, and DiDeMo.
Extensive ablation studies demonstrate the effectiveness of the proposed
schemes.

本文提出一个基于语义蒙版建模的 Mask for Semantics Completion (MASCOT) 框架，通过基于注意力的视频蒙版生成高信息和低信息蒙版，利用蒙版内容与文本上下文对齐，实现了蒙版语义信息的恢复，并通过双蒙版协同学习来提高模型的视频表示性能，在四个主要的文本 - 视频检索基准上取得了最先进的性能。