Recent advancements in video saliency prediction (VSP) have shown promising
performance compared to the human visual system, whose emulation is the primary
goal of VSP. However, current state-of-the-art models employ spatio-temporal
transformers trained on limited amounts of data, hindering generalizability
adaptation to downstream tasks. The benefits of vision foundation models
present a potential solution to improve the VSP process. However, adapting
image foundation models to the video domain presents significant challenges in
modeling scene dynamics and capturing temporal information. To address these
challenges, and as the first initiative to design a VSP model based on video
foundation models, we introduce SalFoM, a novel encoder-decoder video
transformer architecture. Our model employs UnMasked Teacher (UMT) as feature
extractor and presents a heterogeneous decoder which features a locality-aware
spatio-temporal transformer and integrates local and global spatio-temporal
information from various perspectives to produce the final saliency map. Our
qualitative and quantitative experiments on the challenging VSP benchmark
datasets of DHF1K, Hollywood-2 and UCF-Sports demonstrate the superiority of
our proposed model in comparison with the state-of-the-art methods.

使用视频基础模型，我们引入了 SalFoM，一种新颖的编码器 - 解码器视频 Transformer 架构，通过特征提取器 UnMasked Teacher（UMT）和包含局部感知的时空 Transformer 的异构解码器，从多个角度融合局部和全局时空信息以生成最终的显著性图，并在 DHF1K、Hollywood-2 和 UCF-Sports 这些具有挑战性的 VSP 基准数据集上进行定性和定量实验证明了我们提出的模型相对于先进方法的优越性。