Incorporating the audio stream enables Video Saliency Prediction (VSP) to
imitate the selective attention mechanism of human brain. By focusing on the
benefits of joint auditory and visual information, most VSP methods are capable
of exploiting semantic correlation between vision and audio modalities but
ignoring the negative effects due to the temporal inconsistency of audio-visual
intrinsics. Inspired by the biological inconsistency-correction within
multi-sensory information, in this study, a consistency-aware audio-visual
saliency prediction network (CASP-Net) is proposed, which takes a comprehensive
consideration of the audio-visual semantic interaction and consistent
perception. In addition a two-stream encoder for elegant association between
video frames and corresponding sound source, a novel consistency-aware
predictive coding is also designed to improve the consistency within audio and
visual representations iteratively. To further aggregate the multi-scale
audio-visual information, a saliency decoder is introduced for the final
saliency map generation. Substantial experiments demonstrate that the proposed
CASP-Net outperforms the other state-of-the-art methods on six challenging
audio-visual eye-tracking datasets. For a demo of our system please see our
project webpage.

本研究提出了一种考虑视听语义交互和一致性感知的一致性感知音视频显著性预测网络（CASP-Net）以及用于改善不一致性的新颖一致性感知预测编码，使用多尺度音视频信息，生成显著性地图。该方法在六个具有挑战性的视听跟踪数据集上性能表现优于其他最先进的方法。