The growing interest in omnidirectional videos (ODVs) that capture the full
field-of-view (FOV) has gained 360-degree saliency prediction importance in
computer vision. However, predicting where humans look in 360-degree scenes
presents unique challenges, including spherical distortion, high resolution,
and limited labelled data. We propose a novel vision-transformer-based model
for omnidirectional videos named SalViT360 that leverages tangent image
representations. We introduce a spherical geometry-aware spatiotemporal
self-attention mechanism that is capable of effective omnidirectional video
understanding. Furthermore, we present a consistency-based unsupervised
regularization term for projection-based 360-degree dense-prediction models to
reduce artefacts in the predictions that occur after inverse projection. Our
approach is the first to employ tangent images for omnidirectional saliency
prediction, and our experimental results on three ODV saliency datasets
demonstrate its effectiveness compared to the state-of-the-art.

提出了一个基于视觉转换器的新型全景视频模型 SalViT360，利用切线图像表示，在球面几何感知的时空自注意机制方面提出了一种有效的全景视频理解方法，并引入了一种基于一致性的无监督正则化项以减少 360 度密集预测模型中逆投影之后出现的伪影，实验证明了该方法在三个全景视觉显著性数据集上的有效性超过了现有技术。