The growing interest in omnidirectional videos (ODVs) that capture the full
field-of-view (FOV) has gained 360-degree saliency prediction importance in
computer vision. However, predicting where humans look in 360-degree scenes
presents unique challenges, including spherical distortion, high resolution,
and limited labelled data. We propose a novel vision-transformer-based model
for omnidirectional videos named SalViT360 that leverages tangent image
representations. We introduce a spherical geometry-aware spatiotemporal
self-attention mechanism that is capable of effective omnidirectional video
understanding. Furthermore, we present a consistency-based unsupervised
regularization term for projection-based 360-degree dense-prediction models to
reduce artefacts in the predictions that occur after inverse projection. Our
approach is the first to employ tangent images for omnidirectional saliency
prediction, and our experimental results on three ODV saliency datasets
demonstrate its effectiveness compared to the state-of-the-art.

提出了一个基于视觉转换器的新型全景视频模型 SalViT360，利用切线图像表示，在球面几何感知的时空自注意机制方面提出了一种有效的全景视频理解方法，并引入了一种基于一致性的无监督正则化项以减少 360 度密集预测模型中逆投影之后出现的伪影，实验证明了该方法在三个全景视觉显著性数据集上的有效性超过了现有技术。

球形视觉 Transformer 用于 360 度视频显著性预测

Spherical Vision Transformer for 360-degree Video Saliency Prediction

360{\deg} cameras can capture complete environments in a single shot, which
makes 360{\deg} imagery alluring in many computer vision tasks. However,
monocular depth estimation remains a challenge for 360{\deg} data, particularly
for high resolutions like 2K (2048x1024) and beyond that are important for
novel-view synthesis and virtual reality applications. Current CNN-based
methods do not support such high resolutions due to limited GPU memory. In this
work, we propose a flexible framework for monocular depth estimation from
high-resolution 360{\deg} images using tangent images. We project the 360{\deg}
input image onto a set of tangent planes that produce perspective views, which
are suitable for the latest, most accurate state-of-the-art perspective
monocular depth estimators. To achieve globally consistent disparity estimates,
we recombine the individual depth estimates using deformable multi-scale
alignment followed by gradient-domain blending. The result is a dense,
high-resolution 360{\deg} depth map with a high level of detail, also for
outdoor scenes which are not supported by existing methods. Our source code and
data are available at this https URL

提出一种基于切线图像的高分辨率 360 度图像单目深度估计方法，通过将输入的 360 度图像投影到切线平面上得到透视视图，以此来支持最新的和最准确的单目深度估计器，采用可变形多尺度对齐和梯度域混合来重新组合单个深度估计，产生具有高分辨率和高细节的稠密 360 度景深图。

360MonoDepth：高分辨率 360° 单目深度估计

360MonoDepth: High-Resolution 360° Monocular Depth Estimation

In this work, we propose "tangent images," a spherical image representation
that facilitates transferable and scalable $360^\circ$ computer vision.
Inspired by techniques in cartography and computer graphics, we render a
spherical image to a set of distortion-mitigated, locally-planar image grids
tangent to a subdivided icosahedron. By varying the resolution of these grids
independently of the subdivision level, we can effectively represent high
resolution spherical images while still benefiting from the low-distortion
icosahedral spherical approximation. We show that training standard
convolutional neural networks on tangent images compares favorably to the many
specialized spherical convolutional kernels that have been developed, while
also scaling efficiently to handle significantly higher spherical resolutions.
Furthermore, because our approach does not require specialized kernels, we show
that we can transfer networks trained on perspective images to spherical data
without fine-tuning and with limited performance drop-off. Finally, we
demonstrate that tangent images can be used to improve the quality of sparse
feature detection on spherical images, illustrating its usefulness for
traditional computer vision tasks like structure-from-motion and SLAM.

本文提出了 “正切图像”，这是一种球形图像表示，可促进可转移和可扩展的 360 度计算机视觉。通过在一个细分的二十面体上渲染球形图像，我们可以将其转换为消除畸变的局部平面图像网格的集合。通过在不改变细分级别的情况下独立地改变这些网格的分辨率，我们可以有效地表示高分辨率的球形图像，同时仍然从低畸变的二十面体球面近似中获益。我们展示了训练标准卷积神经网络在正切图像上的比较优越性，同时也可有效地处理显著更高的球面分辨率并具有高的可转移性。此外，由于我们的方法不需要专门的卷积核，因此我们证明了我们可以将在透视图像上训练的网络转移到球形数据上而不需要微调，而性能下降有限。最后，我们演示了正切图像可以用于改善球形图像上的稀疏特征检测的质量，说明了其在传统计算机视觉任务中的实用性，例如运动估计和 SLAM。