Panoramic distortion poses a significant challenge in 360 depth estimation,
particularly pronounced at the north and south poles. Existing methods either
adopt a bi-projection fusion strategy to remove distortions or model long-range
dependencies to capture global structures, which can result in either unclear
structure or insufficient local perception. In this paper, we propose a
spherical geometry transformer, named SGFormer, to address the above issues,
with an innovative step to integrate spherical geometric priors into vision
transformers. To this end, we retarget the transformer decoder to a spherical
prior decoder (termed SPDecoder), which endeavors to uphold the integrity of
spherical structures during decoding. Concretely, we leverage bipolar
re-projection, circular rotation, and curve local embedding to preserve the
spherical characteristics of equidistortion, continuity, and surface distance,
respectively. Furthermore, we present a query-based global conditional position
embedding to compensate for spatial structure at varying resolutions. It not
only boosts the global perception of spatial position but also sharpens the
depth structure across different patches. Finally, we conduct extensive
experiments on popular benchmarks, demonstrating our superiority over
state-of-the-art solutions.

本文提出了一种名为 SGFormer 的球面几何转换器，旨在解决 360 深度估计中的全景畸变问题，并通过将球面几何先验引入视觉转换器来增强其性能。在实验中，我们对流行的基准数据集进行了广泛的测试，证明了我们方法优于现有最先进的解决方案。

SGFormer：用于 360 深度估计的球面几何变换器

SGFormer: Spherical Geometry Transformer for 360 Depth Estimation

360 depth estimation has recently received great attention for 3D
reconstruction owing to its omnidirectional field of view (FoV). Recent
approaches are predominantly focused on cross-projection fusion with
geometry-based re-projection: they fuse 360 images with equirectangular
projection (ERP) and another projection type, e.g., cubemap projection to
estimate depth with the ERP format. However, these methods suffer from 1)
limited local receptive fields, making it hardly possible to capture large FoV
scenes, and 2) prohibitive computational cost, caused by the complex
cross-projection fusion module design. In this paper, we propose Elite360D, a
novel framework that inputs the ERP image and icosahedron projection (ICOSAP)
point set, which is undistorted and spatially continuous. Elite360D is superior
in its capacity in learning a representation from a local-with-global
perspective. With a flexible ERP image encoder, it includes an ICOSAP point
encoder, and a Bi-projection Bi-attention Fusion (B2F) module (totally ~1M
parameters). Specifically, the ERP image encoder can take various perspective
image-trained backbones (e.g., ResNet, Transformer) to extract local features.
The point encoder extracts the global features from the ICOSAP. Then, the B2F
module captures the semantic- and distance-aware dependencies between each
pixel of the ERP feature and the entire ICOSAP feature set. Without specific
backbone design and obvious computational cost increase, Elite360D outperforms
the prior arts on several benchmark datasets.

360 深度估计是重建三维场景的关键技术之一。本文提出了一种名为 Elite360D 的新框架，通过使用 ERP 图像和 ICOSAP 点集，结合局部和全局特征，有效地估计 360 深度，其在多个基准数据集上优于之前的方法。

Elite360D：通过语义和距离感知双投影融合实现高效的 360 度深度估计

Elite360D: Towards Efficient 360 Depth Estimation via Semantic- and  Distance-Aware Bi-Projection Fusion

Due to difficulties in acquiring ground truth depth of equirectangular (360)
images, the quality and quantity of equirectangular depth data today is
insufficient to represent the various scenes in the world. Therefore, 360 depth
estimation studies, which relied solely on supervised learning, are destined to
produce unsatisfactory results. Although self-supervised learning methods
focusing on equirectangular images (EIs) are introduced, they often have
incorrect or non-unique solutions, causing unstable performance. In this paper,
we propose 360 monocular depth estimation methods which improve on the areas
that limited previous studies. First, we introduce a self-supervised 360 depth
learning method that only utilizes gravity-aligned videos, which has the
potential to eliminate the needs for depth data during the training procedure.
Second, we propose a joint learning scheme realized by combining supervised and
self-supervised learning. The weakness of each learning is compensated, thus
leading to more accurate depth estimation. Third, we propose a non-local fusion
block, which can further retain the global information encoded by vision
transformer when reconstructing the depths. With the proposed methods, we
successfully apply the transformer to 360 depth estimations, to the best of our
knowledge, which has not been tried before. On several benchmarks, our approach
achieves significant improvements over previous works and establishes a state
of the art.

本文提出了一个 360 度深度估计的方法，结合自监督学习和有监督学习，并使用全局信息编码的视觉 transformer 实现全局非局部融合块，提高了深度估计的准确性和整体效果。在多个基准测试中，我们的方法在之前的研究工作中取得了显著的改进和最先进的表现。