The presence of spherical distortion on the Equirectangular image is an
acknowledged challenge in dense regression computer vision tasks, such as
surface normal estimation. Recent advances in convolutional neural networks
(CNNs) strive to mitigate spherical distortion but often fall short in
capturing holistic structures effectively, primarily due to their fixed
receptive field. On the other hand, vision transformers (ViTs) excel in
establishing long-range dependencies through a global self-attention mechanism,
yet they encounter limitations in preserving local details. We introduce
\textit{PanoNormal}, a monocular surface normal estimation architecture
designed for 360{\deg} images, which combines the strengths of CNNs and ViTs.
Specifically, we employ a multi-level global self-attention scheme with the
consideration of the spherical feature distribution, enhancing the
comprehensive understanding of the scene. Our experimental results demonstrate
that our approach achieves state-of-the-art performance across multiple popular
360{\deg} monocular datasets. The code and models will be released.

介绍了一种用于 360 度图像的单目表面法线估计架构 PanoNormal，通过使用多级全局自注意机制和考虑球面特征分布，结合了 CNN 和 ViTs 的优势，实现了在多个流行的 360 度单目数据集上的最先进性能。

PanoNormal：单目室内 360° 表面法线估计

PanoNormal: Monocular Indoor 360° Surface Normal Estimation

Controllable spherical panoramic image generation holds substantial
applicative potential across a variety of domains.However, it remains a
challenging task due to the inherent spherical distortion and geometry
characteristics, resulting in low-quality content generation.In this paper, we
introduce a novel framework of SphereDiffusion to address these unique
challenges, for better generating high-quality and precisely controllable
spherical panoramic images.For the spherical distortion characteristic, we
embed the semantics of the distorted object with text encoding, then explicitly
construct the relationship with text-object correspondence to better use the
pre-trained knowledge of the planar images.Meanwhile, we employ a deformable
technique to mitigate the semantic deviation in latent space caused by
spherical distortion.For the spherical geometry characteristic, in virtue of
spherical rotation invariance, we improve the data diversity and optimization
objectives in the training process, enabling the model to better learn the
spherical geometry characteristic.Furthermore, we enhance the denoising process
of the diffusion model, enabling it to effectively use the learned geometric
characteristic to ensure the boundary continuity of the generated images.With
these specific techniques, experiments on Structured3D dataset show that
SphereDiffusion significantly improves the quality of controllable spherical
image generation and relatively reduces around 35% FID on average.

通过引入 SphereDiffusion 框架来解决球面畸变和几何特性带来的挑战，以更好地生成高质量且可精确控制的球面全景图像。通过嵌入扭曲对象的语义和文本编码，构建文本 - 物体对应关系，利用预训练的平面图像的先验知识；同时采用可变形技术减少球面畸变导致的潜空间中的语义偏差；通过改进训练过程中的数据多样性和优化目标，以适应球面旋转不变性的特点，并增强扩散模型的去噪过程，以保证生成图像的边界连续性。实验结果表明，SphereDiffusion 显著提高了可控球面图像生成的质量，平均降低了约 35% 的 FID。

SphereDiffusion：球面几何感知的失真鲁棒扩散模型

SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion  Model

A well-known challenge in applying deep-learning methods to omnidirectional
images is spherical distortion. In dense regression tasks such as depth
estimation, where structural details are required, using a vanilla CNN layer on
the distorted 360 image results in undesired information loss. In this paper,
we propose a 360 monocular depth estimation pipeline, OmniFusion, to tackle the
spherical distortion issue. Our pipeline transforms a 360 image into
less-distorted perspective patches (i.e. tangent images) to obtain patch-wise
predictions via CNN, and then merge the patch-wise results for final output. To
handle the discrepancy between patch-wise predictions which is a major issue
affecting the merging quality, we propose a new framework with the following
key components. First, we propose a geometry-aware feature fusion mechanism
that combines 3D geometric features with 2D image features to compensate for
the patch-wise discrepancy. Second, we employ the self-attention-based
transformer architecture to conduct a global aggregation of patch-wise
information, which further improves the consistency. Last, we introduce an
iterative depth refinement mechanism, to further refine the estimated depth
based on the more accurate geometric features. Experiments show that our method
greatly mitigates the distortion issue, and achieves state-of-the-art
performances on several 360 monocular depth estimation benchmark datasets.

本篇论文提出了一种名为 OmniFusion 的 360 单目深度估计流程，通过将 360 图像转换为切线图像并采用 CNN 进行分段预测，最后合并分段结果以处理球形畸变问题，并通过几何感知特征融合机制、自注意力变换架构和迭代深度细化机制处理分段预测不一致问题，成功的在多个 360 单目深度估计基准数据集上取得了最先进的性能。