As virtual and augmented reality applications gain popularity,
omnidirectional image (ODI) super-resolution has become increasingly important.
Unlike 2D plain images that are formed on a plane, ODIs are projected onto
spherical surfaces. Applying established image super-resolution methods to
ODIs, therefore, requires performing equirectangular projection (ERP) to map
the ODIs onto a plane. ODI super-resolution needs to take into account
geometric distortion resulting from ERP. However, without considering such
geometric distortion of ERP images, previous deep-learning-based methods only
utilize a limited range of pixels and may easily miss self-similar textures for
reconstruction. In this paper, we introduce a novel Geometric Distortion Guided
Transformer for Omnidirectional image Super-Resolution (GDGT-OSR).
Specifically, a distortion modulated rectangle-window self-attention mechanism,
integrated with deformable self-attention, is proposed to better perceive the
distortion and thus involve more self-similar textures. Distortion modulation
is achieved through a newly devised distortion guidance generator that produces
guidance by exploiting the variability of distortion across latitudes.
Furthermore, we propose a dynamic feature aggregation scheme to adaptively fuse
the features from different self-attention modules. We present extensive
experimental results on public datasets and show that the new GDGT-OSR
outperforms methods in existing literature.

介绍了一种新的基于几何畸变引导变压器的全向图像超分辨率方法，通过引入畸变调制的矩形窗口自注意机制和变形自注意机制，提高了感知畸变的能力，并在不同自注意模块间采用动态特征融合方案。在公开数据集上进行了广泛的实验，表明该方法在全向图像超分辨率方面优于现有方法。

几何畸变导向的全向图像超分辨率变换器

Geometric Distortion Guided Transformer for Omnidirectional Image  Super-Resolution

Augmented reality (AR) using camera images in mobile devices is becoming
popular for tourism promotion. However, obstructions such as tourists appearing
in the camera images may cause the camera pose estimation error, resulting in
CG misalignment and reduced visibility of the contents. To avoid this problem,
Indirect AR (IAR), which does not use real-time camera images, has been
proposed. In this method, an omnidirectional image is captured and virtual
objects are synthesized on the image in advance. Users can experience AR by
viewing a scene extracted from the synthesized omnidirectional image according
to the device's sensor. This enables robustness and high visibility. However,
if the weather conditions and season in the pre-captured 360 images differs
from the current weather conditions and season when AR is experienced, the
realism of the AR experience is reduced. To overcome the problem, we propose a
method for correcting the intensity and texture of a past omnidirectional image
using camera images from mobile devices. We first perform semantic
segmentation. We then reproduce the current sky pattern by panoramic image
composition and inpainting. For the other areas, we correct the intensity by
histogram matching. In experiments, we show the effectiveness of the proposed
method using various scenes.

通过使用移动设备中的相机图像，增强现实（AR）在旅游推广中越来越受欢迎。然而，游客出现在相机图像中可能会导致相机姿态估计误差，进而产生 CG 不对齐和内容可见性降低的问题。为了避免这个问题，提出了一种不使用实时相机图像的间接 AR（IAR）。在该方法中，首先捕获全景图像，然后预先在图像上合成虚拟物体。用户可以通过查看从合成全景图像中提取的场景来体验 AR，以配合设备的传感器。这样可以实现鲁棒性和高可见性。然而，如果预先捕获的 360 全景图像中的天气条件和季节与体验 AR 时的当前天气条件和季节不同，AR 体验的真实感将降低。为了克服这个问题，我们提出了一种使用移动设备的相机图像来纠正过去全景图像强度和纹理的方法。首先进行语义分割。然后通过全景图像合成和修补来再现当前天空的模式。对于其他区域，通过直方图匹配来修正强度。通过实验证明了所提出方法在各种场景下的有效性。