In panorama understanding, the widely used equirectangular projection (ERP) entails boundary discontinuity and spatial distortion. It severely deteriorates the conventional CNNs and vision Transformers on panoramas. In this paper, we propose a simple yet effective architecture named PanoSwin to learn panorama representations with ERP. To deal with the challenges brought by equirectangular projection, we explore a pano-style shift windowing scheme and novel pitch attention to address the boundary discontinuity and the spatial distortion, respectively. Besides, based on spherical distance and Cartesian coordinates, we adapt absolute positional embeddings and relative positional biases for panoramas to enhance panoramic geometry information. Realizing that planar image understanding might share some common knowledge with panorama understanding, we devise a novel two-stage learning framework to facilitate knowledge transfer from the planar images to panoramas. We conduct experiments against the state-of-the-art on various panoramic tasks, i.e., panoramic object detection, panoramic classification, and panoramic layout estimation. The experimental results demonstrate the effectiveness of PanoSwin in panorama understanding.

本文提出了一种名为PanoSwin的简单而有效的架构，用于学习具有ERP的全景表示。通过探索全景式位移窗口方案和新颖的俯仰注意力分别解决边界不连续和空间失真的挑战，并利用绝对位置嵌入和相对位置偏差来增强全景几何信息。同时，通过设计一种新颖的两阶段学习框架，从平面图像向全景图像进行知识转移，实验结果表明PanoSwin在全景理解方面具有有效性。

PanoSwin：全景风格的Swin Transformer用于全景理解