This paper raises the new task of Fisheye Semantic Completion (FSC), where
dense texture, structure, and semantics of a fisheye image are inferred even
beyond the sensor field-of-view (FoV). Fisheye cameras have larger FoV than
ordinary pinhole cameras, yet its unique special imaging model naturally leads
to a blind area at the edge of the image plane. This is suboptimal for
safety-critical applications since important perception tasks, such as semantic
segmentation, become very challenging within the blind zone. Previous works
considered the out-FoV outpainting and in-FoV segmentation separately. However,
we observe that these two tasks are actually closely coupled. To jointly
estimate the tightly intertwined complete fisheye image and scene semantics, we
introduce the new FishDreamer which relies on successful ViTs enhanced with a
novel Polar-aware Cross Attention module (PCA) to leverage dense context and
guide semantically-consistent content generation while considering different
polar distributions. In addition to the contribution of the novel task and
architecture, we also derive Cityscapes-BF and KITTI360-BF datasets to
facilitate training and evaluation of this new track. Our experiments
demonstrate that the proposed FishDreamer outperforms methods solving each task
in isolation and surpasses alternative approaches on the Fisheye Semantic
Completion. Code and datasets are publicly available at
this https URL

介绍了针对鱼眼相机照片的新任务 —— 鱼眼语义完成（FSC），以及设计的新算法 FishDreamer。FishDreamer 利用成功的 ViTs，通过一种新颖的极坐标感知交叉注意力模块（PCA）来实现密集背景信息的利用和语义一致性的内容生成，并重视不同的极坐标分布。代码和数据集在该 URL 公开。