Audio-visual navigation is an audio-targeted wayfinding task where a robot
agent is entailed to travel a never-before-seen 3D environment towards the
sounding source. In this article, we present ORAN, an omnidirectional
audio-visual navigator based on cross-task navigation skill transfer. In
particular, ORAN sharpens its two basic abilities for a such challenging task,
namely wayfinding and audio-visual information gathering. First, ORAN is
trained with a confidence-aware cross-task policy distillation (CCPD) strategy.
CCPD transfers the fundamental, point-to-point wayfinding skill that is well
trained on the large-scale PointGoal task to ORAN, so as to help ORAN to better
master audio-visual navigation with far fewer training samples. To improve the
efficiency of knowledge transfer and address the domain gap, CCPD is made to be
adaptive to the decision confidence of the teacher policy. Second, ORAN is
equipped with an omnidirectional information gathering (OIG) mechanism, i.e.,
gleaning visual-acoustic observations from different directions before
decision-making. As a result, ORAN yields more robust navigation behaviour.
Taking CCPD and OIG together, ORAN significantly outperforms previous
competitors. After the model ensemble, we got 1st in Soundspaces Challenge
2022, improving SPL and SR by 53% and 35% relatively.

基于交叉任务导航技能迁移的全向音频视觉导航器 ORAN，通过信心感知的交叉任务策略提炼（CCPD）和全向信息收集（OIG）的机制，显著超越以往的竞争对手，提高了音空间挑战 2022 年的 SPL 和 SR 等指标。

基于知识转移的全向信息获取的音视频导航

Omnidirectional Information Gathering for Knowledge Transfer-based  Audio-Visual Navigation

A large field-of-view fisheye camera allows for capturing a large area with
minimal numbers of cameras when they are mounted on a high position facing
downwards. This top-view omnidirectional setup greatly reduces the work and
cost for deployment compared to traditional solutions with multiple perspective
cameras. In recent years, deep learning has been widely employed for vision
related tasks, including for such omnidirectional settings. In this survey, we
look at the application of deep learning in combination with omnidirectional
top-view cameras, including the available datasets, human and object detection,
human pose estimation, activity recognition and other miscellaneous
applications.

该论文综述了深度学习在结合全景鱼眼摄像机的视觉任务中的应用，包括物体检测、活动识别等，利用鱼眼镜头的全向视野可以大大减少部署所需的工作和成本。

深度学习在俯视全景成像中的应用：综述

Applications of Deep Learning for Top-View Omnidirectional Imaging: A  Survey

Monocular omnidirectional depth estimation is receiving considerable research
attention due to its broad applications for sensing 360{\deg} surroundings.
Existing approaches in this field suffer from limitations in recovering small
object details and data lost during the ground-truth depth map acquisition. In
this paper, a novel monocular omnidirectional depth estimation model, namely
HiMODE is proposed based on a hybrid CNN+Transformer (encoder-decoder)
architecture whose modules are efficiently designed to mitigate distortion and
computational cost, without performance degradation. Firstly, we design a
feature pyramid network based on the HNet block to extract high-resolution
features near the edges. The performance is further improved, benefiting from a
self and cross attention layer and spatial/temporal patches in the Transformer
encoder and decoder, respectively. Besides, a spatial residual block is
employed to reduce the number of parameters. By jointly passing the deep
features extracted from an input image at each backbone block, along with the
raw depth maps predicted by the transformer encoder-decoder, through a context
adjustment layer, our model can produce resulting depth maps with better visual
quality than the ground-truth. Comprehensive ablation studies demonstrate the
significance of each individual module. Extensive experiments conducted on
three datasets; Stanford3D, Matterport3D, and SunCG, demonstrate that HiMODE
can achieve state-of-the-art performance for 360{\deg} monocular depth
estimation.

本论文研究了使用单目全向视角图进行深度估计的问题，提出了一种基于混合 CNN 和 Transformer 的 HiMODE 深度估计模型，该模型可以很好地恢复小目标细节，避免由于获取地面实际深度图时可能出现的数据损失问题，并且具有更高的计算效率和更好的视觉效果。

HiMODE: 一种混合单目全方位深度估计模型

HiMODE: A Hybrid Monocular Omnidirectional Depth Estimation Model

Omnidirectional (or 360-degree) images and videos are emergent signals in
many areas such as robotics and virtual/augmented reality. In particular, for
virtual reality, they allow an immersive experience in which the user is
provided with a 360-degree field of view and can navigate throughout a scene,
e.g., through the use of Head Mounted Displays. Since it represents the full
360-degree field of view from one point of the scene, omnidirectional content
is naturally represented as spherical visual signals. Current approaches for
capturing, processing, delivering, and displaying 360-degree content, however,
present many open technical challenges and introduce several types of
distortions in these visual signals. Some of the distortions are specific to
the nature of 360-degree images, and often different from those encountered in
the classical image communication framework. This paper provides a first
comprehensive review of the most common visual distortions that alter
360-degree signals undergoing state of the art processing in common
applications. While their impact on viewers' visual perception and on the
immersive experience at large is still unknown ---thus, it stays an open
research topic--- this review serves the purpose of identifying the main causes
of visual distortions in the end-to-end 360-degree content distribution
pipeline. It is essential as a basis for benchmarking different processing
techniques, allowing the effective design of new algorithms and applications.
It is also necessary to the deployment of proper psychovisual studies to
characterise the human perception of these new images in interactive and
immersive applications.

对于全景图像和全景视频这样的视频信号，其常常存在各种视觉畸变，这篇文章对这些畸变进行了综述，为算法设计提供了依据，并且必要于心理视觉学习及交互性和沉浸式应用研究。