Human perceives rich auditory experience with distinct sound heard by ears.
Videos recorded with binaural audio particular simulate how human receives
ambient sound. However, a large number of videos are with monaural audio only,
which would degrade the user experience due to the lack of ambient information.
To address this issue, we propose an audio spatialization framework to convert
a monaural video into a binaural one exploiting the relationship across audio
and visual components. By preserving the left-right consistency in both audio
and visual modalities, our learning strategy can be viewed as a self-supervised
learning technique, and alleviates the dependency on a large amount of video
data with ground truth binaural audio data during training. Experiments on
benchmark datasets confirm the effectiveness of our proposed framework in both
semi-supervised and fully supervised scenarios, with ablation studies and
visualization further support the use of our model for audio spatialization.

该研究提出了一种音频空间化框架，利用音频和视觉组件之间的关系将单通道视频转换为双耳音频，通过在音频和视觉模态中保留左右一致性的学习策略来实现自监督学习，验证结果表明该框架在半监督和全监督方案中具有很好的效果。

利用部分监督的视听一致性生成空间音频

Exploiting Audio-Visual Consistency with Partial Supervision for Spatial  Audio Generation

Stereophonic audio is an indispensable ingredient to enhance human auditory
experience. Recent research has explored the usage of visual information as
guidance to generate binaural or ambisonic audio from mono ones with stereo
supervision. However, this fully supervised paradigm suffers from an inherent
drawback: the recording of stereophonic audio usually requires delicate devices
that are expensive for wide accessibility. To overcome this challenge, we
propose to leverage the vastly available mono data to facilitate the generation
of stereophonic audio. Our key observation is that the task of visually
indicated audio separation also maps independent audios to their corresponding
visual positions, which shares a similar objective with stereophonic audio
generation. We integrate both stereo generation and source separation into a
unified framework, Sep-Stereo, by considering source separation as a particular
type of audio spatialization. Specifically, a novel associative pyramid network
architecture is carefully designed for audio-visual feature fusion. Extensive
experiments demonstrate that our framework can improve the stereophonic audio
generation results while performing accurate sound separation with a shared
backbone.

该研究提出了一种统一的框架，将立体声音频生成和源分离相结合，通过使用可用的单声道数据来提高立体声音频生成的结果。

Sep-Stereo: 基于可视化引导的立体声音频生成方法，通过关联源分离

Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating  Source Separation

Spatial audio is an essential medium to audiences for 3D visual and auditory
experience. However, the recording devices and techniques are expensive or
inaccessible to the general public. In this work, we propose a self-supervised
audio spatialization network that can generate spatial audio given the
corresponding video and monaural audio. To enhance spatialization performance,
we use an auxiliary classifier to classify ground-truth videos and those with
audio where the left and right channels are swapped. We collect a large-scale
video dataset with spatial audio to validate the proposed method. Experimental
results demonstrate the effectiveness of the proposed model on the audio
spatialization task.

本文介绍了一种使用自监督网络进行音频空间化的方法，并使用辅助分类器和大规模视频数据集来提高性能，取得了较好的实验结果。