Stereophonic audio is an indispensable ingredient to enhance human auditory
experience. Recent research has explored the usage of visual information as
guidance to generate binaural or ambisonic audio from mono ones with stereo
supervision. However, this fully supervised paradigm suffers from an inherent
drawback: the recording of stereophonic audio usually requires delicate devices
that are expensive for wide accessibility. To overcome this challenge, we
propose to leverage the vastly available mono data to facilitate the generation
of stereophonic audio. Our key observation is that the task of visually
indicated audio separation also maps independent audios to their corresponding
visual positions, which shares a similar objective with stereophonic audio
generation. We integrate both stereo generation and source separation into a
unified framework, Sep-Stereo, by considering source separation as a particular
type of audio spatialization. Specifically, a novel associative pyramid network
architecture is carefully designed for audio-visual feature fusion. Extensive
experiments demonstrate that our framework can improve the stereophonic audio
generation results while performing accurate sound separation with a shared
backbone.

该研究提出了一种统一的框架，将立体声音频生成和源分离相结合，通过使用可用的单声道数据来提高立体声音频生成的结果。