Binaural stereo audio is recorded by imitating the way the human ear receives
sound, which provides people with an immersive listening experience. Existing
approaches leverage autoencoders and directly exploit visual spatial
information to synthesize binaural stereo, resulting in a limited
representation of visual guidance. For the first time, we propose a visually
guided generative adversarial approach for generating binaural stereo audio
from mono audio. Specifically, we develop a Stereo Audio Generation Model
(SAGM), which utilizes shared spatio-temporal visual information to guide the
generator and the discriminator to work separately. The shared visual
information is updated alternately in the generative adversarial stage,
allowing the generator and discriminator to deliver their respective guided
knowledge while visually sharing. The proposed method learns bidirectional
complementary visual information, which facilitates the expression of visual
guidance in generation. In addition, spatial perception is a crucial attribute
of binaural stereo audio, and thus the evaluation of stereo spatial perception
is essential. However, previous metrics failed to measure the spatial
perception of audio. To this end, a metric to measure the spatial perception of
audio is proposed for the first time. The proposed metric is capable of
measuring the magnitude and direction of spatial perception in the temporal
dimension. Further, considering its function, it is feasible to utilize it
instead of demanding user studies to some extent. The proposed method achieves
state-of-the-art performance on 2 datasets and 5 evaluation metrics.
Qualitative experiments and user studies demonstrate that the method generates
space-realistic stereo audio.

通过使用视觉引导生成对抗方法，本论文第一次提出了从单声道音频生成双元立体音频的方法，并且引入了一个测量音频空间感知的度量标准，通过定量和定向的方式评估音频的空间感知，在 2 个数据集和 5 个评价指标上实现了最先进的性能。定性实验和用户研究证明了该方法生成的空间逼真的立体音频。