Generating combined visual and auditory sensory experiences is critical for
the consumption of immersive content. Recent advances in neural generative
models have enabled the creation of high-resolution content across multiple
modalities such as images, text, speech, and videos. Despite these successes,
there remains a significant gap in the generation of high-quality spatial audio
that complements generated visual content. Furthermore, current audio
generation models excel in either generating natural audio or speech or music
but fall short in integrating spatial audio cues necessary for immersive
experiences. In this work, we introduce SEE-2-SOUND, a zero-shot approach that
decomposes the task into (1) identifying visual regions of interest; (2)
locating these elements in 3D space; (3) generating mono-audio for each; and
(4) integrating them into spatial audio. Using our framework, we demonstrate
compelling results for generating spatial audio for high-quality videos,
images, and dynamic images from the internet, as well as media generated by
learned approaches.

通过 SEE-2-SOUND 引入的零样本方法，将多模式内容生成、神经生成模型和空间音频相结合，可以为高质量视频、图像和互联网动态图像生成空间音频，实现沉浸式体验。