We are witnessing a revolution in conditional image synthesis with the recent
success of large scale text-to-image generation methods. This success also
opens up new opportunities in controlling the generation and editing process
using multi-modal input. While spatial control using cues such as depth,
sketch, and other images has attracted a lot of research, we argue that another
equally effective modality is audio since sound and sight are two main
components of human perception. Hence, we propose a method to enable
audio-conditioning in large scale image diffusion models. Our method first maps
features obtained from audio clips to tokens that can be injected into the
diffusion model in a fashion similar to text tokens. We introduce additional
audio-image cross attention layers which we finetune while freezing the weights
of the original layers of the diffusion model. In addition to audio conditioned
image generation, our method can also be utilized in conjuction with diffusion
based editing methods to enable audio conditioned image editing. We demonstrate
our method on a wide range of audio and image datasets. We perform extensive
comparisons with recent methods and show favorable performance.

我们提出了一种能够在大规模图像扩散模型中实现音频条件的方法，通过将从音频剪辑中获取的特征映射到可以注入扩散模型的令牌中，引入了额外的音频 - 图像交叉注意力层，可以与扩散编辑方法相结合进行音频条件的图像编辑。