In recent years, with the realistic generation results and a wide range of
personalized applications, diffusion-based generative models gain huge
attention in both visual and audio generation areas. Compared to the
considerable advancements of text2image or text2audio generation, research in
audio2visual or visual2audio generation has been relatively slow. The recent
audio-visual generation methods usually resort to huge large language model or
composable diffusion models. Instead of designing another giant model for
audio-visual generation, in this paper we take a step back showing a simple and
lightweight generative transformer, which is not fully investigated in
multi-modal generation, can achieve excellent results on image2audio
generation. The transformer operates in the discrete audio and visual
Vector-Quantized GAN space, and is trained in the mask denoising manner. After
training, the classifier-free guidance could be deployed off-the-shelf
achieving better performance, without any extra training or modification. Since
the transformer model is modality symmetrical, it could also be directly
deployed for audio2image generation and co-generation. In the experiments, we
show that our simple method surpasses recent image2audio generation methods.
Generated audio samples can be found at
this https URL

通过使用简单轻量级的生成型 Transformer 模型，本论文在图像到音频生成任务上取得了更好的性能，同时也可用于音频到图像生成和协同生成。