In recent years, image generation has shown a great leap in performance,
where diffusion models play a central role. Although generating high-quality
images, such models are mainly conditioned on textual descriptions. This begs
the question: "how can we adopt such models to be conditioned on other
modalities?". In this paper, we propose a novel method utilizing latent
diffusion models trained for text-to-image-generation to generate images
conditioned on audio recordings. Using a pre-trained audio encoding model, the
proposed method encodes audio into a new token, which can be considered as an
adaptation layer between the audio and text representations. Such a modeling
paradigm requires a small number of trainable parameters, making the proposed
approach appealing for lightweight optimization. Results suggest the proposed
method is superior to the evaluated baseline methods, considering objective and
subjective metrics. Code and samples are available at:
this https URL

本论文提出了一种新方法，利用文本 - 图像生成中训练的潜在扩散模型，生成基于音频记录的图像。该方法使用预训练的音频编码模型将音频编码成新令牌，这可以被视为音频和文本表示之间的自适应层。结果表明，相较于基准方法，该方法在客观和主观度量方面表现优异。

AudioToken：基于文本条件扩散模型的音频到图像生成适应性

AudioToken: Adaptation of Text-Conditioned Diffusion Models for  Audio-to-Image Generation

Generative models, especially diffusion models (DMs), have achieved promising
results for generating feature-rich geometries and advancing foundational
science problems such as molecule design. Inspired by the recent huge success
of Stable (latent) Diffusion models, we propose a novel and principled method
for 3D molecule generation named Geometric Latent Diffusion Models (GeoLDM).
GeoLDM is the first latent DM model for the molecular geometry domain, composed
of autoencoders encoding structures into continuous latent codes and DMs
operating in the latent space. Our key innovation is that for modeling the 3D
molecular geometries, we capture its critical roto-translational equivariance
constraints by building a point-structured latent space with both invariant
scalars and equivariant tensors. Extensive experiments demonstrate that GeoLDM
can consistently achieve better performance on multiple molecule generation
benchmarks, with up to 7\% improvement for the valid percentage of large
biomolecules. Results also demonstrate GeoLDM's higher capacity for
controllable generation thanks to the latent modeling. Code is provided at
https://github.com/MinkaiXu/GeoLDM.

本研究提出了一种基于几何潜变扩散模型（GeoLDM）的新方法，通过在潜空间中运行扩散模型来生成分子的三维几何结构。该方法对生成大分子的有效百分比有 7％的提升。