The vast applications of deep generative models are anchored in three core
capabilities -- generating new instances, reconstructing inputs, and learning
compact representations -- across various data types, such as discrete
text/protein sequences and continuous images. Existing model families, like
Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs),
autoregressive models, and diffusion models, generally excel in specific
capabilities and data types but fall short in others. We introduce generalized
diffusion with learnable encoder-decoder (DiLED), that seamlessly integrates
the core capabilities for broad applicability and enhanced performance. DiLED
generalizes the Gaussian noising-denoising in standard diffusion by introducing
parameterized encoding-decoding. Crucially, DiLED is compatible with the
well-established diffusion model objective and training recipes, allowing
effective learning of the encoder-decoder parameters jointly with diffusion. By
choosing appropriate encoder/decoder (e.g., large language models), DiLED
naturally applies to different data types. Extensive experiments on text,
proteins, and images demonstrate DiLED's flexibility to handle diverse data and
tasks and its strong improvement over various existing models.

通过引入可学习的编码器 - 解码器，DiLED 广义扩展了常规扩散过程的高斯噪声去噪方法，在不同数据类型上具有广泛适用性和提升性能。实验证明 DiLED 在处理各种数据和任务上都有很好的灵活性，并在不同已有模型上取得了显著的改进。