We present EasyGen, an efficient model designed to enhance multimodal
understanding and generation by harnessing the capabilities of diffusion models
and large language models (LLMs). Unlike existing multimodal models that
predominately depend on encoders like CLIP or ImageBind and need ample amounts
of training data to bridge the gap between modalities, EasyGen is built upon a
bidirectional conditional diffusion model named BiDiffuser, which promotes more
efficient interactions between modalities. EasyGen handles image-to-text
generation by integrating BiDiffuser and an LLM via a simple projection layer.
Unlike most existing multimodal models that are limited to generating text
responses, EasyGen can also facilitate text-to-image generation by leveraging
the LLM to create textual descriptions, which can be interpreted by BiDiffuser
to generate appropriate visual responses. Extensive quantitative and
qualitative experiments demonstrate the effectiveness of EasyGen, whose
training can be easily achieved in a lab setting. The source code is available
at this https URL

EasyGen 是一个高效的模型，通过结合扩散模型和大型语言模型（LLMs）的能力，来提高多模态理解和生成。与现有的多模态模型不同，EasyGen 建立在一种名为 BiDiffuser 的双向条件扩散模型的基础上，推动了模态之间更有效的交互。EasyGen 通过整合 BiDiffuser 和 LLM，通过一个简单的投影层处理图像到文本的生成。与大多数现有的多模态模型仅限于生成文本响应不同，EasyGen 还可以利用 LLM 生成文本描述来生成适当的视觉响应。大量的定量和定性实验证明了 EasyGen 的有效性，其在实验室环境中易于实现的训练。源代码可以在此 https URL 中获得。