Diffusion models have demonstrated great success in the field of text-to-image generation. However, alleviating the misalignment between the text prompts and images is still challenging. The root reason behind the misalignment has not been extensively investigated. We observe that the misalignment is caused by inadequate token attention activation. We further attribute this phenomenon to the diffusion model's insufficient condition utilization, which is caused by its training paradigm. To address the issue, we propose CoMat, an end-to-end diffusion model fine-tuning strategy with an image-to-text concept matching mechanism. We leverage an image captioning model to measure image-to-text alignment and guide the diffusion model to revisit ignored tokens. A novel attribute concentration module is also proposed to address the attribute binding problem. Without any image or human preference data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL. Extensive experiments show that CoMat-SDXL significantly outperforms the baseline model SDXL in two text-to-image alignment benchmarks and achieves start-of-the-art performance.

提出了CoMat，一种将图像到文本概念匹配机制与端到端扩散模型微调策略相结合的方法，通过利用图像字幕模型衡量图像到文本的对齐性并指导扩散模型重新访问忽略的标记，解决了文本提示与图像之间的对齐不足问题。在两个文本到图像对齐基准测试中，CoMat-SDXL相对于基准模型SDXL表现出色，达到了最先进的性能。

CoMat: 文本到图像扩散模型与图像到文本概念匹配的对齐