Recently, discrete diffusion language models have demonstrated promising results in NLP. However, there has been limited research on integrating Pretrained Language Models (PLMs) into discrete diffusion models, resulting in underwhelming performance in downstream NLP generation tasks. This integration is particularly challenging because of the discrepancy between step-wise denoising strategy of diffusion models and single-step mask prediction approach of MLM-based PLMs. In this paper, we introduce Diffusion-EAGS, a novel approach that effectively integrates PLMs with the diffusion models. Furthermore, as it is challenging for PLMs to determine where to apply denoising during the diffusion process, we integrate an entropy tracking module to assist them. Finally, we propose entropy-based noise scheduling in the forward process to improve the effectiveness of entropy-adaptive sampling throughout the generation phase. Experimental results show that Diffusion-EAGS outperforms existing diffusion baselines in downstream generation tasks, achieving high text quality and diversity with precise token-level control. We also show that our model is capable of adapting to bilingual and low-resource settings, which are common in real-world applications.

本研究针对离散扩散语言模型与预训练语言模型(PLM)整合中存在的有效性不足问题，提出了一种新颖的方法Diffusion-EAGS，能够有效结合两者。研究成果表明，该模型在文本生成任务中表现优异，不仅在文本质量和多样性上超越了现有的基线模型，还在双语和低资源环境中展现出良好的适应能力。

基于PLM的离散扩散语言模型与熵自适应Gibbs采样