Despite the potential benefits of Diffusion Models for NLP applications,
publicly available implementations, trained models, or reproducible training
procedures currently need to be publicly available. We present the Democratized
Diffusion Language Model (DDLM), based on the Continuous Diffusion for
Categorical Data (CDCD) framework, to address these challenges. We propose a
simplified training procedure for DDLM using the C4 dataset and perform an
in-depth analysis of the trained model's behavior. Furthermore, we introduce a
novel early-exiting strategy for faster sampling with models trained with score
interpolation. Since no previous works aimed at solving downstream tasks with
pre-trained Diffusion LM (e.g., classification tasks), we experimented with
GLUE Benchmark to study the ability of DDLM to transfer knowledge. With this
paper, we propose available training and evaluation pipelines to other
researchers and pre-trained DDLM models, which could be used in future research
with Diffusion LMs.

本文旨在提出一种去中心化的扩散语言模型（DDLM），该模型基于连续性扩散的范畴数据（CDCD）框架，使用 C4 数据集进行简化的培训过程，并提出一种新颖的早期退出策略，通过 GLUE 基准研究 DDLM 的知识转移能力。