We present Diffusion Model Patching (DMP), a simple method to boost the performance of pre-trained diffusion models that have already reached convergence, with a negligible increase in parameters. DMP inserts a small, learnable set of prompts into the model's input space while keeping the original model frozen. The effectiveness of DMP is not merely due to the addition of parameters but stems from its dynamic gating mechanism, which selects and combines a subset of learnable prompts at every step of the generative process (e.g., reverse denoising steps). This strategy, which we term "mixture-of-prompts", enables the model to draw on the distinct expertise of each prompt, essentially "patching" the model's functionality at every step with minimal yet specialized parameters. Uniquely, DMP enhances the model by further training on the same dataset on which it was originally trained, even in a scenario where significant improvements are typically not expected due to model convergence. Experiments show that DMP significantly enhances the converged FID of DiT-L/2 on FFHQ 256x256 by 10.38%, achieved with only a 1.43% parameter increase and 50K additional training iterations.

Diffusion Model Patching (DMP) 是一种简单的方法，通过在模型的输入空间中插入一小组可学习的提示来提升已经达到收敛状态的预训练扩散模型的性能，而不会显著增加参数。该方法通过一种动态门控机制（称为“mixture-of-prompts”），在生成过程的每个步骤中选择并组合一子集的可学习提示，充分发挥每个提示的独特专长，从而在每个步骤中“修补”模型的功能。实验证明，DMP显著提升了在FFHQ 256x256数据集上DiT-L/2模型的达到收敛状态的FID（Fréchet Inception Distance）指标，仅增加了1.43%的参数和5万次额外的训练迭代。

通过提示混合实现扩散模型修补