We propose a Bayesian framework for fine-tuning large diffusion models with a
novel network structure called Bayesian Power Steering (BPS). We clarify the
meaning behind adaptation from a \textit{large probability space} to a
\textit{small probability space} and explore the task of fine-tuning
pre-trained models using learnable modules from a Bayesian perspective. BPS
extracts task-specific knowledge from a pre-trained model's learned prior
distribution. It efficiently leverages large diffusion models, differentially
intervening different hidden features with a head-heavy and foot-light
configuration. Experiments highlight the superiority of BPS over contemporary
methods across a range of tasks even with limited amount of data. Notably, BPS
attains an FID score of 10.49 under the sketch condition on the COCO17 dataset.

我们提出了一种贝叶斯框架，用于通过一种名为贝叶斯动力转向（BPS）的新型网络结构对大规模扩散模型进行微调。BPS 从预训练模型的学习先验分布中提取任务特定的知识，通过头重脚轻的配置差异地介入不同的隐藏特征。实验证明，即使在有限的数据量下，BPS 在各种任务上均优于现有方法，特别是在 COCO17 数据集的素描条件下，BPS 达到了 10.49 的 FID 得分。