We present Diffusion Soup, a compartmentalization method for Text-to-Image
Generation that averages the weights of diffusion models trained on sharded
data. By construction, our approach enables training-free continual learning
and unlearning with no additional memory or inference costs, since models
corresponding to data shards can be added or removed by re-averaging. We show
that Diffusion Soup samples from a point in weight space that approximates the
geometric mean of the distributions of constituent datasets, which offers
anti-memorization guarantees and enables zero-shot style mixing. Empirically,
Diffusion Soup outperforms a paragon model trained on the union of all data
shards and achieves a 30% improvement in Image Reward (.34 $\to$ .44) on domain
sharded data, and a 59% improvement in IR (.37 $\to$ .59) on aesthetic data. In
both cases, souping also prevails in TIFA score (respectively, 85.5 $\to$ 86.5
and 85.6 $\to$ 86.8). We demonstrate robust unlearning -- removing any
individual domain shard only lowers performance by 1% in IR (.45 $\to$ .44) --
and validate our theoretical insights on anti-memorization using real data.
Finally, we showcase Diffusion Soup's ability to blend the distinct styles of
models finetuned on different shards, resulting in the zero-shot generation of
hybrid styles.

Diffusion Soup 通过对分片数据上训练的扩散模型的权重进行平均，实现了训练免费的持续学习和遗忘，从而在权重空间中从几何平均的分布中采样，提供反记忆化保证，并实现零样式混合和混合样式的零样生成。