Diffusion models trained on large-scale datasets have achieved remarkable
progress in image synthesis. However, due to the randomness in the diffusion
process, they often struggle with handling diverse low-level tasks that require
details preservation. To overcome this limitation, we present a new Diff-Plugin
framework to enable a single pre-trained diffusion model to generate
high-fidelity results across a variety of low-level tasks. Specifically, we
first propose a lightweight Task-Plugin module with a dual branch design to
provide task-specific priors, guiding the diffusion process in preserving image
content. We then propose a Plugin-Selector that can automatically select
different Task-Plugins based on the text instruction, allowing users to edit
images by indicating multiple low-level tasks with natural language. We conduct
extensive experiments on 8 low-level vision tasks. The results demonstrate the
superiority of Diff-Plugin over existing methods, particularly in real-world
scenarios. Our ablations further validate that Diff-Plugin is stable,
schedulable, and supports robust training across different dataset sizes.

通过 Diff-Plugin 框架，利用扩展的扩散模型能够以高保真度完成多个低水平任务，通过轻量级的 Task-Plugin 模块提供任务特定的先验知识并根据自然语言指令自动选择不同的 Task-Plugin 进行图像编辑。

Diff-Plugin：为基于扩散的低层任务注入新的细节

Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks

Pre-training has marked numerous state of the arts in high-level computer
vision, while few attempts have ever been made to investigate how pre-training
acts in image processing systems. In this paper, we tailor transformer-based
pre-training regimes that boost various low-level tasks. To comprehensively
diagnose the influence of pre-training, we design a whole set of principled
evaluation tools that uncover its effects on internal representations. The
observations demonstrate that pre-training plays strikingly different roles in
low-level tasks. For example, pre-training introduces more local information to
higher layers in super-resolution (SR), yielding significant performance gains,
while pre-training hardly affects internal feature representations in
denoising, resulting in limited gains. Further, we explore different methods of
pre-training, revealing that multi-related-task pre-training is more effective
and data-efficient than other alternatives. Finally, we extend our study to
varying data scales and model sizes, as well as comparisons between
transformers and CNNs-based architectures. Based on the study, we successfully
develop state-of-the-art models for multiple low-level tasks. Code is released
at this https URL

本文研究预训练在图像处理系统中的作用，采用基于 Transformer 的预训练机制提高各种低级任务的性能，并设计一整套对内部表示影响的评估工具。实验证明，预训练对低级任务有着截然不同的影响，其中多任务预训练是最有效和数据高效的方法，本研究成功地开发了多种低级任务的最新模型。