Vision-language models such as CLIP have shown great impact on diverse
downstream tasks for zero-shot or label-free predictions. However, when it
comes to low-level vision such as image restoration their performance
deteriorates dramatically due to corrupted inputs. In this paper, we present a
degradation-aware vision-language model (DA-CLIP) to better transfer pretrained
vision-language models to low-level vision tasks as a universal framework for
image restoration. More specifically, DA-CLIP trains an additional controller
that adapts the fixed CLIP image encoder to predict high-quality feature
embeddings. By integrating the embedding into an image restoration network via
cross-attention, we are able to pilot the model to learn a high-fidelity image
reconstruction. The controller itself will also output a degradation feature
that matches the real corruptions of the input, yielding a natural classifier
for different degradation types. In addition, we construct a mixed degradation
dataset with synthetic captions for DA-CLIP training. Our approach advances
state-of-the-art performance on both degradation-specific and unified image
restoration tasks, showing a promising direction of prompting image restoration
with large-scale pretrained vision-language models. Our code is available at
this https URL

我们提出了一种降级感知的视觉 - 语言模型（DA-CLIP），用于将预训练的视觉 - 语言模型应用于低级视觉任务，作为通用的图像恢复框架，通过集成嵌入到图像恢复网络中并进行交叉注意力，使模型能够学习高保真度的图像重建。