In image restoration (IR), leveraging semantic priors from segmentation
models has been a common approach to improve performance. The recent segment
anything model (SAM) has emerged as a powerful tool for extracting advanced
semantic priors to enhance IR tasks. However, the computational cost of SAM is
prohibitive for IR, compared to existing smaller IR models. The incorporation
of SAM for extracting semantic priors considerably hampers the model inference
efficiency. To address this issue, we propose a general framework to distill
SAM's semantic knowledge to boost exiting IR models without interfering with
their inference process. Specifically, our proposed framework consists of the
semantic priors fusion (SPF) scheme and the semantic priors distillation (SPD)
scheme. SPF fuses two kinds of information between the restored image predicted
by the original IR model and the semantic mask predicted by SAM for the refined
restored image. SPD leverages a self-distillation manner to distill the fused
semantic priors to boost the performance of original IR models. Additionally,
we design a semantic-guided relation (SGR) module for SPD, which ensures
semantic feature representation space consistency to fully distill the priors.
We demonstrate the effectiveness of our framework across multiple IR models and
tasks, including deraining, deblurring, and denoising.

通过融合和提取语义先验，我们提出了一个通用框架来改善图像恢复任务中的效果，同时提高模型的推理效率，并通过引入语义引导的关系模块来进一步改善恢复模型的性能。