Despite substantial progress, all-in-one image restoration (IR) grapples with
persistent challenges in handling intricate real-world degradations. This paper
introduces MPerceiver: a novel multimodal prompt learning approach that
harnesses Stable Diffusion (SD) priors to enhance adaptiveness,
generalizability and fidelity for all-in-one image restoration. Specifically,
we develop a dual-branch module to master two types of SD prompts: textual for
holistic representation and visual for multiscale detail representation. Both
prompts are dynamically adjusted by degradation predictions from the CLIP image
encoder, enabling adaptive responses to diverse unknown degradations. Moreover,
a plug-in detail refinement module improves restoration fidelity via direct
encoder-to-decoder information transformation. To assess our method, MPerceiver
is trained on 9 tasks for all-in-one IR and outperforms state-of-the-art
task-specific methods across most tasks. Post multitask pre-training,
MPerceiver attains a generalized representation in low-level vision, exhibiting
remarkable zero-shot and few-shot capabilities in unseen tasks. Extensive
experiments on 16 IR tasks and 26 benchmarks underscore the superiority of
MPerceiver in terms of adaptiveness, generalizability and fidelity.

本研究介绍了 MPerceiver：一种新颖的多模态提示学习方法，利用稳定扩散先验增强了全方位图像修复的适应性、普适性和保真性。通过开发双分支模块，该方法掌握了两种类型的稳定扩散提示：文本提示用于整体表示和视觉提示用于多尺度细节表示。MPerceiver 使用 CLIP 图像编码器的退化预测动态调整这两种提示，从而能够对不同的未知退化进行自适应响应。此外，插件细节优化模块通过直接的编码器到解码器信息转换提高了修复保真性。实验证明，MPerceiver 在全方位图像修复的 9 个任务上经过多任务预训练，优于大多数专门任务的特定方法。在低级视觉方面，MPerceiver 表现出出色的零样本和少样本能力。在 16 个图像修复任务和 26 个基准测试上的大量实验证实了 MPerceiver 在适应性、普适性和保真性方面的优越性。

多模态提示感知器：全能图像恢复的适应性、普适性和保真度

Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and  Fidelity for All-in-One Image Restoration

Face images captured in heterogeneous environments, e.g., sketches generated
by the artists or composite-generation software, photos taken by common cameras
and infrared images captured by corresponding infrared imaging devices, usually
subject to large texture (i.e., style) differences. This results in heavily
degraded performance of conventional face recognition methods in comparison
with the performance on images captured in homogeneous environments. In this
paper, we propose a novel sparse graphical representation based discriminant
analysis (SGR-DA) approach to address aforementioned face recognition in
heterogeneous scenarios. An adaptive sparse graphical representation scheme is
designed to represent heterogeneous face images, where a Markov networks model
is constructed to generate adaptive sparse vectors. To handle the complex
facial structure and further improve the discriminability, a spatial
partition-based discriminant analysis framework is presented to refine the
adaptive sparse vectors for face matching. We conducted experiments on six
commonly used heterogeneous face datasets and experimental results illustrate
that our proposed SGR-DA approach achieves superior performance in comparison
with state-of-the-art methods.

本文提出一种基于稀疏图形表示的判别分析方法，针对异构人脸识别场景下出现的纹理差异等问题，设计了自适应的稀疏图形表示方案，通过 Markov 网络生成自适应稀疏向量；并引入基于空间分块的判别分析方法以优化自适应稀疏向量，实验证明该方法在 6 个常用异构人脸数据集上达到了优异表现。