Recent advances have demonstrated the powerful capability of transformer
architecture in image restoration. However, our analysis indicates that
existing transformerbased methods can not establish both exact global and local
dependencies simultaneously, which are much critical to restore the details and
missing content of degraded images. To this end, we present an efficient image
processing transformer architecture with hierarchical attentions, called IPTV2,
adopting a focal context self-attention (FCSA) and a global grid self-attention
(GGSA) to obtain adequate token interactions in local and global receptive
fields. Specifically, FCSA applies the shifted window mechanism into the
channel self-attention, helps capture the local context and mutual interaction
across channels. And GGSA constructs long-range dependencies in the
cross-window grid, aggregates global information in spatial dimension.
Moreover, we introduce structural re-parameterization technique to feed-forward
network to further improve the model capability. Extensive experiments
demonstrate that our proposed IPT-V2 achieves state-of-the-art results on
various image processing tasks, covering denoising, deblurring, deraining and
obtains much better trade-off for performance and computational complexity than
previous methods. Besides, we extend our method to image generation as latent
diffusion backbone, and significantly outperforms DiTs.

近期的研究表明，基于 Transformer 架构的方法在图像恢复方面具有强大的能力。然而，我们的分析表明，现有的基于 Transformer 的方法无法同时建立精确的全局依赖关系和局部依赖关系，而这对于恢复退化图像的细节和缺失内容非常关键。为此，我们提出了一种具有分层注意力的高效图像处理 Transformer 架构，称为 IPTV2，采用了一种聚焦上下文自注意力（FCSA）和全局网格自注意力（GGSA），以在局部和全局感受野中获取充分的令牌交互。具体而言，FCSA 将移位窗口机制应用于通道自注意力中，有助于捕捉局部上下文和通道间的相互作用。GGSA 在跨窗格中构建长距离的依赖关系，在空间维度中聚合全局信息。此外，我们引入结构重新参数化技术来改进前馈网络的模型能力。大量实验证明，我们提出的 IPT-V2 在各种图像处理任务上取得了最先进的结果，包括去噪、去模糊、去雨等，并且在性能和计算复杂度的权衡方面获得了很大的改进。此外，我们将我们的方法扩展到图像生成作为潜在扩散的支撑，显著优于 DiTs。