Recent advances have demonstrated the powerful capability of transformer
architecture in image restoration. However, our analysis indicates that
existing transformerbased methods can not establish both exact global and local
dependencies simultaneously, which are much critical to restore the details and
missing content of degraded images. To this end, we present an efficient image
processing transformer architecture with hierarchical attentions, called IPTV2,
adopting a focal context self-attention (FCSA) and a global grid self-attention
(GGSA) to obtain adequate token interactions in local and global receptive
fields. Specifically, FCSA applies the shifted window mechanism into the
channel self-attention, helps capture the local context and mutual interaction
across channels. And GGSA constructs long-range dependencies in the
cross-window grid, aggregates global information in spatial dimension.
Moreover, we introduce structural re-parameterization technique to feed-forward
network to further improve the model capability. Extensive experiments
demonstrate that our proposed IPT-V2 achieves state-of-the-art results on
various image processing tasks, covering denoising, deblurring, deraining and
obtains much better trade-off for performance and computational complexity than
previous methods. Besides, we extend our method to image generation as latent
diffusion backbone, and significantly outperforms DiTs.

近期的研究表明，基于 Transformer 架构的方法在图像恢复方面具有强大的能力。然而，我们的分析表明，现有的基于 Transformer 的方法无法同时建立精确的全局依赖关系和局部依赖关系，而这对于恢复退化图像的细节和缺失内容非常关键。为此，我们提出了一种具有分层注意力的高效图像处理 Transformer 架构，称为 IPTV2，采用了一种聚焦上下文自注意力（FCSA）和全局网格自注意力（GGSA），以在局部和全局感受野中获取充分的令牌交互。具体而言，FCSA 将移位窗口机制应用于通道自注意力中，有助于捕捉局部上下文和通道间的相互作用。GGSA 在跨窗格中构建长距离的依赖关系，在空间维度中聚合全局信息。此外，我们引入结构重新参数化技术来改进前馈网络的模型能力。大量实验证明，我们提出的 IPT-V2 在各种图像处理任务上取得了最先进的结果，包括去噪、去模糊、去雨等，并且在性能和计算复杂度的权衡方面获得了很大的改进。此外，我们将我们的方法扩展到图像生成作为潜在扩散的支撑，显著优于 DiTs。

IPT-V2: 高效的图像处理转换器使用层次注意力

IPT-V2: Efficient Image Processing Transformer using Hierarchical  Attentions

Text-driven diffusion models have become increasingly popular for various
image editing tasks, including inpainting, stylization, and object replacement.
However, it still remains an open research problem to adopt this
language-vision paradigm for more fine-level image processing tasks, such as
denoising, super-resolution, deblurring, and compression artifact removal. In
this paper, we develop TIP, a Text-driven Image Processing framework that
leverages natural language as a user-friendly interface to control the image
restoration process. We consider the capacity of text information in two
dimensions. First, we use content-related prompts to enhance the semantic
alignment, effectively alleviating identity ambiguity in the restoration
outcomes. Second, our approach is the first framework that supports fine-level
instruction through language-based quantitative specification of the
restoration strength, without the need for explicit task-specific design. In
addition, we introduce a novel fusion mechanism that augments the existing
ControlNet architecture by learning to rescale the generative prior, thereby
achieving better restoration fidelity. Our extensive experiments demonstrate
the superior restoration performance of TIP compared to the state of the arts,
alongside offering the flexibility of text-based control over the restoration
effects.

开发了一个基于文本的图像处理框架（TIP），可以通过自然语言作为用户友好界面来控制图像恢复过程，并通过语言的定量规范实现细粒度的指导，以达到更好的恢复性能。

基于文本的图像处理方法：语义与恢复指导

TIP: Text-Driven Image Processing with Semantic and Restoration  Instructions

Recent advances in image data processing through machine learning and
especially deep neural networks (DNNs) allow for new optimization and
performance-enhancement schemes for radiation detectors and imaging hardware
through data-endowed artificial intelligence. We give an overview of data
generation at photon sources, deep learning-based methods for image processing
tasks, and hardware solutions for deep learning acceleration. Most existing
deep learning approaches are trained offline, typically using large amounts of
computational resources. However, once trained, DNNs can achieve fast inference
speeds and can be deployed to edge devices. A new trend is edge computing with
less energy consumption (hundreds of watts or less) and real-time analysis
potential. While popularly used for edge computing, electronic-based hardware
accelerators ranging from general purpose processors such as central processing
units (CPUs) to application-specific integrated circuits (ASICs) are constantly
reaching performance limits in latency, energy consumption, and other physical
constraints. These limits give rise to next-generation analog neuromorhpic
hardware platforms, such as optical neural networks (ONNs), for high parallel,
low latency, and low energy computing to boost deep learning acceleration.

最近在图像数据处理方面的研究进展表明，通过机器学习，尤其是深度神经网络（DNNs）的使用，可以通过数据驱动的人工智能为辐射探测器和成像设备提供新的优化和性能增强方案。我们概述了光子源的数据生成、基于深度学习的图像处理方法以及深度学习加速的硬件解决方案。目前大多数现有的深度学习方法是离线训练的，通常需要大量的计算资源。然而，一旦训练完成，DNNs 可以实现快速推理速度，并且可以部署到边缘设备上。边缘计算是一种新趋势，它具有较低的能量消耗（数百瓦或更少）和实时分析能力。虽然以往通常用于边缘计算的基于电子的硬件加速器（从中央处理器（CPU）到应用特定集成电路（ASICs）的通用目的处理器）正不断接近性能极限，在延迟、能量消耗和其他物理约束方面存在限制。这些限制催生了下一代模拟神经形态硬件平台，如光学神经网络（ONNs），用于高并行、低延迟和低能量计算，以提高深度学习加速。