Cross-spectral image guided denoising has shown its great potential in
recovering clean images with rich details, such as using the near-infrared
image to guide the denoising process of the visible one. To obtain such image
pairs, a feasible and economical way is to employ a stereo system, which is
widely used on mobile devices. Current works attempt to generate an aligned
guidance image to handle the disparity between two images. However, due to
occlusion, spectral differences and noise degradation, the aligned guidance
image generally exists ghosting and artifacts, leading to an unsatisfactory
denoised result. To address this issue, we propose a one-stage
transformer-based architecture, named SGDFormer, for cross-spectral Stereo
image Guided Denoising. The architecture integrates the correspondence modeling
and feature fusion of stereo images into a unified network. Our transformer
block contains a noise-robust cross-attention (NRCA) module and a spatially
variant feature fusion (SVFF) module. The NRCA module captures the long-range
correspondence of two images in a coarse-to-fine manner to alleviate the
interference of noise. The SVFF module further enhances salient structures and
suppresses harmful artifacts through dynamically selecting useful information.
Thanks to the above design, our SGDFormer can restore artifact-free images with
fine structures, and achieves state-of-the-art performance on various datasets.
Additionally, our SGDFormer can be extended to handle other unaligned
cross-model guided restoration tasks such as guided depth super-resolution.

我们提出了一种名为 SGDFormer 的单阶段变压器架构，用于交叉光谱立体图像引导去噪，该架构将立体图像的对应建模和特征融合集成到一个统一的网络中，以恢复没有伪影和良好细节结构的图像，并在各种数据集上取得了最先进的性能。

SGDFormer：一阶段基于 Transformer 的跨光谱立体图像引导去噪架构

SGDFormer: One-stage Transformer-based Architecture for Cross-Spectral  Stereo Image Guided Denoising

It is well known that the passive stereo system cannot adapt well to weak
texture objects, e.g., white walls. However, these weak texture targets are
very common in indoor environments. In this paper, we present a novel stereo
system, which consists of two cameras (an RGB camera and an IR camera) and an
IR speckle projector. The RGB camera is used both for depth estimation and
texture acquisition. The IR camera and the speckle projector can form a
monocular structured-light (MSL) subsystem, while the two cameras can form a
binocular stereo subsystem. The depth map generated by the MSL subsystem can
provide external guidance for the stereo matching networks, which can improve
the matching accuracy significantly. In order to verify the effectiveness of
the proposed system, we build a prototype and collect a test dataset in indoor
scenes. The evaluation results show that the Bad 2.0 error of the proposed
system is 28.2% of the passive stereo system when the network RAFT is used. The
dataset and trained models are available at
this https URL

本文介绍了一种新型立体系统，利用 IR 相机和结构光方法提高了匹配精度，通过在室内场景中的测试数据证明了该系统的有效性。