Guided depth super-resolution (GDSR) involves restoring missing depth details
using the high-resolution RGB image of the same scene. Previous approaches have
struggled with the heterogeneity and complementarity of the multi-modal inputs,
and neglected the issues of modal misalignment, geometrical misalignment, and
feature selection. In this study, we rethink some essential components in GDSR
networks and propose a simple yet effective Dynamic Dual Alignment and
Aggregation network (D2A2). D2A2 mainly consists of 1) a dynamic dual alignment
module that adapts to alleviate the modal misalignment via a learnable domain
alignment block and geometrically align cross-modal features by learning the
offset; and 2) a mask-to-pixel feature aggregate module that uses the gated
mechanism and pixel attention to filter out irrelevant texture noise from RGB
features and combine the useful features with depth features. By combining the
strengths of RGB and depth features while minimizing disturbance introduced by
the RGB image, our method with simple reuse and redesign of basic components
achieves state-of-the-art performance on multiple benchmark datasets. The code
is available at this https URL

通过重新审视 GDSR 网络的一些关键组件并提出简单但有效的 Dynamic Dual Alignment and Aggregation 网络（D2A2），我们结合 RGB 和深度特征的优势，同时最小化 RGB 图像引入的干扰，通过对模态不匹配进行学习的领域对齐块和学习偏移来减轻模态不匹配的动态双重对齐模块，并通过使用门控机制和像素注意力来过滤 IR 纹理噪声并将有用的特征与深度特征结合起来的掩码到像素特征聚合模块，我们的方法在多个基准数据集上实现了最先进的性能。

细节之中的魔鬼：通过重新思考跨模态对齐和聚合来提升引导式深度超分辨率

The Devil is in the Details: Boosting Guided Depth Super-Resolution via  Rethinking Cross-Modal Alignment and Aggregation

This work aims at designing a lightweight convolutional neural network for
image super resolution (SR). With simplicity bare in mind, we construct a
pretty concise and effective network with a newly proposed pixel attention
scheme. Pixel attention (PA) is similar as channel attention and spatial
attention in formulation. The difference is that PA produces 3D attention maps
instead of a 1D attention vector or a 2D map. This attention scheme introduces
fewer additional parameters but generates better SR results. On the basis of
PA, we propose two building blocks for the main branch and the reconstruction
branch, respectively. The first one - SC-PA block has the same structure as the
Self-Calibrated convolution but with our PA layer. This block is much more
efficient than conventional residual/dense blocks, for its twobranch
architecture and attention scheme. While the second one - UPA block combines
the nearest-neighbor upsampling, convolution and PA layers. It improves the
final reconstruction quality with little parameter cost. Our final model- PAN
could achieve similar performance as the lightweight networks - SRResNet and
CARN, but with only 272K parameters (17.92% of SRResNet and 17.09% of CARN).
The effectiveness of each proposed component is also validated by ablation
study. The code is available at this https URL

本研究旨在设计一种轻量级卷积神经网络，用于图像超分辨率，提出了一种像素关注机制用于构建简洁且有效的网络，并分别提出了基于像素关注的主干和重建分支的构建块。该模型具有较好效果且参数量只有 SRResNet 和 CARN 的 17.92％和 17.09％。