Guided depth super-resolution (GDSR) involves restoring missing depth details
using the high-resolution RGB image of the same scene. Previous approaches have
struggled with the heterogeneity and complementarity of the multi-modal inputs,
and neglected the issues of modal misalignment, geometrical misalignment, and
feature selection. In this study, we rethink some essential components in GDSR
networks and propose a simple yet effective Dynamic Dual Alignment and
Aggregation network (D2A2). D2A2 mainly consists of 1) a dynamic dual alignment
module that adapts to alleviate the modal misalignment via a learnable domain
alignment block and geometrically align cross-modal features by learning the
offset; and 2) a mask-to-pixel feature aggregate module that uses the gated
mechanism and pixel attention to filter out irrelevant texture noise from RGB
features and combine the useful features with depth features. By combining the
strengths of RGB and depth features while minimizing disturbance introduced by
the RGB image, our method with simple reuse and redesign of basic components
achieves state-of-the-art performance on multiple benchmark datasets. The code
is available at this https URL

通过重新审视 GDSR 网络的一些关键组件并提出简单但有效的 Dynamic Dual Alignment and Aggregation 网络（D2A2），我们结合 RGB 和深度特征的优势，同时最小化 RGB 图像引入的干扰，通过对模态不匹配进行学习的领域对齐块和学习偏移来减轻模态不匹配的动态双重对齐模块，并通过使用门控机制和像素注意力来过滤 IR 纹理噪声并将有用的特征与深度特征结合起来的掩码到像素特征聚合模块，我们的方法在多个基准数据集上实现了最先进的性能。

细节之中的魔鬼：通过重新思考跨模态对齐和聚合来提升引导式深度超分辨率

The Devil is in the Details: Boosting Guided Depth Super-Resolution via  Rethinking Cross-Modal Alignment and Aggregation

Guided depth super-resolution is a practical task where a low-resolution and
noisy input depth map is restored to a high-resolution version, with the help
of a high-resolution RGB guide image. Existing methods usually view this task
as a generalized guided filtering problem that relies on designing explicit
filters and objective functions, or a dense regression problem that directly
predicts the target image via deep neural networks. These methods suffer from
either model capability or interpretability. Inspired by the recent progress in
implicit neural representation, we propose to formulate the guided
super-resolution as a neural implicit image interpolation problem, where we
take the form of a general image interpolation but use a novel Joint Implicit
Image Function (JIIF) representation to learn both the interpolation weights
and values. JIIF represents the target image domain with spatially distributed
local latent codes extracted from the input image and the guide image, and uses
a graph attention mechanism to learn the interpolation weights at the same time
in one unified deep implicit function. We demonstrate the effectiveness of our
JIIF representation on guided depth super-resolution task, significantly
outperforming state-of-the-art methods on three public benchmarks. Code can be
found at https://git.io/JC2sU.

本文提出了一种将指导性先验信息引入深度超分辨率模型的方法，使用联合隐式函数 (JIIF) 来表示目标图像领域，并使用图注意力机制在联合深度隐式函数中学习插值权重，该方法在三个公共基准测试中显着优于现有方法。

引导深度超分辨率的联合隐式图像函数

Joint Implicit Image Function for Guided Depth Super-Resolution

Guided depth super-resolution (GDSR) is an essential topic in multi-modal
image processing, which reconstructs high-resolution (HR) depth maps from
low-resolution ones collected with suboptimal conditions with the help of HR
RGB images of the same scene. To solve the challenges in interpreting the
working mechanism, extracting cross-modal features and RGB texture
over-transferred, we propose a novel Discrete Cosine Transform Network (DCTNet)
to alleviate the problems from three aspects. First, the Discrete Cosine
Transform (DCT) module reconstructs the multi-channel HR depth features by
using DCT to solve the channel-wise optimization problem derived from the image
domain. Second, we introduce a semi-coupled feature extraction module that uses
shared convolutional kernels to extract common information and private kernels
to extract modality-specific information. Third, we employ an edge attention
mechanism to highlight the contours informative for guided upsampling.
Extensive quantitative and qualitative evaluations demonstrate the
effectiveness of our DCTNet, which outperforms previous state-of-the-art
methods with a relatively small number of parameters. The code is available at
https://github.com/Zhaozixiang1228/GDSR-DCTNet.

提出一种新颖的离散余弦变换网络（DCTNet）来解决多模态图像处理中导向深度超分辨率（GDSR）的挑战，包括理解工作机制、提取跨模态特征和减弱 RGB 纹理过度传递，通过离散余弦变换模块、半耦合特征提取模块和边缘注意机制来分别解决这些问题。实验结果表明，DCTNet 的性能比之前最先进的方法更好，在参数数量相对较小的情况下能够更好地处理导向深度超分辨率。