Recently, feature relation learning has drawn widespread attention in
cross-spectral image patch matching. However, existing related research focuses
on extracting diverse relations between image patch features and ignores
sufficient intrinsic feature representations of individual image patches.
Therefore, an innovative relational representation learning idea is proposed
for the first time, which simultaneously focuses on sufficiently mining the
intrinsic features of individual image patches and the relations between image
patch features. Based on this, we construct a lightweight Relational
Representation Learning Network (RRL-Net). Specifically, we innovatively
construct an autoencoder to fully characterize the individual intrinsic
features, and introduce a Feature Interaction Learning (FIL) module to extract
deep-level feature relations. To further fully mine individual intrinsic
features, a lightweight Multi-dimensional Global-to-Local Attention (MGLA)
module is constructed to enhance the global feature extraction of individual
image patches and capture local dependencies within global features. By
combining the MGLA module, we further explore the feature extraction network
and construct an Attention-based Lightweight Feature Extraction (ALFE) network.
In addition, we propose a Multi-Loss Post-Pruning (MLPP) optimization strategy,
which greatly promotes network optimization while avoiding increases in
parameters and inference time. Extensive experiments demonstrate that our
RRL-Net achieves state-of-the-art (SOTA) performance on multiple public
datasets. Our code will be made public later.

通过创新的关系表示学习方法，同时关注个体内在特征和图像补丁特征之间的关系，构建了一种轻量级的关系表示学习网络（RRL-Net），包括自动编码器、特征交互学习模块、多维全局到局部注意力机制以及基于注意力的轻量级特征提取网络。经过大量实验验证，该网络在多个公共数据集上取得了最先进的性能。

跨光谱图像块匹配的关系表示学习网络

Relational Representation Learning Network for Cross-Spectral Image  Patch Matching

Recently, learning-based algorithms have achieved promising performance on
cross-spectral image patch matching, which, however, is still far from
satisfactory for practical application. On the one hand, a lack of large-scale
dataset with diverse scenes haunts its further improvement for learning-based
algorithms, whose performances and generalization rely heavily on the dataset
size and diversity. On the other hand, more emphasis has been put on feature
relation in the spatial domain whereas the scale dependency between features
has often been ignored, leading to performance degeneration especially when
encountering significant appearance variations for cross-spectral patches. To
address these issues, we publish, to be best of our knowledge, the largest
visible and Long-wave Infrared (LWIR) image patch matching dataset, termed
VL-CMIM, which contains 1300 pairs of strictly aligned visible and LWIR images
and over 2 million patch pairs covering diverse scenes such as asteroid, field,
country, build, street and water.In addition, a multi-domain feature relation
learning network (MD-FRN) is proposed. Input by the features extracted from a
four-branch network, both feature relations in spatial and scale domains are
learned via a spatial correlation module (SCM) and multi-scale adaptive
aggregation module (MSAG), respectively. To further aggregate the multi-domain
relations, a deep domain interactive mechanism (DIM) is applied, where the
learnt spatial-relation and scale-relation features are exchanged and further
input into MSCRM and SCM. This mechanism allows our model to learn interactive
cross-domain feature relations, leading to improved robustness to significant
appearance changes due to different modality.

学习算法在交叉光谱图像块匹配方面取得了有希望的性能，然而在实际应用中仍然存在不足。我们发布了迄今为止最大的可见光和长波红外图像块匹配数据集 VL-CMIM，其中包含 1300 对严格对准的可见光和红外图像，覆盖了包括星际、田地、乡村、建筑、街道和水域等不同场景的超过 200 万个块对。此外，我们提出了一种多域特征关系学习网络（MD-FRN），通过四分支网络提取的特征来学习空间域和尺度域的特征关系。为了更进一步聚合多域关系，我们应用了深度域交互机制（DIM），其中学到的空间关系和尺度关系特征被交换并进一步输入到 MSCRM 和 SCM。通过学习交互式跨域特征关系，我们的模型能够更好地应对不同模态导致的显著外观变化，从而提高了稳健性。