We introduce, XoFTR, a cross-modal cross-view method for local feature
matching between thermal infrared (TIR) and visible images. Unlike visible
images, TIR images are less susceptible to adverse lighting and weather
conditions but present difficulties in matching due to significant texture and
intensity differences. Current hand-crafted and learning-based methods for
visible-TIR matching fall short in handling viewpoint, scale, and texture
diversities. To address this, XoFTR incorporates masked image modeling
pre-training and fine-tuning with pseudo-thermal image augmentation to handle
the modality differences. Additionally, we introduce a refined matching
pipeline that adjusts for scale discrepancies and enhances match reliability
through sub-pixel level refinement. To validate our approach, we collect a
comprehensive visible-thermal dataset, and show that our method outperforms
existing methods on many benchmarks.

我们介绍了一种跨模态跨视角方法，用于热红外和可见图像之间的局部特征匹配。该方法通过引入掩膜图像建模预训练和微调，并加入伪热图像增强解决了模态差异的问题，同时通过子像素级的细化调整解决了尺度差异和增强匹配的可靠性。我们通过一个综合的可见 - 热红外数据集验证了我们的方法，并展示了在多项基准测试中我们方法的优越性。

跨模态特征匹配转换器

XoFTR: Cross-modal Feature Matching Transformer

In recent years, deep learning has become a very active research tool which
is used in many image processing fields. In this paper, we propose an effective
image fusion method using a deep learning framework to generate a single image
which contains all the features from infrared and visible images. First, the
source images are decomposed into base parts and detail content. Then the base
parts are fused by weighted-averaging. For the detail content, we use a deep
learning network to extract multi-layer features. Using these features, we use
l_1-norm and weighted-average strategy to generate several candidates of the
fused detail content. Once we get these candidates, the max selection strategy
is used to get final fused detail content. Finally, the fused image will be
reconstructed by combining the fused base part and detail content. The
experimental results demonstrate that our proposed method achieves
state-of-the-art performance in both objective assessment and visual quality.
The Code of our fusion method is available at
this https URL

本文提出了一种基于深度学习框架的有效图像融合方法，该方法能够将红外图像和可见光图像的所有特征生成单个图像。实验结果表明，该方法在客观评估和视觉质量方面均达到了最先进的性能。