Image matching is still challenging in such scenes with large viewpoints or
illumination changes or with low textures. In this paper, we propose a
Transformer-based pseudo 3D image matching method. It upgrades the 2D features
extracted from the source image to 3D features with the help of a reference
image and matches to the 2D features extracted from the destination image by
the coarse-to-fine 3D matching. Our key discovery is that by introducing the
reference image, the source image's fine points are screened and furtherly
their feature descriptors are enriched from 2D to 3D, which improves the match
performance with the destination image. Experimental results on multiple
datasets show that the proposed method achieves the state-of-the-art on the
tasks of homography estimation, pose estimation and visual localization
especially in challenging scenes.

本论文提出了一种基于 Transformer 的伪三维图像匹配方法，通过引入参考图像将源图像的 2D 特征升级为 3D 特征，并通过由粗到细的三维匹配与目标图像的 2D 特征进行匹配，从而提高了在具有大视角或光照变化或低纹理的场景中的匹配性能。实验结果表明，所提出的方法在同态估计、姿态估计和视觉定位任务中，在具有挑战性的场景中达到了最先进的水平。

TP3M: 基于 Transformer 的伪 3D 图像匹配带参考

TP3M: Transformer-based Pseudo 3D Image Matching with Reference

Existing optical flow methods are erroneous in challenging scenes, such as
fog, rain, and night because the basic optical flow assumptions such as
brightness and gradient constancy are broken. To address this problem, we
present an unsupervised learning approach that fuses gyroscope into optical
flow learning. Specifically, we first convert gyroscope readings into motion
fields named gyro field. Second, we design a self-guided fusion module to fuse
the background motion extracted from the gyro field with the optical flow and
guide the network to focus on motion details. To the best of our knowledge,
this is the first deep learning-based framework that fuses gyroscope data and
image content for optical flow learning. To validate our method, we propose a
new dataset that covers regular and challenging scenes. Experiments show that
our method outperforms the state-of-art methods in both regular and challenging
scenes. Code and dataset are available at
this https URL

本论文提出了一种融合了陀螺仪数据和图像内容进行光流学习的基于深度学习的框架，并设计了自我引导融合模块来引导网络集中在运动细节上。实验证明，该方法在正常和困难情况下均优于现有方法。