Image Matching is a core component of all best-performing algorithms and
pipelines in 3D vision. Yet despite matching being fundamentally a 3D problem,
intrinsically linked to camera pose and scene geometry, it is typically treated
as a 2D problem. This makes sense as the goal of matching is to establish
correspondences between 2D pixel fields, but also seems like a potentially
hazardous choice. In this work, we take a different stance and propose to cast
matching as a 3D task with DUSt3R, a recent and powerful 3D reconstruction
framework based on Transformers. Based on pointmaps regression, this method
displayed impressive robustness in matching views with extreme viewpoint
changes, yet with limited accuracy. We aim here to improve the matching
capabilities of such an approach while preserving its robustness. We thus
propose to augment the DUSt3R network with a new head that outputs dense local
features, trained with an additional matching loss. We further address the
issue of quadratic complexity of dense matching, which becomes prohibitively
slow for downstream applications if not carefully treated. We introduce a fast
reciprocal matching scheme that not only accelerates matching by orders of
magnitude, but also comes with theoretical guarantees and, lastly, yields
improved results. Extensive experiments show that our approach, coined MASt3R,
significantly outperforms the state of the art on multiple matching tasks. In
particular, it beats the best published methods by 30% (absolute improvement)
in VCRE AUC on the extremely challenging Map-free localization dataset.

通过在 DUSt3R 网络中引入新的头部，并训练密集局部特征以及附加匹配损失，我们改进了匹配能力，同时保持其鲁棒性，取得了显著的结果改进，击败了现有方法，并在多个匹配任务中实现了 30％的绝对改善。

用 MASt3R 在 3D 中实现图像匹配

Grounding Image Matching in 3D with MASt3R

Multi-view stereo reconstruction (MVS) in the wild requires to first estimate
the camera parameters e.g. intrinsic and extrinsic parameters. These are
usually tedious and cumbersome to obtain, yet they are mandatory to triangulate
corresponding pixels in 3D space, which is the core of all best performing MVS
algorithms. In this work, we take an opposite stance and introduce DUSt3R, a
radically novel paradigm for Dense and Unconstrained Stereo 3D Reconstruction
of arbitrary image collections, i.e. operating without prior information about
camera calibration nor viewpoint poses. We cast the pairwise reconstruction
problem as a regression of pointmaps, relaxing the hard constraints of usual
projective camera models. We show that this formulation smoothly unifies the
monocular and binocular reconstruction cases. In the case where more than two
images are provided, we further propose a simple yet effective global alignment
strategy that expresses all pairwise pointmaps in a common reference frame. We
base our network architecture on standard Transformer encoders and decoders,
allowing us to leverage powerful pretrained models. Our formulation directly
provides a 3D model of the scene as well as depth information, but
interestingly, we can seamlessly recover from it, pixel matches, relative and
absolute camera. Exhaustive experiments on all these tasks showcase that the
proposed DUSt3R can unify various 3D vision tasks and set new SoTAs on
monocular/multi-view depth estimation as well as relative pose estimation. In
summary, DUSt3R makes many geometric 3D vision tasks easy.

DUSt3R 是一种新颖的范式，用于进行任意图像集合的密集和无约束立体三维重建，通过将成对的重建问题作为点图的回归来实现，进而实现单眼和双眼重建的统一，提供场景的三维模型和深度信息，并能够轻松恢复像素匹配、相对和绝对相机，DUSt3R 能够简化多种几何三维视觉任务。