Keypoint detection is a pivotal step in 3D reconstruction, whereby sets of
(up to) K points are detected in each view of a scene. Crucially, the detected
points need to be consistent between views, i.e., correspond to the same 3D
point in the scene. One of the main challenges with keypoint detection is the
formulation of the learning objective. Previous learning-based methods
typically jointly learn descriptors with keypoints, and treat the keypoint
detection as a binary classification task on mutual nearest neighbours.
However, basing keypoint detection on descriptor nearest neighbours is a proxy
task, which is not guaranteed to produce 3D-consistent keypoints. Furthermore,
this ties the keypoints to a specific descriptor, complicating downstream
usage. In this work, we instead learn keypoints directly from 3D consistency.
To this end, we train the detector to detect tracks from large-scale SfM. As
these points are often overly sparse, we derive a semi-supervised two-view
detection objective to expand this set to a desired number of detections. To
train a descriptor, we maximize the mutual nearest neighbour objective over the
keypoints with a separate network. Results show that our approach, DeDoDe,
achieves significant gains on multiple geometry benchmarks. Code is provided at
this https URL .

基于 3D 连续性，我们通过从大规模 SfM 中训练探测器来直接学习关键点，并且通过分离的网络最大化互相最近邻目标来训练描述符。结果表明，我们的方法 DeDoDe 在多个几何基准测试中取得了显著的增益。