In the context of image-to-point cloud registration, acquiring point-to-pixel
correspondences presents a challenging task since the similarity between
individual points and pixels is ambiguous due to the visual differences in data
modalities. Nevertheless, the same object present in the two data formats can
be readily identified from the local perspective of point sets and pixel
patches. Motivated by this intuition, we propose a coarse-to-fine framework
that emphasizes the establishment of correspondences between local point sets
and pixel patches, followed by the refinement of results at both the point and
pixel levels. On a coarse scale, we mimic the classic Visual Transformer to
translate both image and point cloud into two sequences of local
representations, namely point and pixel proxies, and employ attention to
capture global and cross-modal contexts. To supervise the coarse matching, we
propose a novel projected point proportion loss, which guides to match point
sets with pixel patches where more points can be projected into. On a finer
scale, point-to-pixel correspondences are then refined from a smaller search
space (i.e., the coarsely matched sets and patches) via well-designed sampling,
attentional learning and fine matching, where sampling masks are embedded in
the last two steps to mitigate the negative effect of sampling. With the
high-quality correspondences, the registration problem is then resolved by EPnP
algorithm within RANSAC. Experimental results on large-scale outdoor benchmarks
demonstrate our superiority over existing methods.

本研究提出了一种基于粗 - 精匹配的框架，在该框架中，将图片和点云转化为局部表示，并利用了 attention 来捕捉全局和跨模态上下文，并通过设计良好的采样和精细匹配来细化点到像素的对应关系，从而解决了图像与点云匹配的问题，并在大规模室外数据集上取得了优于现有方法的结果。