Stereo matching has become a key technique for 3D environment perception in
intelligent vehicles. For a considerable time, convolutional neural networks
(CNNs) have remained the mainstream choice for feature extraction in this
domain. Nonetheless, there is a growing consensus that the existing paradigm
should evolve towards vision foundation models (VFM), particularly those
developed based on vision Transformers (ViTs) and pre-trained through
self-supervision on extensive, unlabeled datasets. While VFMs are adept at
extracting informative, general-purpose visual features, specifically for dense
prediction tasks, their performance often lacks in geometric vision tasks. This
study serves as the first exploration of a viable approach for adapting VFMs to
stereo matching. Our ViT adapter, referred to as ViTAS, is constructed upon
three types of modules: spatial differentiation, patch attention fusion, and
cross-attention. The first module initializes feature pyramids, while the
latter two aggregate stereo and multi-scale contextual information into
fine-grained features, respectively. ViTAStereo, which combines ViTAS with cost
volume-based stereo matching back-end processes, achieves the top rank on the
KITTI Stereo 2012 dataset and outperforms the second-best network StereoBase by
approximately 7.9% in terms of the percentage of error pixels, with a tolerance
of 3 pixels. Additional experiments across diverse scenarios further
demonstrate its superior generalizability compared to all other
state-of-the-art approaches. We believe this new paradigm will pave the way for
the next generation of stereo matching networks.

该研究是首次探索一种可行的方法，将 VFMs 适应于立体匹配，并通过将 ViTAS 与基于代价体积的立体匹配后端处理相结合，在 KITTI Stereo 2012 数据集上取得了最高排名，并以误差像素比例方面优于次优网络 StereoBase 约 7.9％，3 个像素误差容差。在各种情境的附加实验进一步证明了其与所有其他最先进方法相比的卓越通用性。我们相信这种新范式将为下一代立体匹配网络铺平道路。

运用视觉基础模型在立体匹配中的优势

Playing to Vision Foundation Model's Strengths in Stereo Matching

We propose learning a depth covariance function with applications to
geometric vision tasks. Given RGB images as input, the covariance function can
be flexibly used to define priors over depth functions, predictive
distributions given observations, and methods for active point selection. We
leverage these techniques for a selection of downstream tasks: depth
completion, bundle adjustment, and monocular dense visual odometry.

我们提出了一种学习深度协方差函数的方法，并将其应用于几何视觉任务中，尤其在深度补全、捆集调整和单目稠密视觉里程计方面。