Stereo matching has become a key technique for 3D environment perception in
intelligent vehicles. For a considerable time, convolutional neural networks
(CNNs) have remained the mainstream choice for feature extraction in this
domain. Nonetheless, there is a growing consensus that the existing paradigm
should evolve towards vision foundation models (VFM), particularly those
developed based on vision Transformers (ViTs) and pre-trained through
self-supervision on extensive, unlabeled datasets. While VFMs are adept at
extracting informative, general-purpose visual features, specifically for dense
prediction tasks, their performance often lacks in geometric vision tasks. This
study serves as the first exploration of a viable approach for adapting VFMs to
stereo matching. Our ViT adapter, referred to as ViTAS, is constructed upon
three types of modules: spatial differentiation, patch attention fusion, and
cross-attention. The first module initializes feature pyramids, while the
latter two aggregate stereo and multi-scale contextual information into
fine-grained features, respectively. ViTAStereo, which combines ViTAS with cost
volume-based stereo matching back-end processes, achieves the top rank on the
KITTI Stereo 2012 dataset and outperforms the second-best network StereoBase by
approximately 7.9% in terms of the percentage of error pixels, with a tolerance
of 3 pixels. Additional experiments across diverse scenarios further
demonstrate its superior generalizability compared to all other
state-of-the-art approaches. We believe this new paradigm will pave the way for
the next generation of stereo matching networks.

该研究是首次探索一种可行的方法，将 VFMs 适应于立体匹配，并通过将 ViTAS 与基于代价体积的立体匹配后端处理相结合，在 KITTI Stereo 2012 数据集上取得了最高排名，并以误差像素比例方面优于次优网络 StereoBase 约 7.9％，3 个像素误差容差。在各种情境的附加实验进一步证明了其与所有其他最先进方法相比的卓越通用性。我们相信这种新范式将为下一代立体匹配网络铺平道路。

运用视觉基础模型在立体匹配中的优势

Playing to Vision Foundation Model's Strengths in Stereo Matching

Vision transformers (ViTs) inherited the success of NLP but their structures
have not been sufficiently investigated and optimized for visual tasks. One of
the simplest solutions is to directly search the optimal one via the widely
used neural architecture search (NAS) in CNNs. However, we empirically find
this straightforward adaptation would encounter catastrophic failures and be
frustratingly unstable for the training of superformer. In this paper, we argue
that since ViTs mainly operate on token embeddings with little inductive bias,
imbalance of channels for different architectures would worsen the
weight-sharing assumption and cause the training instability as a result.
Therefore, we develop a new cyclic weight-sharing mechanism for token
embeddings of the ViTs, which enables each channel could more evenly contribute
to all candidate architectures. Besides, we also propose identity shifting to
alleviate the many-to-one issue in superformer and leverage weak augmentation
and regularization techniques for more steady training empirically. Based on
these, our proposed method, ViTAS, has achieved significant superiority in both
DeiT- and Twins-based ViTs. For example, with only $1.4$G FLOPs budget, our
searched architecture has $3.3\%$ ImageNet-$1$k accuracy than the baseline
DeiT. With $3.0$G FLOPs, our results achieve $82.0\%$ accuracy on
ImageNet-$1$k, and $45.9\%$ mAP on COCO$2017$ which is $2.4\%$ superior than
other ViTs.

本文提出了 ViTAS 方法，其中使用循环权重共享机制和身份移位来解决目标嵌入的失衡问题，并辅以弱数据增强和规范化技术，以获得稳定的训练结果。实验证明，与其他常用方法相比，ViTAS 取得了极高的性能提升。