Semantic segmentation and stereo matching are two essential components of 3D
environmental perception systems for autonomous driving. Nevertheless,
conventional approaches often address these two problems independently,
employing separate models for each task. This approach poses practical
limitations in real-world scenarios, particularly when computational resources
are scarce or real-time performance is imperative. Hence, in this article, we
introduce S$^3$M-Net, a novel joint learning framework developed to perform
semantic segmentation and stereo matching simultaneously. Specifically,
S$^3$M-Net shares the features extracted from RGB images between both tasks,
resulting in an improved overall scene understanding capability. This feature
sharing process is realized using a feature fusion adaption (FFA) module, which
effectively transforms the shared features into semantic space and subsequently
fuses them with the encoded disparity features. The entire joint learning
framework is trained by minimizing a novel semantic consistency-guided (SCG)
loss, which places emphasis on the structural consistency in both tasks.
Extensive experimental results conducted on the vKITTI2 and KITTI datasets
demonstrate the effectiveness of our proposed joint learning framework and its
superior performance compared to other state-of-the-art single-task networks.
Our project webpage is accessible at mias.group/S3M-Net.

本文介绍了一种新的联合学习框架 S^3M-Net，用于同时进行语义分割和立体匹配，通过特征共享和融合来提高整体场景理解能力，通过最小化语义一致性引导损失来训练整个联合学习框架，在 vKITTI2 和 KITTI 数据集上进行实验，证明了该框架的有效性和超越其他单任务网络的性能。