In this paper, we propose a unified method to jointly learn optical flow and stereo matching. Our first intuition is stereo matching can be modeled as a special case of optical flow, and we can leverage 3D geometry behind stereoscopic videos to guide the learning of these two forms of correspondences. We then enroll this knowledge into the state-of-the-art self-supervised learning framework, and train one single network to estimate both flow and stereo. Second, we unveil the bottlenecks in prior self-supervised learning approaches, and propose to create a new set of challenging proxy tasks to boost performance. These two insights yield a single model that achieves the highest accuracy among all existing unsupervised flow and stereo methods on KITTI 2012 and 2015 benchmarks. More remarkably, our self-supervised method even outperforms several state-of-the-art fully supervised methods, including PWC-Net and FlowNet2 on KITTI 2012.

我们提出了一种统一的方法来共同学习光流和立体匹配。我们的第一条直觉是，立体匹配可以被建模为光流的一种特殊情况，我们可以利用立体视频背后的三维几何来指导这两种形式的对应关系的学习。然后，我们将这个知识纳入到最先进的自我监督学习框架中，并训练一个单一的网络来估计流和立体。其次，我们揭示了先前自我监督学习方法中的瓶颈，并提出了创建一组新的具有挑战性的代理任务来提高性能的方法。这两个洞察力产生了一个单一的模型，在KITTI 2012和2015基准测试中，这些模型的准确性甚至超过了包括PWC-Net和FlowNet2在内的几种最先进的全监督方法。

Flow2Stereo：光流和立体匹配的有效自监督学习