Self-supervised monocular scene flow estimation, aiming to understand both 3D
structures and 3D motions from two temporally consecutive monocular images, has
received increasing attention for its simple and economical sensor setup.
However, the accuracy of current methods suffers from the bottleneck of
less-efficient network architecture and lack of motion rigidity for
regularization. In this paper, we propose a superior model named EMR-MSF by
borrowing the advantages of network architecture design under the scope of
supervised learning. We further impose explicit and robust geometric
constraints with an elaborately constructed ego-motion aggregation module where
a rigidity soft mask is proposed to filter out dynamic regions for stable
ego-motion estimation using static regions. Moreover, we propose a motion
consistency loss along with a mask regularization loss to fully exploit static
regions. Several efficient training strategies are integrated including a
gradient detachment technique and an enhanced view synthesis process for better
performance. Our proposed method outperforms the previous self-supervised works
by a large margin and catches up to the performance of supervised methods. On
the KITTI scene flow benchmark, our approach improves the SF-all metric of the
state-of-the-art self-supervised monocular method by 44% and demonstrates
superior performance across sub-tasks including depth and visual odometry,
amongst other self-supervised single-task or multi-task methods.

我们提出了一种名为 EMR-MSF 的优越模型，通过借鉴监督学习范围内的网络架构设计，在两个时间连续的单目图像中理解三维结构和三维运动，并进一步通过精心构建的相机运动聚合模块塑造出显式和强健的几何约束；该模块提出了一个刚性软遮罩来过滤出稳定的自运动估计中的动态区域，并采用静态区域。此外，我们提出了一个运动一致性损失和一个遮罩正则化损失来充分利用静态区域。整合了几种高效的训练策略，包括梯度分离技术和增强的视角合成过程以获得更好的性能。我们的方法在 KITTI 场景流基准中将自监督单目方法的 SF-all 指标提高了 44％，并在深度和视觉里程计等子任务以及其他自监督单一任务或多任务方法中展现出卓越的性能。

EMR-MSF: 自监督循环单目场景流利用自运动刚体性

EMR-MSF: Self-Supervised Recurrent Monocular Scene Flow Exploiting  Ego-Motion Rigidity

In this paper, we study the problem of jointly estimating the optical flow
and scene flow from synchronized 2D and 3D data. Previous methods either employ
a complex pipeline that splits the joint task into independent stages, or fuse
2D and 3D information in an "early-fusion" or "late-fusion" manner. Such
one-size-fits-all approaches suffer from a dilemma of failing to fully utilize
the characteristic of each modality or to maximize the inter-modality
complementarity. To address the problem, we propose a novel end-to-end
framework, called CamLiFlow. It consists of 2D and 3D branches with multiple
bidirectional connections between them in specific layers. Different from
previous work, we apply a point-based 3D branch to better extract the geometric
features and design a symmetric learnable operator to fuse dense image features
and sparse point features. Experiments show that CamLiFlow achieves better
performance with fewer parameters. Our method ranks 1st on the KITTI Scene Flow
benchmark, outperforming the previous art with 1/7 parameters. Code is
available at this https URL

本文提出了一种新的端到端框架 CamLiFlow，充分利用 2D 和 3D 数据的特征，通过双向连接完成光流和场景流的估计，实验表明该方法在 KITTI Scene Flow 基准测试中性能表现优异。