This paper addresses the challenging unsupervised scene flow estimation
problem by jointly learning four low-level vision sub-tasks: optical flow
$\textbf{F}$, stereo-depth $\textbf{D}$, camera pose $\textbf{P}$ and motion
segmentation $\textbf{S}$. Our key insight is that the rigidity of the scene
shares the same inherent geometrical structure with object movements and scene
depth. Hence, rigidity from $\textbf{S}$ can be inferred by jointly coupling
$\textbf{F}$, $\textbf{D}$ and $\textbf{P}$ to achieve more robust estimation.
To this end, we propose a novel scene flow framework named EffiScene with
efficient joint rigidity learning, going beyond the existing pipeline with
independent auxiliary structures. In EffiScene, we first estimate optical flow
and depth at the coarse level and then compute camera pose by
Perspective-$n$-Points method. To jointly learn local rigidity, we design a
novel Rigidity From Motion (RfM) layer with three principal components:
\emph{}{(i)} correlation extraction; \emph{}{(ii)} boundary learning; and
\emph{}{(iii)} outlier exclusion. Final outputs are fused based on the rigid
map $M_R$ from RfM at finer levels. To efficiently train EffiScene, two new
losses $\mathcal{L}_{bnd}$ and $\mathcal{L}_{unc}$ are designed to prevent
trivial solutions and to regularize the flow boundary discontinuity. Extensive
experiments on scene flow benchmark KITTI show that our method is effective and
significantly improves the state-of-the-art approaches for all sub-tasks, i.e.
optical flow ($5.19 \rightarrow 4.20$), depth estimation ($3.78 \rightarrow
3.46$), visual odometry ($0.012 \rightarrow 0.011$) and motion segmentation
($0.57 \rightarrow 0.62$).

本论文提出一种 EffiScene 场景流框架，通过 Rigidity from Motion 层的学习，同时联合学习光流、深度、相机姿态和运动分割 4 个底层视觉子任务，有效地解决了无监督场景流估计问题。实验证明，该方法在 KITTI 基准测试中，明显提高了所有子任务的表现。