In this paper, we propose a geometric neural network with edge-aware
refinement (GeoNet++) to jointly predict both depth and surface normal maps
from a single image. Building on top of two-stream CNNs, GeoNet++ captures the
geometric relationships between depth and surface normals with the proposed
depth-to-normal and normal-to-depth modules. In particular, the
"depth-to-normal" module exploits the least square solution of estimating
surface normals from depth to improve their quality, while the
"normal-to-depth" module refines the depth map based on the constraints on
surface normals through kernel regression. Boundary information is exploited
via an edge-aware refinement module. GeoNet++ effectively predicts depth and
surface normals with strong 3D consistency and sharp boundaries resulting in
better reconstructed 3D scenes. Note that GeoNet++ is generic and can be used
in other depth/normal prediction frameworks to improve the quality of 3D
reconstruction and pixel-wise accuracy of depth and surface normals.
Furthermore, we propose a new 3D geometric metric (3DGM) for evaluating depth
prediction in 3D. In contrast to current metrics that focus on evaluating
pixel-wise error/accuracy, 3DGM measures whether the predicted depth can
reconstruct high-quality 3D surface normals. This is a more natural metric for
many 3D application domains. Our experiments on NYUD-V2 and KITTI datasets
verify that GeoNet++ produces fine boundary details, and the predicted depth
can be used to reconstruct high-quality 3D surfaces. Code has been made
publicly available.

本文提出一个新的几何神经网络 GeoNet++，用于从单张图像中联合预测深度和表面法向图，并通过边缘感知提升预测质量。此网络可用于其他深度 / 法向预测框架中，提高了在 3D 重建应用领域的预测质量和像素级精度，并提出了高质量 3D 表面深度预测的新几何度量。