Recently, it is increasingly popular to equip mobile RGB cameras with
Time-of-Flight (ToF) sensors for active depth sensing. However, for
off-the-shelf ToF sensors, one must tackle two problems in order to obtain
high-quality depth with respect to the RGB camera, namely 1) online calibration
and alignment; and 2) complicated error correction for ToF depth sensing. In
this work, we propose a framework for jointly alignment and refinement via deep
learning. First, a cross-modal optical flow between the RGB image and the ToF
amplitude image is estimated for alignment. The aligned depth is then refined
via an improved kernel predicting network that performs kernel normalization
and applies the bias prior to the dynamic convolution. To enrich our data for
end-to-end training, we have also synthesized a dataset using tools from
computer graphics. Experimental results demonstrate the effectiveness of our
approach, achieving state-of-the-art for ToF refinement.

本文提出了一种基于深度学习的框架，用于对移动 RGB 相机配备的飞行时间（ToF）传感器进行在线校准和对齐以及复杂误差校正。通过将 RGB 图像和 ToF 幅度图之间进行交叉模态光流估计以进行对齐，并通过一个改进的内核预测网络直接回归时间飞行深度进行深度精修。通过计算机图形学的方法，我们还合成了一个数据集，为数据的端到端训练提供更多丰富的信息。实验结果表明，本文提出的方法在 ToF 精修方面取得了最先进的效果。

深度端到端匹配和细化用于时差自动调焦 RGB-D 模块

Deep End-to-End Alignment and Refinement for Time-of-Flight RGB-D Module

In this work we present a novel approach for single depth map
super-resolution. Modern consumer depth sensors, especially Time-of-Flight
sensors, produce dense depth measurements, but are affected by noise and have a
low lateral resolution. We propose a method that combines the benefits of
recent advances in machine learning based single image super-resolution, i.e.
deep convolutional networks, with a variational method to recover accurate
high-resolution depth maps. In particular, we integrate a variational method
that models the piecewise affine structures apparent in depth data via an
anisotropic total generalized variation regularization term on top of a deep
network. We call our method ATGV-Net and train it end-to-end by unrolling the
optimization procedure of the variational method. To train deep networks, a
large corpus of training data with accurate ground-truth is required. We
demonstrate that it is feasible to train our method solely on synthetic data
that we generate in large quantities for this task. Our evaluations show that
we achieve state-of-the-art results on three different benchmarks, as well as
on a challenging Time-of-Flight dataset, all without utilizing an additional
intensity image as guidance.

本文介绍了一种结合深度学习和变分方法的 ATGV-Net 技术，用于实现单个深度图超分辨率过程，并展示其在不同基准测试下的最新成果。