We present ANYU, a new virtually augmented version of the NYU depth v2
dataset, designed for monocular depth estimation. In contrast to the well-known
approach where full 3D scenes of a virtual world are utilized to generate
artificial datasets, ANYU was created by incorporating RGB-D representations of
virtual reality objects into the original NYU depth v2 images. We specifically
did not match each generated virtual object with an appropriate texture and a
suitable location within the real-world image. Instead, an assignment of
texture, location, lighting, and other rendering parameters was randomized to
maximize a diversity of the training data, and to show that it is randomness
that can improve the generalizing ability of a dataset. By conducting extensive
experiments with our virtually modified dataset and validating on the original
NYU depth v2 and iBims-1 benchmarks, we show that ANYU improves the monocular
depth estimation performance and generalization of deep neural networks with
considerably different architectures, especially for the current
state-of-the-art VPD model. To the best of our knowledge, this is the first
work that augments a real-world dataset with randomly generated virtual 3D
objects for monocular depth estimation. We make our ANYU dataset publicly
available in two training configurations with 10% and 100% additional
synthetically enriched RGB-D pairs of training images, respectively, for
efficient training and empirical exploration of virtual augmentation at
this https URL

该研究中介绍了 ANYU，它是 NYU depth v2 数据集的一种扩展版本，通过将虚拟现实对象的 RGB-D 表示融合到原始图像中，提高了深度估计性能和深度神经网络的泛化能力。该研究是首次利用随机生成的虚拟 3D 对象扩增现实世界数据集进行单目深度估计的工作。

用于单目深度估计的虚拟增强 NYU 深度 V2 数据集：需要人工增强吗？

Virtually Enriched NYU Depth V2 Dataset for Monocular Depth Estimation:  Do We Need Artificial Augmentation?

Monocular depth estimation is a challenging task in complex compositions
depicting multiple objects of diverse scales. Albeit the recent great progress
thanks to the deep convolutional neural networks (CNNs), the state-of-the-art
monocular depth estimation methods still fall short to handle such real-world
challenging scenarios. In this paper, we propose a deep end-to-end learning
framework to tackle these challenges, which learns the direct mapping from a
color image to the corresponding depth map. First, we represent monocular depth
estimation as a multi-category dense labeling task by contrast to the
regression based formulation. In this way, we could build upon the recent
progress in dense labeling such as semantic segmentation. Second, we fuse
different side-outputs from our front-end dilated convolutional neural network
in a hierarchical way to exploit the multi-scale depth cues for depth
estimation, which is critical to achieve scale-aware depth estimation. Third,
we propose to utilize soft-weighted-sum inference instead of the hard-max
inference, transforming the discretized depth score to continuous depth value.
Thus, we reduce the influence of quantization error and improve the robustness
of our method. Extensive experiments on the NYU Depth V2 and KITTI datasets
show the superiority of our method compared with current state-of-the-art
methods. Furthermore, experiments on the NYU V2 dataset reveal that our model
is able to learn the probability distribution of depth.

本文提出了一种利用深度学习解决复杂多物体单目深度估计问题的端到端学习框架，通过将单目深度估计表述为一种多类别密集标注任务以及采用软加权和推断方法等策略，实现了对多尺度深度信息的有效利用，降低了量化误差并提高了方法的鲁棒性。实验结果表明，该方法在 NYU Depth V2 数据集上表现优于当前最先进方法。