NeRF-Det has achieved impressive performance in indoor multi-view 3D
detection by innovatively utilizing NeRF to enhance representation learning.
Despite its notable performance, we uncover three decisive shortcomings in its
current design, including semantic ambiguity, inappropriate sampling, and
insufficient utilization of depth supervision. To combat the aforementioned
problems, we present three corresponding solutions: 1) Semantic Enhancement. We
project the freely available 3D segmentation annotations onto the 2D plane and
leverage the corresponding 2D semantic maps as the supervision signal,
significantly enhancing the semantic awareness of multi-view detectors. 2)
Perspective-aware Sampling. Instead of employing the uniform sampling strategy,
we put forward the perspective-aware sampling policy that samples densely near
the camera while sparsely in the distance, more effectively collecting the
valuable geometric clues. 3)Ordinal Residual Depth Supervision. As opposed to
directly regressing the depth values that are difficult to optimize, we divide
the depth range of each scene into a fixed number of ordinal bins and
reformulate the depth prediction as the combination of the classification of
depth bins as well as the regression of the residual depth values, thereby
benefiting the depth learning process. The resulting algorithm, NeRF-Det++, has
exhibited appealing performance in the ScanNetV2 and ARKITScenes datasets.
Notably, in ScanNetV2, NeRF-Det++ outperforms the competitive NeRF-Det by +1.9%
in mAP@0.25 and +3.5% in mAP@0.50$. The code will be publicly at
this https URL

NeRF-Det++ 通过创新地利用 NeRF 来增强表示学习，在室内多视角 3D 检测方面取得了令人瞩目的性能。对于其目前的设计，我们揭示了三个关键性的缺点，包括语义模糊、不恰当的采样和对深度监督的利用不足。为了解决上述问题，我们提出了三种相应的解决方案：1）语义增强；2）透视感知采样；3）序数残差深度监督。在 ScanNetV2 和 ARKITScenes 数据集中，NeRF-Det++ 呈现出令人满意的性能。尤其是在 ScanNetV2 中，NeRF-Det++ 在 mAP@0.25 方面超过竞争对手 NeRF-Det 1.9%，在 mAP@0.50 方面超过 3.5%。该代码将在公开的网址上提供。