Novel-view synthesis with sparse input views is important for real-world
applications like AR/VR and autonomous driving. Recent methods have integrated
depth information into NeRFs for sparse input synthesis, leveraging depth prior
for geometric and spatial understanding. However, most existing works tend to
overlook inaccuracies within depth maps and have low time efficiency. To
address these issues, we propose a depth-guided robust and fast point cloud
fusion NeRF for sparse inputs. We perceive radiance fields as an explicit voxel
grid of features. A point cloud is constructed for each input view,
characterized within the voxel grid using matrices and vectors. We accumulate
the point cloud of each input view to construct the fused point cloud of the
entire scene. Each voxel determines its density and appearance by referring to
the point cloud of the entire scene. Through point cloud fusion and voxel grid
fine-tuning, inaccuracies in depth values are refined or substituted by those
from other views. Moreover, our method can achieve faster reconstruction and
greater compactness through effective vector-matrix decomposition. Experimental
results underline the superior performance and time efficiency of our approach
compared to state-of-the-art baselines.

通过深度引导的稀疏输入点云融合构建的辐射场点云可以有效地解决深度图的不准确性和低时间效率问题，实验证明该方法在性能和效率上优于现有技术。

基于深度引导的鲁棒快速稀疏输入视图点云融合 NeRF

Depth-Guided Robust and Fast Point Cloud Fusion NeRF for Sparse Input  Views

Monocular 3D object detection has long been a challenging task in autonomous
driving, which requires to decode 3D predictions solely from a single 2D image.
Most existing methods follow conventional 2D object detectors to localize
objects based on their centers, and predict 3D attributes by neighboring
features around the centers. However, only using local features is insufficient
to understand the scene-level 3D spatial structures and ignore the inter-object
depth relations from contextual cues. In this paper, we introduce a novel
framework for Monocular DEtection with a depth-guided TRansformer, named
MonoDETR. The vanilla transformer is modified to be depth-aware and the whole
detection process is then guided by depth. Specifically, we represent 3D object
candidates as a set of queries and adopt an attention-based depth encoder to
produce non-local depth embeddings of the input image. Then, we propose a
depth-guided decoder with depth cross-attention modules to conduct both
inter-query and query-scene depth feature interactions. In this way, each
object query estimates its 3D attributes adaptively from the depth-guided
regions from the image and is no longer constrained to use only neighboring
visual features. MonoDETR is an end-to-end network without extra data or NMS
post-processing and achieves state-of-the-art performance on KITTI benchmark
with significant gains. Extensive ablation studies demonstrate the
effectiveness of our approach and its potential to serve as a transformer
baseline for future monocular 3D object detection research. Code is available
at this https URL.

本研究介绍了一种名为 MonoDETR 的使用深度感知 Transformer 的新型单目三维物体检测框架。通过将物体候选项表示为一组查询，并采用基于注意力的深度编码器生成二维输入图像的非局部深度嵌入，然后提出了一个深度引导的解码器，通过深度交叉关注模块对查询与场景深度特征进行交互，每个物体查询从图像的深度引导区域自适应地估计其三维属性。该系统具有端到端的结构，在 KITTI 基准测试上取得了最新成果，并在广泛的消融研究中证明了该方法的有效性，具有成为未来单目三维物体检测研究的 Transformer 基线的潜力。