Passive, compact, single-shot 3D sensing is useful in many application areas
such as microscopy, medical imaging, surgical navigation, and autonomous
driving where form factor, time, and power constraints can exist. Obtaining
RGB-D scene information over a short imaging distance, in an ultra-compact form
factor, and in a passive, snapshot manner is challenging. Dual-pixel (DP)
sensors are a potential solution to achieve the same. DP sensors collect light
rays from two different halves of the lens in two interleaved pixel arrays,
thus capturing two slightly different views of the scene, like a stereo camera
system. However, imaging with a DP sensor implies that the defocus blur size is
directly proportional to the disparity seen between the views. This creates a
trade-off between disparity estimation vs. deblurring accuracy. To improve this
trade-off effect, we propose CADS (Coded Aperture Dual-Pixel Sensing), in which
we use a coded aperture in the imaging lens along with a DP sensor. In our
approach, we jointly learn an optimal coded pattern and the reconstruction
algorithm in an end-to-end optimization setting. Our resulting CADS imaging
system demonstrates improvement of $>$1.5dB PSNR in all-in-focus (AIF)
estimates and 5-6% in depth estimation quality over naive DP sensing for a wide
range of aperture settings. Furthermore, we build the proposed CADS prototypes
for DSLR photography settings and in an endoscope and a dermoscope form factor.
Our novel coded dual-pixel sensing approach demonstrates accurate RGB-D
reconstruction results in simulations and real-world experiments in a passive,
snapshot, and compact manner.

通过使用 CADS 方法，我们在紧凑而被动的形式下借助双像素传感器实现了准确的 RGB-D 重建结果。

被动快照编码孔径双像素 RGB-D 成像

Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging

We introduce MIPS-Fusion, a robust and scalable online RGB-D reconstruction
method based on a novel neural implicit representation --
multi-implicit-submap. Different from existing neural RGB-D reconstruction
methods lacking either flexibility with a single neural map or scalability due
to extra storage of feature grids, we propose a pure neural representation
tackling both difficulties with a divide-and-conquer design. In our method,
neural submaps are incrementally allocated alongside the scanning trajectory
and efficiently learned with local neural bundle adjustments. The submaps can
be refined individually in a back-end optimization and optimized jointly to
realize submap-level loop closure. Meanwhile, we propose a hybrid tracking
approach combining randomized and gradient-based pose optimizations. For the
first time, randomized optimization is made possible in neural tracking with
several key designs to the learning process, enabling efficient and robust
tracking even under fast camera motions. The extensive evaluation demonstrates
that our method attains higher reconstruction quality than the state of the
arts for large-scale scenes and under fast camera motions.

我们引入 MIPS-Fusion，这是一种基于新颖的神经隐式表示 - 多隐式子地图的强大且可扩展的在线 RGB-D 重建方法。

MIPS-Fusion: 可扩展和稳健的在线神经网络 RGB-D 重建的多隐式子地图

MIPS-Fusion: Multi-Implicit-Submaps for Scalable and Robust Online  Neural RGB-D Reconstruction

Online semantic 3D segmentation in company with real-time RGB-D
reconstruction poses special challenges such as how to perform 3D convolution
directly over the progressively fused 3D geometric data, and how to smartly
fuse information from frame to frame. We propose a novel fusion-aware 3D point
convolution which operates directly on the geometric surface being
reconstructed and exploits effectively the inter-frame correlation for high
quality 3D feature learning. This is enabled by a dedicated dynamic data
structure which organizes the online acquired point cloud with global-local
trees. Globally, we compile the online reconstructed 3D points into an
incrementally growing coordinate interval tree, enabling fast point insertion
and neighborhood query. Locally, we maintain the neighborhood information for
each point using an octree whose construction benefits from the fast query of
the global tree.Both levels of trees update dynamically and help the 3D
convolution effectively exploits the temporal coherence for effective
information fusion across RGB-D frames.

该研究提出了一种基于动态数据结构的卷积方法，通过全局 - 本地树结构维护局部领域信息，以及增量坐标区间树实现快速点云插入和邻域查询，从而解决在线语义三维分割与 RGB-D 重建中的挑战，实现跨帧时间上的信息融合。