We investigate the benefit of combining blind audio recordings with 3D scene
information for novel-view acoustic synthesis. Given audio recordings from 2-4
microphones and the 3D geometry and material of a scene containing multiple
unknown sound sources, we estimate the sound anywhere in the scene. We identify
the main challenges of novel-view acoustic synthesis as sound source
localization, separation, and dereverberation. While naively training an
end-to-end network fails to produce high-quality results, we show that
incorporating room impulse responses (RIRs) derived from 3D reconstructed rooms
enables the same network to jointly tackle these tasks. Our method outperforms
existing methods designed for the individual tasks, demonstrating its
effectiveness at utilizing 3D visual information. In a simulated study on the
Matterport3D-NVAS dataset, our model achieves near-perfect accuracy on source
localization, a PSNR of 26.44 dB and a SDR of 14.23 dB for source separation
and dereverberation, resulting in a PSNR of 25.55 dB and a SDR of 14.20 dB on
novel-view acoustic synthesis. Code, pretrained model, and video results are
available on the project webpage (this https URL).

探究结合盲音频录音和 3D 场景信息对新视角声学合成的益处。通过 2-4 个麦克风的音频录音和包含多个未知声源的场景的 3D 几何学和材料，估计场景中的任何声音。我们确定了新视角声学合成的主要挑战，即声源定位、分离和去混响。虽然简单地训练端到端网络不能产生高质量的结果，但我们证明了融入从 3D 重建房间导出的房间脉冲响应（RIRs）使得相同的网络能够统一解决这些任务。我们的方法优于针对各个任务设计的现有方法，证明其在利用 3D 视觉信息方面的有效性。在 Matterport3D-NVAS 数据集上的模拟研究中，我们的模型在源定位方面实现了接近完美的准确性，源分离和去混响方面的 PSNR 为 26.44 dB 和 SDR 为 14.23 dB，从而在新视角声学合成方面获得了 PSNR 为 25.55 dB 和 SDR 为 14.20 dB 的结果。项目网页提供了代码、预训练模型和视频结果。

三维重建房间的新颖视角声学合成

Novel-View Acoustic Synthesis from 3D Reconstructed Rooms

General robot grasping in clutter requires the ability to synthesize grasps
that work for previously unseen objects and that are also robust to physical
interactions, such as collisions with other objects in the scene. In this work,
we design and train a network that predicts 6 DOF grasps from 3D scene
information gathered from an on-board sensor such as a wrist-mounted depth
camera. Our proposed Volumetric Grasping Network (VGN) accepts a Truncated
Signed Distance Function (TSDF) representation of the scene and directly
outputs the predicted grasp quality and the associated gripper orientation and
opening width for each voxel in the queried 3D volume. We show that our
approach can plan grasps in only 10 ms and is able to clear 92% of the objects
in real-world clutter removal experiments without the need for explicit
collision checking. The real-time capability opens up the possibility for
closed-loop grasp planning, allowing robots to handle disturbances, recover
from errors and provide increased robustness. Code is available at
this https URL

本研究介紹了一種使用體積把握網絡（VGN）的方法，該網絡通過使用由手腕安裝式深度相機收集的 3D 場景信息，從過去未見過的物體中預測並設計出能夠穩健地擷取物體的抓取方法。

体积抓取网络：杂乱环境中实时六自由度抓取检测

Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter

Self-supervised monocular depth estimation presents a powerful method to
obtain 3D scene information from single camera images, which is trainable on
arbitrary image sequences without requiring depth labels, e.g., from a LiDAR
sensor. In this work we present a new self-supervised semantically-guided depth
estimation (SGDepth) method to deal with moving dynamic-class (DC) objects,
such as moving cars and pedestrians, which violate the static-world assumptions
typically made during training of such models. Specifically, we propose (i)
mutually beneficial cross-domain training of (supervised) semantic segmentation
and self-supervised depth estimation with task-specific network heads, (ii) a
semantic masking scheme providing guidance to prevent moving DC objects from
contaminating the photometric loss, and (iii) a detection method for frames
with non-moving DC objects, from which the depth of DC objects can be learned.
We demonstrate the performance of our method on several benchmarks, in
particular on the Eigen split, where we exceed all baselines without test-time
refinement.

本文提出了一种自主引导的语义深度估计方法（SGDepth），通过交叉域训练避免移动车辆和行人等动态类别对象对深度标签的干扰，利用异构网络头进行语义分割和深度估计，采用语义遮罩机制预防动态物体的污染，并引入一种检测算法学习移动类别对象的深度。通过在多个数据集上评估验证了该方法的性能。