This paper presents an investigation into the estimation of optical and scene
flow using RGBD information in scenarios where the RGB modality is affected by
noise or captured in dark environments. Existing methods typically rely solely
on RGB images or fuse the modalities at later stages, which can result in lower
accuracy when the RGB information is unreliable. To address this issue, we
propose a novel deep neural network approach named FusionRAFT, which enables
early-stage information fusion between sensor modalities (RGB and depth). Our
approach incorporates self- and cross-attention layers at different network
levels to construct informative features that leverage the strengths of both
modalities. Through comparative experiments, we demonstrate that our approach
outperforms recent methods in terms of performance on the synthetic dataset
Flyingthings3D, as well as the generalization on the real-world dataset KITTI.
We illustrate that our approach exhibits improved robustness in the presence of
noise and low-lighting conditions that affect the RGB images. We release the
code, models and dataset at this https URL

通过使用深度神经网络，并在不同的网络级别中应用自注意力机制以及交叉注意力机制，我们提出了一种名为 FusionRAFT 的方法，实现了早期传感器模态（RGB 和深度）之间的信息融合，以解决 RGB 信息不可靠的问题，在主动配准中取得了比最近方法更好的性能。

光学和场景流的注意力多模态融合

Attentive Multimodal Fusion for Optical and Scene Flow

We present a robot navigation system that uses an imitation learning
framework to successfully navigate in complex environments. Our framework takes
a pre-built 3D scan of a real environment and trains an agent from
pre-generated expert trajectories to navigate to any position given a panoramic
view of the goal and the current visual input without relying on map, compass,
odometry, or relative position of the target at runtime. Our end-to-end trained
agent uses RGB and depth (RGBD) information and can handle large environments
(up to $1031m^2$) across multiple rooms (up to $40$) and generalizes to unseen
targets. We show that when compared to several baselines our method (1)
requires fewer training examples and less training time, (2) reaches the goal
location with higher accuracy, and (3) produces better solutions with shorter
paths for long-range navigation tasks.

本文提出了一种机器人导航系统，采用模仿学习框架在复杂环境下成功导航，使用 RGB 和深度 (RGBD) 信息，可以应对大型环境和多个房间及未知目标的导航，相较于多个基准测试具有更好的性能。