End-to-end autonomous driving has witnessed remarkable progress. However, the
extensive deployment of autonomous vehicles has yet to be realized, primarily
due to 1) inefficient multi-modal environment perception: how to integrate data
from multi-modal sensors more efficiently; 2) non-human-like scene
understanding: how to effectively locate and predict critical risky agents in
traffic scenarios like an experienced driver. To overcome these challenges, in
this paper, we propose a Multi-Modal fusion transformer incorporating Driver
Attention (M2DA) for autonomous driving. To better fuse multi-modal data and
achieve higher alignment between different modalities, a novel
Lidar-Vision-Attention-based Fusion (LVAFusion) module is proposed. By
incorporating driver attention, we empower the human-like scene understanding
ability to autonomous vehicles to identify crucial areas within complex
scenarios precisely and ensure safety. We conduct experiments on the CARLA
simulator and achieve state-of-the-art performance with less data in
closed-loop benchmarks. Source codes are available at
this https URL

提出了一种多模态融合变压器，结合驾驶员注意力用于自动驾驶，通过 Lidar-Vision-Attention-based Fusion 模块更好地融合多模态数据，并实现不同模态之间更高的对齐，同时结合驾驶员的注意力，赋予自动驾驶车辆类似于人类的场景理解能力，准确识别复杂场景中的关键区域，确保安全。在 CARLA 模拟器上进行实验，以较少的数据在闭环基准测试中取得了最先进的性能。