How should we integrate representations from complementary sensors for
autonomous driving? Geometry-based fusion has shown promise for perception
(e.g. object detection, motion forecasting). However, in the context of
end-to-end driving, we find that imitation learning based on existing sensor
fusion methods underperforms in complex driving scenarios with a high density
of dynamic agents. Therefore, we propose TransFuser, a mechanism to integrate
image and LiDAR representations using self-attention. Our approach uses
transformer modules at multiple resolutions to fuse perspective view and bird's
eye view feature maps. We experimentally validate its efficacy on a challenging
new benchmark with long routes and dense traffic, as well as the official
leaderboard of the CARLA urban driving simulator. At the time of submission,
TransFuser outperforms all prior work on the CARLA leaderboard in terms of
driving score by a large margin. Compared to geometry-based fusion, TransFuser
reduces the average collisions per kilometer by 48%.

该研究介绍了 TransFuser - 一种使用自我注意力机制将图像和雷达的信息进行融合，以提高自动驾驶中辨识物体和行驶路径的性能，实验证明该方法的表现在 CARLA 排行榜上优于以往所有的算法。

TransFuser：基于 Transformer 传感器融合的自动驾驶模仿

TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving

How should representations from complementary sensors be integrated for
autonomous driving? Geometry-based sensor fusion has shown great promise for
perception tasks such as object detection and motion forecasting. However, for
the actual driving task, the global context of the 3D scene is key, e.g. a
change in traffic light state can affect the behavior of a vehicle
geometrically distant from that traffic light. Geometry alone may therefore be
insufficient for effectively fusing representations in end-to-end driving
models. In this work, we demonstrate that imitation learning policies based on
existing sensor fusion methods under-perform in the presence of a high density
of dynamic agents and complex scenarios, which require global contextual
reasoning, such as handling traffic oncoming from multiple directions at
uncontrolled intersections. Therefore, we propose TransFuser, a novel
Multi-Modal Fusion Transformer, to integrate image and LiDAR representations
using attention. We experimentally validate the efficacy of our approach in
urban settings involving complex scenarios using the CARLA urban driving
simulator. Our approach achieves state-of-the-art driving performance while
reducing collisions by 76% compared to geometry-based fusion.

本文提出使用多模态融合转换器 TransFuser 对图像和 LiDAR 传感器的信息进行整合以实现先进的自动驾驶技术，在 CARLA 城市驾驶模拟器中实验验证证明该方法在处理复杂场景时比传统基于几何的融合方法有更好的效果，并且在减少碰撞方面表现出色。