In end-to-end autonomous driving, the utilization of existing sensor fusion
techniques for imitation learning proves inadequate in challenging situations
that involve numerous dynamic agents. To address this issue, we introduce
LeTFuser, a transformer-based algorithm for fusing multiple RGB-D camera
representations. To perform perception and control tasks simultaneously, we
utilize multi-task learning. Our model comprises of two modules, the first
being the perception module that is responsible for encoding the observation
data obtained from the RGB-D cameras. It carries out tasks such as semantic
segmentation, semantic depth cloud mapping (SDC), and traffic light state
recognition. Our approach employs the Convolutional vision Transformer (CvT)
\cite{wu2021cvt} to better extract and fuse features from multiple RGB cameras
due to local and global feature extraction capability of convolution and
transformer modules, respectively. Following this, the control module
undertakes the decoding of the encoded characteristics together with
supplementary data, comprising a rough simulator for static and dynamic
environments, as well as various measurements, in order to anticipate the
waypoints associated with a latent feature space. We use two methods to process
these outputs and generate the vehicular controls (e.g. steering, throttle, and
brake) levels. The first method uses a PID algorithm to follow the waypoints on
the fly, whereas the second one directly predicts the control policy using the
measurement features and environmental state. We evaluate the model and conduct
a comparative analysis with recent models on the CARLA simulator using various
scenarios, ranging from normal to adversarial conditions, to simulate
real-world scenarios. Our code is available at
https://github.com/pagand/e2etransfuser/tree/cvpr-w to facilitate future
studies.

通过引入 LeTFuser 算法，利用 Convolutional vision Transformer 实现多个 RGB-D 相机表示的融合，以解决端到端自动驾驶中涉及多个动态主体的复杂场景下现有传感器融合技术在模仿学习中的不足，并通过多任务学习在感知和控制任务之间同时进行相关处理。