Transformer architectures have exhibited promising performance in various
autonomous driving applications in recent years. On the other hand, its
dedicated hardware acceleration on portable computational platforms has become
the next critical step for practical deployment in real autonomous vehicles.
This survey paper provides a comprehensive overview, benchmark, and analysis of
Transformer-based models specifically tailored for autonomous driving tasks
such as lane detection, segmentation, tracking, planning, and decision-making.
We review different architectures for organizing Transformer inputs and
outputs, such as encoder-decoder and encoder-only structures, and explore their
respective advantages and disadvantages. Furthermore, we discuss
Transformer-related operators and their hardware acceleration schemes in depth,
taking into account key factors such as quantization and runtime. We
specifically illustrate the operator level comparison between layers from
convolutional neural network, Swin-Transformer, and Transformer with 4D
encoder. The paper also highlights the challenges, trends, and current insights
in Transformer-based models, addressing their hardware deployment and
acceleration issues within the context of long-term autonomous driving
applications.

系统概述了针对自动驾驶领域的 Transformer 模型，分析了编码器解码器和仅编码器模型的优缺点，并深入探讨了量化和运行时等关键因素对 Transformer 操作符及其硬件加速的影响，同时在卷积神经网络、Swin-Transformer 和具有 4D 编码器的 Transformers 层之间进行操作符水平比较。