Data-driven autonomous driving motion generation tasks are frequently
impacted by the limitations of dataset size and the domain gap between
datasets, which precludes their extensive application in real-world scenarios.
To address this issue, we introduce SMART, a novel autonomous driving motion
generation paradigm that models vectorized map and agent trajectory data into
discrete sequence tokens. These tokens are then processed through a
decoder-only transformer architecture to train for the next token prediction
task across spatial-temporal series. This GPT-style method allows the model to
learn the motion distribution in real driving scenarios. SMART achieves
state-of-the-art performance across most of the metrics on the generative Sim
Agents challenge, ranking 1st on the leaderboards of Waymo Open Motion Dataset
(WOMD), demonstrating remarkable inference speed. Moreover, SMART represents
the generative model in the autonomous driving motion domain, exhibiting
zero-shot generalization capabilities: Using only the NuPlan dataset for
training and WOMD for validation, SMART achieved a competitive score of 0.71 on
the Sim Agents challenge. Lastly, we have collected over 1 billion motion
tokens from multiple datasets, validating the model's scalability. These
results suggest that SMART has initially emulated two important properties:
scalability and zero-shot generalization, and preliminarily meets the needs of
large-scale real-time simulation applications. We have released all the code to
promote the exploration of models for motion generation in the autonomous
driving field.

SMART 是一种新的自主驾驶运动生成范例，通过将矢量地图和代理轨迹数据建模为离散序列记号，并通过仅有解码器的 Transformer 架构处理这些记号，从而实现在实际驾驶场景中学习运动分布的性能。

SMART：通过下一个标记预测实现可扩展的多智能体实时模拟

SMART: Scalable Multi-agent Real-time Simulation via Next-token  Prediction

Research on Large Language Models (LLMs) has recently seen exponential
growth, largely focused on transformer-based architectures, as introduced by
[1] and further advanced by the decoder-only variations in [2]. Contemporary
studies typically aim to improve model capabilities by increasing both the
architecture's complexity and the volume of training data. However, research
exploring how to reduce model sizes while maintaining performance is limited.
This study introduces three modifications to the decoder-only transformer
architecture: ParallelGPT (p-gpt), LinearlyCompressedGPT (lc-gpt), and
ConvCompressedGPT (cc-gpt). These variants achieve comparable performance to
conventional architectures in code generation tasks while benefiting from
reduced model sizes and faster training times. We open-source the model weights
and codebase to support future research and development in this domain.

本研究引入了三种变体（ParallelGPT、LinearlyCompressedGPT 和 ConvCompressedGPT）来修改解码器专用的 Transformer 架构，这些变体在代码生成任务中取得了可比较的性能，同时具有模型尺寸较小和训练时间更快的优势。

朝着更小、更快的只解码 Transformer：架构变体及其影响

Towards smallers, faster decoder-only transformers: Architectural  variants and their implications

The decoder-only Transformer architecture with causal masking and relative
position encoding (RPE) has become the de facto choice in language modeling.
Despite its exceptional performance across various tasks, we have identified
two limitations: First, it requires all attention scores to be non-zero and sum
up to 1, even if the current embedding has sufficient self-contained
information. This compels the model to assign disproportional excessive
attention to specific tokens. Second, RPE-based Transformers are not universal
approximators due to their limited capacity at encoding absolute positional
information, which limits their application in position-critical tasks. In this
work, we propose StableMask: a parameter-free method to address both
limitations by refining the causal mask. It introduces pseudo-attention values
to balance attention distributions and encodes absolute positional information
via a progressively decreasing mask ratio. StableMask's effectiveness is
validated both theoretically and empirically, showing significant enhancements
in language models with parameter sizes ranging from 71M to 1.4B across diverse
datasets and encoding methods. We further show that it naturally supports (1)
efficient extrapolation without special tricks such as StreamingLLM and (2)
easy integration with existing attention optimization techniques.

通过改进因果掩码的方法，提出了 StableMask，解决了 decoder-only Transformer 架构中的限制，并在语言模型中得到了显著的增强。