Forecasting the future states of surrounding traffic participants is a
crucial capability for autonomous vehicles. The recently proposed occupancy
flow field prediction introduces a scalable and effective representation to
jointly predict surrounding agents' future motions in a scene. However, the
challenging part is to model the underlying social interactions among traffic
agents and the relations between occupancy and flow. Therefore, this paper
proposes a novel Multi-modal Hierarchical Transformer network that fuses the
vectorized (agent motion) and visual (scene flow, map, and occupancy)
modalities and jointly predicts the flow and occupancy of the scene.
Specifically, visual and vector features from sensory data are encoded through
a multi-stage Transformer module and then a late-fusion Transformer module with
temporal pixel-wise attention. Importantly, a flow-guided multi-head
self-attention (FG-MSA) module is designed to better aggregate the information
on occupancy and flow and model the mathematical relations between them. The
proposed method is comprehensively validated on the Waymo Open Motion Dataset
and compared against several state-of-the-art models. The results reveal that
our model with much more compact architecture and data inputs than other
methods can achieve comparable performance. We also demonstrate the
effectiveness of incorporating vectorized agent motion features and the
proposed FG-MSA module. Compared to the ablated model without the FG-MSA
module, which won 2nd place in the 2022 Waymo Occupancy and Flow Prediction
Challenge, the current model shows better separability for flow and occupancy
and further performance improvements.

本文提出了一种新颖的多模态分层 transformer 网络，通过融合被向量化的（流动代理）和视觉场景（流场、地图和占用）模式，以及交通代理之间的社会交互和占用和流之间的关系，来共同预测场景的流和占用，证明了该方法在 Waymo 开放运动数据集上的有效性。