In contrast to extensive studies on general vision, pre-training for scalable
visual autonomous driving remains seldom explored. Visual autonomous driving
applications require features encompassing semantics, 3D geometry, and temporal
information simultaneously for joint perception, prediction, and planning,
posing dramatic challenges for pre-training. To resolve this, we bring up a new
pre-training task termed as visual point cloud forecasting - predicting future
point clouds from historical visual input. The key merit of this task captures
the synergic learning of semantics, 3D structures, and temporal dynamics. Hence
it shows superiority in various downstream tasks. To cope with this new
problem, we present ViDAR, a general model to pre-train downstream visual
encoders. It first extracts historical embeddings by the encoder. These
representations are then transformed to 3D geometric space via a novel Latent
Rendering operator for future point cloud prediction. Experiments show
significant gain in downstream tasks, e.g., 3.1% NDS on 3D detection, ~10%
error reduction on motion forecasting, and ~15% less collision rate on
planning.

由于视觉自主驾驶应用需要同时处理语义、3D 几何和时间信息以进行联合感知、预测和规划，因此我们提出了一种名为 “ViDAR” 的预训练模型，用于预训练下游视觉编码器，并通过新颖的隐变换操作将历史嵌入式表示转换为 3D 几何空间，以实现未来点云的预测。实验结果表明，在下游任务中取得了显著的增益，如在 3D 检测上的 NDS 提高了 3.1％，运动预测的错误率减少了约 10％，规划中碰撞率减少了约 15％。