Due to the lack of depth cues in images, multi-frame inputs are important for
the success of vision-based perception, prediction, and planning in autonomous
driving. Observations from different angles enable the recovery of 3D object
states from 2D image inputs if we can identify the same instance in different
input frames. However, the dynamic nature of autonomous driving scenes leads to
significant changes in the appearance and shape of each instance captured by
the camera at different time steps. To this end, we propose a novel contrastive
learning algorithm, Cohere3D, to learn coherent instance representations in a
long-term input sequence robust to the change in distance and perspective. The
learned representation aids in instance-level correspondence across multiple
input frames in downstream tasks. In the pretraining stage, the raw point
clouds from LiDAR sensors are utilized to construct the long-term temporal
correspondence for each instance, which serves as guidance for the extraction
of instance-level representation from the vision-based bird's eye-view (BEV)
feature map. Cohere3D encourages a consistent representation for the same
instance at different frames but distinguishes between representations of
different instances. We evaluate our algorithm by finetuning the pretrained
model on various downstream perception, prediction, and planning tasks. Results
show a notable improvement in both data efficiency and task performance.

为了克服图像中缺乏深度线索的问题，本文提出了一种新的对比学习算法 Cohere3D，用于在长期输入序列中学习一致的实例表示，从而在自动驾驶的下游任务中实现多个输入帧的实例级对应。通过预训练阶段中从 LiDAR 传感器中的原始点云构建长期时序对应来辅助从基于视觉的鸟瞰特征图中提取实例级表示。实验证明该算法在数据效率和任务性能方面都取得显著改进。