Perception is crucial in the realm of autonomous driving systems, where
bird's eye view (BEV)-based architectures have recently reached
state-of-the-art performance. The desirability of self-supervised
representation learning stems from the expensive and laborious process of
annotating 2D and 3D data. Although previous research has investigated
pretraining methods for both LiDAR and camera-based 3D object detection, a
unified pretraining framework for multimodal BEV perception is missing. In this
study, we introduce CALICO, a novel framework that applies contrastive
objectives to both LiDAR and camera backbones. Specifically, CALICO
incorporates two stages: point-region contrast (PRC) and region-aware
distillation (RAD). PRC better balances the region- and scene-level
representation learning on the LiDAR modality and offers significant
performance improvement compared to existing methods. RAD effectively achieves
contrastive distillation on our self-trained teacher model. CALICO's efficacy
is substantiated by extensive evaluations on 3D object detection and BEV map
segmentation tasks, where it delivers significant performance improvements.
Notably, CALICO outperforms the baseline method by 10.5% and 8.6% on NDS and
mAP. Moreover, CALICO boosts the robustness of multimodal 3D object detection
against adversarial attacks and corruption. Additionally, our framework can be
tailored to different backbones and heads, positioning it as a promising
approach for multimodal BEV perception.

研究了无人驾驶系统中多模态 BEV 感知的统一预训练框架，引入了 CALICO 框架，应用反差目标在 LiDAR 和相机骨干网络上实现教师模型的对比蒸馏，成功提高了 3D 目标检测和 BEV 地图分割等任务的表现。

CALICO: 用于 BEV 感知的自监督相机 - LiDAR 对比预训练

CALICO: Self-Supervised Camera-LiDAR Contrastive Pre-training for BEV  Perception

We propose a self-supervised learning approach for videos that learns
representations of both the RGB frames and the accompanying audio without human
supervision. In contrast to images that capture the static scene appearance,
videos also contain sound and temporal scene dynamics. To leverage the temporal
and aural dimension inherent to videos, our method extends temporal
self-supervision to the audio-visual setting and integrates it with multi-modal
contrastive objectives. As temporal self-supervision, we pose playback speed
and direction recognition in both modalities and propose intra- and inter-modal
temporal ordering tasks. Furthermore, we design a novel contrastive objective
in which the usual pairs are supplemented with additional sample-dependent
positives and negatives sampled from the evolving feature space. In our model,
we apply such losses among video clips and between videos and their temporally
corresponding audio clips. We verify our model design in extensive ablation
experiments and evaluate the video and audio representations in transfer
experiments to action recognition and retrieval on UCF101 and HMBD51, audio
classification on ESC50, and robust video fingerprinting on VGG-Sound, with
state-of-the-art results.

本文提出了一种自我监督学习方法，用于学习视频的表示，结合了 RGB 帧和相关的音频，通过多模式对比目标来扩展时间自我监督的音频 - 视觉设置，并提出了新的对比目标。