Large vision-language models have impressively promote the performance of 2D
visual recognition under zero/few-shot scenarios. In this paper, we focus on
exploiting the large vision-language model, i.e., CLIP, to address
zero/few-shot 3D shape recognition based on multi-view representations. The key
challenge for both tasks is to generate a discriminative descriptor of the 3D
shape represented by multiple view images under the scenarios of either without
explicit training (zero-shot 3D shape recognition) or training with a limited
number of data (few-shot 3D shape recognition). We analyze that both tasks are
relevant and can be considered simultaneously. Specifically, leveraging the
descriptor which is effective for zero-shot inference to guide the tuning of
the aggregated descriptor under the few-shot training can significantly improve
the few-shot learning efficacy. Hence, we propose Prompt-Enhanced View
Aggregation Network (PEVA-Net) to simultaneously address zero/few-shot 3D shape
recognition. Under the zero-shot scenario, we propose to leverage the prompts
built up from candidate categories to enhance the aggregation process of
multiple view-associated visual features. The resulting aggregated feature
serves for effective zero-shot recognition of the 3D shapes. Under the few-shot
scenario, we first exploit a transformer encoder to aggregate the
view-associated visual features into a global descriptor. To tune the encoder,
together with the main classification loss, we propose a self-distillation
scheme via a feature distillation loss by treating the zero-shot descriptor as
the guidance signal for the few-shot descriptor. This scheme can significantly
enhance the few-shot learning efficacy.

基于多视图表征，利用大型视觉语言模型 CLIP，探索零 / 少样本三维形状识别任务，并提出 Prompt-Enhanced View Aggregation Network (PEVA-Net) 以同时解决零 / 少样本场景下的三维形状识别问题，通过自我蒸馏方案显著提高了少样本学习效果。

PEVA-Net: 针对零 / 少样本多视角三维形状识别的提示增强的视图聚合网络

PEVA-Net: Prompt-Enhanced View Aggregation Network for Zero/Few-Shot  Multi-View 3D Shape Recognition

We present an end-to-end method for object detection and trajectory
prediction utilizing multi-view representations of LiDAR returns and camera
images. In this work, we recognize the strengths and weaknesses of different
view representations, and we propose an efficient and generic fusing method
that aggregates benefits from all views. Our model builds on a state-of-the-art
Bird's-Eye View (BEV) network that fuses voxelized features from a sequence of
historical LiDAR data as well as rasterized high-definition map to perform
detection and prediction tasks. We extend this model with additional LiDAR
Range-View (RV) features that use the raw LiDAR information in its native,
non-quantized representation. The RV feature map is projected into BEV and
fused with the BEV features computed from LiDAR and high-definition map. The
fused features are then further processed to output the final detections and
trajectories, within a single end-to-end trainable network. In addition, the RV
fusion of LiDAR and camera is performed in a straightforward and
computationally efficient manner using this framework. The proposed multi-view
fusion approach improves the state-of-the-art on proprietary large-scale
real-world data collected by a fleet of self-driving vehicles, as well as on
the public nuScenes data set with minimal increases on the computational cost.

本文提出了一种利用多视角 LiDAR 返回和摄像头图像的终端到终端物体检测和轨迹预测方法，使用 Bird's-Eye View (BEV) 网络的状态 -of-the-art 融合历史 LiDAR 数据和高清地图的体素化特征来执行检测和预测任务。在此基础上，我们使用 LiDAR Range-View （RV）功能扩展了此模型，使用 LiDAR 原始信息的非量化表示。RV 特征图被投影到 BEV 中，并与计算自 LiDAR 和高清地图的 BEV 特征融合。最终输出检测和轨迹，这是一个单一的可训练终端到终端网络。在实际的无人驾驶车辆数据和公共 nuScenes 数据集上，所提出的多视角融合方法的表现都优于最先进的技术，并且增加了较少的计算成本。