Deep neural network models have achieved remarkable progress in 3D scene
understanding while trained in the closed-set setting and with full labels.
However, the major bottleneck for current 3D recognition approaches is that
they do not have the capacity to recognize any unseen novel classes beyond the
training categories in diverse kinds of real-world applications. In the
meantime, current state-of-the-art 3D scene understanding approaches primarily
require high-quality labels to train neural networks, which merely perform well
in a fully supervised manner. This work presents a generalized and simple
framework for dealing with 3D scene understanding when the labeled scenes are
quite limited. To extract knowledge for novel categories from the pre-trained
vision-language models, we propose a hierarchical feature-aligned pre-training
and knowledge distillation strategy to extract and distill meaningful
information from large-scale vision-language models, which helps benefit the
open-vocabulary scene understanding tasks. To leverage the boundary
information, we propose a novel energy-based loss with boundary awareness
benefiting from the region-level boundary predictions. To encourage latent
instance discrimination and to guarantee efficiency, we propose the
unsupervised region-level semantic contrastive learning scheme for point
clouds, using confident predictions of the neural network to discriminate the
intermediate feature embeddings at multiple stages. Extensive experiments with
both indoor and outdoor scenes demonstrated the effectiveness of our approach
in both data-efficient learning and open-world few-shot learning. All codes,
models, and data are made publicly available at:
this https URL

通过使用预先训练的视觉 - 语言模型和无监督的区域级语义对比学习方案，本研究提出了一种通用且简单的框架，用于处理有限标记情况下的 3D 场景理解，从而在数据效率学习和开放世界少样本学习方面展现了有效性。

通过层次特征对齐预训练和区域感知微调的通用标签高效三维场景解析

Generalized Label-Efficient 3D Scene Parsing via Hierarchical Feature  Aligned Pre-Training and Region-Aware Fine-tuning

Novelty detection is crucial for real-life applications. While it is common
in activity recognition to assume a closed-set setting, i.e. test samples are
always of training categories, this assumption is impractical in a real-world
scenario. Test samples can be of various categories including those never seen
before during training. Thus, being able to know what we know and what we do
not know is decisive for the model to avoid what can be catastrophic
consequences. We present in this work a novel approach for identifying samples
of activity classes that are not previously seen by the classifier. Our model
employs a voting-based scheme that leverages the estimated uncertainty of the
individual classifiers in their predictions to measure the novelty of a new
input sample. Furthermore, the voting is privileged to a subset of informed
classifiers that can best estimate whether a sample is novel or not when it is
classified to a certain known category. In a thorough evaluation on UCF-101 and
HMDB-51, we show that our model consistently outperforms state-of-the-art in
novelty detection. Additionally, by combining our model with off-the-shelf
zero-shot learning (ZSL) approaches, our model leads to a significant
improvement in action classification accuracy for the generalized ZSL setting.

本论文提出了一种新的方法，通过投票机制结合分类器的不确定度来测量新输入样本的新颖性，并且在 UCF-101 和 HMDB-51 数据集上取得了比最先进的方法更好的结果，在与零样本学习 (ZSL) 相结合的情况下，可以显著提高广义 ZSL 设置下的行动分类准确性。