Recent work on action recognition leverages 3D features and textual
information to achieve state-of-the-art performance. However, most of the
current few-shot action recognition methods still rely on 2D frame-level
representations, often require additional components to model temporal
relations, and employ complex distance functions to achieve accurate alignment
of these representations. In addition, existing methods struggle to effectively
integrate textual semantics, some resorting to concatenation or addition of
textual and visual features, and some using text merely as an additional
supervision without truly achieving feature fusion and information transfer
from different modalities. In this work, we propose a simple yet effective
Semantic-Aware Few-Shot Action Recognition (SAFSAR) model to address these
issues. We show that directly leveraging a 3D feature extractor combined with
an effective feature-fusion scheme, and a simple cosine similarity for
classification can yield better performance without the need of extra
components for temporal modeling or complex distance functions. We introduce an
innovative scheme to encode the textual semantics into the video representation
which adaptively fuses features from text and video, and encourages the visual
encoder to extract more semantically consistent features. In this scheme,
SAFSAR achieves alignment and fusion in a compact way. Experiments on five
challenging few-shot action recognition benchmarks under various settings
demonstrate that the proposed SAFSAR model significantly improves the
state-of-the-art performance.

我们提出了一种简单而有效的语义感知少样本动作识别模型（SAFSAR），该模型通过直接利用 3D 特征提取器与有效的特征融合方案以及简单的余弦相似度分类，实现了更好的性能，无需复杂的距离函数和额外的时间建模组件。在这种方案中，SAFSAR 以一种紧凑的方式实现了对文本语义的编码、视频表示的自适应特征融合，并鼓励视觉编码器提取更具语义一致性的特征。在各种设置下对五个具有挑战性的少样本动作识别基准进行的实验证明，所提出的 SAFSAR 模型显著提高了最先进的性能。

面向少样本动作识别的语义感知视频表示

Semantic-aware Video Representation for Few-shot Action Recognition

Humans perceive and construct the surrounding world as an arrangement of
simple parametric models. In particular, man-made environments commonly consist
of volumetric primitives such as cuboids or cylinders. Inferring these
primitives is an important step to attain high-level, abstract scene
descriptions. Previous approaches directly estimate shape parameters from a 2D
or 3D input, and are only able to reproduce simple objects, yet unable to
accurately parse more complex 3D scenes. In contrast, we propose a robust
estimator for primitive fitting, which can meaningfully abstract real-world
environments using cuboids. A RANSAC estimator guided by a neural network fits
these primitives to 3D features, such as a depth map. We condition the network
on previously detected parts of the scene, thus parsing it one-by-one. To
obtain 3D features from a single RGB image, we additionally optimise a feature
extraction CNN in an end-to-end manner. However, naively minimising
point-to-primitive distances leads to large or spurious cuboids occluding parts
of the scene behind. We thus propose an occlusion-aware distance metric
correctly handling opaque scenes. The proposed algorithm does not require
labour-intensive labels, such as cuboid annotations, for training. Results on
the challenging NYU Depth v2 dataset demonstrate that the proposed algorithm
successfully abstracts cluttered real-world 3D scene layouts.

基于 RANSAC 估计器和神经网络的图元拟合算法，通过处理场景中的一部分来解析 3D 特征，进而获得更具抽象性的场景描述，在实现端到端训练的基础上，引入遮挡感知距离度量，成功地在不需要繁琐标注的情况下，实现了对真实世界 3D 场景布局的抽象。

立方体再探：从单张 RGB 图像学习鲁棒的三维形状拟合

Cuboids Revisited: Learning Robust 3D Shape Fitting to Single RGB Images

Indoor scene semantic parsing from RGB images is very challenging due to
occlusions, object distortion, and viewpoint variations. Going beyond prior
works that leverage geometry information, typically paired depth maps, we
present a new approach, a 3D-to-2D distillation framework, that enables us to
leverage 3D features extracted from large-scale 3D data repository (e.g.,
ScanNet-v2) to enhance 2D features extracted from RGB images. Our work has
three novel contributions. First, we distill 3D knowledge from a pretrained 3D
network to supervise a 2D network to learn simulated 3D features from 2D
features during the training, so the 2D network can infer without requiring 3D
data. Second, we design a two-stage dimension normalization scheme to calibrate
the 2D and 3D features for better integration. Third, we design a
semantic-aware adversarial training model to extend our framework for training
with unpaired 3D data. Extensive experiments on various datasets, ScanNet-V2,
S3DIS, and NYU-v2, demonstrate the superiority of our approach. Also,
experimental results show that our 3D-to-2D distillation improves the model
generalization.

该论文提出了一种新的室内场景语义分割方法，使用一个新的 3D-to-2D 分解框架，可以从大规模的 3D 数据中提取 3D 特征，以增强从 RGB 图像中提取的 2D 特征，并通过对其进行标准化和语义相关的对抗性训练对其进行改进。