Generalized zero-shot skeleton-based action recognition (GZSSAR) is a new
challenging problem in computer vision community, which requires models to
recognize actions without any training samples. Previous studies only utilize
the action labels of verb phrases as the semantic prototypes for learning the
mapping from skeleton-based actions to a shared semantic space. However, the
limited semantic information of action labels restricts the generalization
ability of skeleton features for recognizing unseen actions. In order to solve
this dilemma, we propose a multi-semantic fusion (MSF) model for improving the
performance of GZSSAR, where two kinds of class-level textual descriptions
(i.e., action descriptions and motion descriptions), are collected as auxiliary
semantic information to enhance the learning efficacy of generalizable skeleton
features. Specially, a pre-trained language encoder takes the action
descriptions, motion descriptions and original class labels as inputs to obtain
rich semantic features for each action class, while a skeleton encoder is
implemented to extract skeleton features. Then, a variational autoencoder (VAE)
based generative module is performed to learn a cross-modal alignment between
skeleton and semantic features. Finally, a classification module is built to
recognize the action categories of input samples, where a seen-unseen
classification gate is adopted to predict whether the sample comes from seen
action classes or not in GZSSAR. The superior performance in comparisons with
previous models validates the effectiveness of the proposed MSF model on
GZSSAR.

提出了一种用于提高广义零样本基于骨骼的动作识别（GZSSAR）性能的多语义融合（MSF）模型，其中采集了两种类级文本描述（动作描述和运动描述）作为辅助语义信息，以增强通用骨骼特征的学习能力。通过预训练的语言编码器和骨骼编码器提取富有语义特征的动作类别和骨骼特征，并利用基于变分自动编码器（VAE）的生成模块学习骨骼和语义特征之间的跨模态对齐，最后构建分类模块识别输入样本的动作类别，并在 GZSSAR 中采用看见未见分类门来预测样本是否来自已知动作类别。与先前模型相比，提出的 MSF 模型在 GZSSAR 上表现出卓越的性能。