In this paper, we propose a novel approach for generalized zero-shot learning
in a multi-modal setting, where we have novel classes of audio/video during
testing that are not seen during training. We use the semantic relatedness of
text embeddings as a means for zero-shot learning by aligning audio and video
embeddings with the corresponding class label text feature space. Our approach
uses a cross-modal decoder and a composite triplet loss. The cross-modal
decoder enforces a constraint that the class label text features can be
reconstructed from the audio and video embeddings of data points. This helps
the audio and video embeddings to move closer to the class label text
embedding. The composite triplet loss makes use of the audio, video, and text
embeddings. It helps bring the embeddings from the same class closer and push
away the embeddings from different classes in a multi-modal setting. This helps
the network to perform better on the multi-modal zero-shot learning task.
Importantly, our multi-modal zero-shot learning approach works even if a
modality is missing at test time. We test our approach on the generalized
zero-shot classification and retrieval tasks and show that our approach
outperforms other models in the presence of a single modality as well as in the
presence of multiple modalities. We validate our approach by comparing it with
previous approaches and using various ablations.

本文提出了一种新的多模式推广零样本学习方法，使用文本嵌入的语义相关性将音频和视频嵌入与相应类别标签文本特征空间对齐，使用交叉模态解码器和组合三元组损失，在多模态设置中来执行零样本学习，实验证明该方法在各种模态下的推广零样本分类和检索任务中都优于其他模型。