One of the major reasons for misclassification of multiplex actions during
action recognition is the unavailability of complementary features that provide
the semantic information about the actions. In different domains these features
are present with different scales and intensities. In existing literature,
features are extracted independently in different domains, but the benefits
from fusing these multidomain features are not realized. To address this
challenge and to extract complete set of complementary information, in this
paper, we propose a novel multidomain multimodal fusion framework that extracts
complementary and distinct features from different domains of the input
modality. We transform input inertial data into signal images, and then make
the input modality multidomain and multimodal by transforming spatial domain
information into frequency and time-spectrum domain using Discrete Fourier
Transform (DFT) and Gabor wavelet transform (GWT) respectively. Features in
different domains are extracted by Convolutional Neural networks (CNNs) and
then fused by Canonical Correlation based Fusion (CCF) for improving the
accuracy of human action recognition. Experimental results on three inertial
datasets show the superiority of the proposed method when compared to the
state-of-the-art.

本文提出了一个多域多模态融合框架，利用卷积神经网络和规范相关融合来提取不同域的特征以识别多重动作，并将输入惯性数据转换为信号图像进行处理，并在三个惯性数据集上得到了比现有文献更准确的结果。