Aphasia, a language disorder resulting from brain damage, requires accurate
identification of specific aphasia types, such as Broca's and Wernicke's
aphasia, for effective treatment. However, little attention has been paid to
developing methods to detect different types of aphasia. Recognizing the
importance of analyzing co-speech gestures for distinguish aphasia types, we
propose a multimodal graph neural network for aphasia type detection using
speech and corresponding gesture patterns. By learning the correlation between
the speech and gesture modalities for each aphasia type, our model can generate
textual representations sensitive to gesture information, leading to accurate
aphasia type detection. Extensive experiments demonstrate the superiority of
our approach over existing methods, achieving state-of-the-art results (F1
84.2\%). We also show that gesture features outperform acoustic features,
highlighting the significance of gesture expression in detecting aphasia types.
We provide the codes for reproducibility purposes\footnote{Code:
https://github.com/DSAIL-SKKU/Multimodal-Aphasia-Type-Detection_EMNLP_2023}.

我们提出了一种基于语音和相应的手势模式的多模态图神经网络，用于检测失语症类型，通过学习每种失语症类型中语音和手势模态之间的相关性，我们的模型能够生成对手势信息敏感的文本表示，实现了准确的失语症类型检测，实验证明了我们方法的优越性 (达到了 84.2% 的 F1 值)，我们还展示了手势特征优于声学特征，凸显了手势表达在检测失语症类型中的重要性。