Aphasia, a language disorder resulting from brain damage, requires accurate
identification of specific aphasia types, such as Broca's and Wernicke's
aphasia, for effective treatment. However, little attention has been paid to
developing methods to detect different types of aphasia. Recognizing the
importance of analyzing co-speech gestures for distinguish aphasia types, we
propose a multimodal graph neural network for aphasia type detection using
speech and corresponding gesture patterns. By learning the correlation between
the speech and gesture modalities for each aphasia type, our model can generate
textual representations sensitive to gesture information, leading to accurate
aphasia type detection. Extensive experiments demonstrate the superiority of
our approach over existing methods, achieving state-of-the-art results (F1
84.2\%). We also show that gesture features outperform acoustic features,
highlighting the significance of gesture expression in detecting aphasia types.
We provide the codes for reproducibility purposes\footnote{Code:
https://github.com/DSAIL-SKKU/Multimodal-Aphasia-Type-Detection_EMNLP_2023}.

我们提出了一种基于语音和相应的手势模式的多模态图神经网络，用于检测失语症类型，通过学习每种失语症类型中语音和手势模态之间的相关性，我们的模型能够生成对手势信息敏感的文本表示，实现了准确的失语症类型检测，实验证明了我们方法的优越性 (达到了 84.2% 的 F1 值)，我们还展示了手势特征优于声学特征，凸显了手势表达在检测失语症类型中的重要性。

学习用于多模式失语类型检测的共言手势

Learning Co-Speech Gesture for Multimodal Aphasia Type Detection

Autism diagnosis presents a major challenge due to the vast heterogeneity of
the condition and the elusive nature of early detection. Atypical gait and
gesture patterns are dominant behavioral characteristics of autism and can
provide crucial insights for diagnosis. Furthermore, these data can be
collected efficiently in a non-intrusive way, facilitating early intervention
to optimize positive outcomes. Existing research mainly focuses on associating
facial and eye-gaze features with autism. However, very few studies have
investigated movement and gesture patterns which can reveal subtle variations
and characteristics that are specific to autism. To address this gap, we
present an analysis of gesture and gait activity in videos to identify children
with autism and quantify the severity of their condition by regressing autism
diagnostic observation schedule scores. Our proposed architecture addresses two
key factors: (1) an effective feature representation to manifest irregular
gesture patterns and (2) a two-stream co-learning framework to enable a
comprehensive understanding of its relation to autism from diverse perspectives
without explicitly using additional data modality. Experimental results
demonstrate the efficacy of utilizing gesture and gait-activity videos for
autism analysis.

本研究使用姿态和步态活动视频进行自闭症分析，通过回归自闭症诊断观察量表分数，识别患有自闭症的儿童及其条件的严重程度，并提出了一个有效特征来展现不规则的姿态模式，通过双流协同学习框架全面理解姿势和自闭症之间的关系。