We present two multimodal fusion-based deep learning models that consume ASR
transcribed speech and acoustic data simultaneously to classify whether a
speaker in a structured diagnostic task has Alzheimer's Disease and to what
degree, evaluating the ADReSSo challenge 2021 data. Our best model, a BiLSTM
with highway layers using words, word probabilities, disfluency features, pause
information, and a variety of acoustic features, achieves an accuracy of 84%
and RSME error prediction of 4.26 on MMSE cognitive scores. While predicting
cognitive decline is more challenging, our models show improvement using the
multimodal approach and word probabilities, disfluency and pause information
over word-only models. We show considerable gains for AD classification using
multimodal fusion and gating, which can effectively deal with noisy inputs from
acoustic features and ASR hypotheses.

本文介绍了两种基于多模态融合的深度学习模型，同时使用 ASR 转录的语音和声纹数据，对结构化诊断任务中的说话人是否存在阿尔茨海默病进行分类，并评估了 ADReSSo 挑战 2021 数据，其中最佳模型 BiLSTM 采用了包括单词、单词概率、口吃特征、停顿信息和各种声学特征的 highway 层，实现了 84％ 的准确率和 4.26 的 MMSE 认知分数预测 RSME 误差。通过使用多模态方法和单词概率、口吃和停顿信息，我们的模型在预测认知衰退时表现出了提高，并且对于使用多模态融合和门控技术进行的 AD 分类，我们展示了相当大的收益。