Algorithmic detection of facial palsy offers the potential to improve current practices, which usually involve labor-intensive and subjective assessments by clinicians. In this paper, we present a multimodal fusion-based deep learning model that utilizes an MLP mixer-based model to process unstructured data (i.e. RGB images or images with facial line segments) and a feed-forward neural network to process structured data (i.e. facial landmark coordinates, features of facial expressions, or handcrafted features) for detecting facial palsy. We then contribute to a study to analyze the effect of different data modalities and the benefits of a multimodal fusion-based approach using videos of 20 facial palsy patients and 20 healthy subjects. Our multimodal fusion model achieved 96.00 F1, which is significantly higher than the feed-forward neural network trained on handcrafted features alone (82.80 F1) and an MLP mixer-based model trained on raw RGB images (89.00 F1).

本研究解决了面瘫检测中传统主观评估的不足，通过提出一种多模态融合的深度学习模型，结合MLP Mixer和结构化数据处理。实验结果显示，该模型的F1得分达96.00，显著优于仅使用手工特征的前馈神经网络和基于原始RGB图像的MLP Mixer模型，展示了多模态方法在提高面瘫检测准确性方面的潜力。

利用多模态融合模型和手工特征的深度学习网络进行面瘫检测