Algorithmic detection of facial palsy offers the potential to improve current
practices, which usually involve labor-intensive and subjective assessment by
clinicians. In this paper, we present a multimodal fusion-based deep learning
model that utilizes unstructured data (i.e. an image frame with facial line
segments) and structured data (i.e. features of facial expressions) to detect
facial palsy. We then contribute to a study to analyze the effect of different
data modalities and the benefits of a multimodal fusion-based approach using
videos of 21 facial palsy patients. Our experimental results show that among
various data modalities (i.e. unstructured data - RGB images and images of
facial line segments and structured data - coordinates of facial landmarks and
features of facial expressions), the feed-forward neural network using features
of facial expression achieved the highest precision of 76.22 while the
ResNet-based model using images of facial line segments achieved the highest
recall of 83.47. When we leveraged both images of facial line segments and
features of facial expressions, our multimodal fusion-based deep learning model
slightly improved the precision score to 77.05 at the expense of a decrease in
the recall score.

通过融合多模态深度学习模型，利用非结构化数据（即带有面部线段的图像帧）和结构化数据（即面部表情特征），本研究旨在检测面部麻痹，并分析不同数据模式和多模态融合方法的效果，实验证明在不同数据模式中使用面部表情特征的前馈神经网络具有最高的准确率，为 76.22，而使用面部线段图像的基于 ResNet 的模型具有最高的召回率，为 83.47。当同时利用面部线段图像和面部表情特征时，多模态融合深度学习模型稍微提高了准确率至 77.05，但减少了召回率。