2D+3D facial expression recognition (FER) can effectively cope with
illumination changes and pose variations by simultaneously merging 2D texture
and more robust 3D depth information. Most deep learning-based approaches
employ the simple fusion strategy that concatenates the multimodal features
directly after fully-connected layers, without considering the different
degrees of significance for each modality. Meanwhile, how to focus on both 2D
and 3D local features in salient regions is still a great challenge. In this
letter, we propose the adaptive fusion network with masks (AFNet-M) for 2D+3D
FER. To enhance 2D and 3D local features, we take the masks annotating salient
regions of the face as prior knowledge and design the mask attention module
(MA) which can automatically learn two modulation vectors to adjust the feature
maps. Moreover, we introduce a novel fusion strategy that can perform adaptive
fusion at convolutional layers through the designed importance weights
computing module (IWC). Experimental results demonstrate that our AFNet-M
achieves the state-of-the-art performance on BU-3DFE and Bosphorus datasets and
requires fewer parameters in comparison with other models.

本研究提出了一种自适应融合网络，利用人脸显著区域的掩码来增强 2D 和 3D 局部特征，通过设计的调制向量自动学习，以及计算重要性权重的新型融合策略可以在卷积层中执行自适应融合。实验结果表明，该方法在 BU-3DFE 和 Bosphorus 数据集上具有最先进的性能并且需要较少的参数。