With wearing masks becoming a new cultural norm, facial expression
recognition (FER) while taking masks into account has become a significant
challenge. In this paper, we propose a unified multi-branch vision transformer
for facial expression recognition and mask wearing classification tasks. Our
approach extracts shared features for both tasks using a dual-branch
architecture that obtains multi-scale feature representations. Furthermore, we
propose a cross-task fusion phase that processes tokens for each task with
separate branches, while exchanging information using a cross attention module.
Our proposed framework reduces the overall complexity compared with using
separate networks for both tasks by the simple yet effective cross-task fusion
phase. Extensive experiments demonstrate that our proposed model performs
better than or on par with different state-of-the-art methods on both facial
expression recognition and facial mask wearing classification task.

我们提出了一种统一的多分支视觉转换器，用于面部表情识别和口罩佩戴分类任务。我们的方法使用双分支架构提取两个任务的共享特征，获得多尺度特征表示。此外，我们提出了一个跨任务融合阶段，在交换信息的同时，使用交叉注意模块处理每个任务的标记。与为两个任务使用单独的网络相比，我们提出的框架通过简单而有效的跨任务融合阶段降低了总体复杂性。广泛的实验证明，我们提出的模型在面部表情识别和口罩佩戴分类任务上的表现优于或与不同的最新方法相当。