With the strong robusticity on illumination variations, near-infrared (NIR)
can be an effective and essential complement to visible (VIS) facial expression
recognition in low lighting or complete darkness conditions. However, facial
expression recognition (FER) from NIR images presents more challenging problem
than traditional FER due to the limitations imposed by the data scale and the
difficulty of extracting discriminative features from incomplete visible
lighting contents. In this paper, we give the first attempt to deep NIR facial
expression recognition and proposed a novel method called near-infrared facial
expression transformer (NFER-Former). Specifically, to make full use of the
abundant label information in the field of VIS, we introduce a Self-Attention
Orthogonal Decomposition mechanism that disentangles the expression information
and spectrum information from the input image, so that the expression features
can be extracted without the interference of spectrum variation. We also
propose a Hypergraph-Guided Feature Embedding method that models some key
facial behaviors and learns the structure of the complex correlations between
them, thereby alleviating the interference of inter-class similarity.
Additionally, we have constructed a large NIR-VIS Facial Expression dataset
that includes 360 subjects to better validate the efficiency of NFER-Former.
Extensive experiments and ablation studies show that NFER-Former significantly
improves the performance of NIR FER and achieves state-of-the-art results on
the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.

本文首次尝试深度近红外 (NIR) 面部表情识别，提出了一种名为近红外面部表情转换器 (NFER-Former) 的新方法。通过引入自注意力正交分解机制和超图引导特征嵌入方法，可以在缺失可见光信息的情况下提取表情特征，并构建大型 NIR-VIS 面部表情数据集来验证 NFER-Former 的效率。通过广泛的实验和剔除研究，证明 NFER-Former 明显改善了 NIR 面部表情识别性能，并在 Oulu-CASIA 和 Large-HFE 这两个唯一可用的 NIR 面部表情识别数据集上取得了最新的成果。