The activations of Facial Action Units (AUs) mutually influence one another.
While the relationship between a pair of AUs can be complex and unique,
existing approaches fail to specifically and explicitly represent such cues for
each pair of AUs in each facial display. This paper proposes an AU relationship
modelling approach that deep learns a unique graph to explicitly describe the
relationship between each pair of AUs of the target facial display. Our
approach first encodes each AU's activation status and its association with
other AUs into a node feature. Then, it learns a pair of multi-dimensional edge
features to describe multiple task-specific relationship cues between each pair
of AUs. During both node and edge feature learning, our approach also considers
the influence of the unique facial display on AUs' relationship by taking the
full face representation as an input. Experimental results on BP4D and DISFA
datasets show that both node and edge feature learning modules provide large
performance improvements for CNN and transformer-based backbones, with our best
systems achieving the state-of-the-art AU recognition results. Our approach not
only has a strong capability in modelling relationship cues for AU recognition
but also can be easily incorporated into various backbones. Our PyTorch code is
made available.

本文提出了一种基于深度学习的 AU 关系建模方法，通过学习多维边特征与节点特征来描述面部表情不同组合之间的关系特征，从而实现更准确的 AU 识别。我们在 BP4D 和 DISFA 数据集上的实验结果表明，我们的方法不仅在各项识别指标上取得最优结果，而且可以轻松嵌入到不同的模型中。

基于多维边缘特征的面部表情动作单元关系图学习

Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition

This paper introduces our approach to the EmotioNet Challenge 2020. We pose
the AU recognition problem as a multi-task learning problem, where the
non-rigid facial muscle motion (mainly the first 17 AUs) and the rigid head
motion (the last 6 AUs) are modeled separately. The co-occurrence of the
expression features and the head pose features are explored. We observe that
different AUs converge at various speed. By choosing the optimal checkpoint for
each AU, the recognition results are improved. We are able to obtain a final
score of 0.746 in validation set and 0.7306 in the test set of the challenge.

该研究论文讨论了 EmotioNet Challenge 2020 上的 AU 识别问题，并将其视为一项多任务学习问题，通过分别对非刚性面部肌肉运动（主要是前 17 个 AUs）和刚性头部运动（最后 6 个 AUs）进行建模。研究观察了表情特征和头部姿态特征的共现关系。通过选择最佳的检查点来进行 AU 识别，提高了识别结果，最终在验证集和测试集中获得了分别为 0.746 和 0.7306 的得分。

2020 TAL EmotioNet 挑战赛：多任务学习中模型选择问题的重新思考

TAL EmotioNet Challenge 2020 Rethinking the Model Chosen Problem in  Multi-Task Learning

Recognizing facial action units (AUs) from spontaneous facial expressions is
still a challenging problem. Most recently, CNNs have shown promise on facial
AU recognition. However, the learned CNNs are often overfitted and do not
generalize well to unseen subjects due to limited AU-coded training images. We
proposed a novel Incremental Boosting CNN (IB-CNN) to integrate boosting into
the CNN via an incremental boosting layer that selects discriminative neurons
from the lower layer and is incrementally updated on successive mini-batches.
In addition, a novel loss function that accounts for errors from both the
incremental boosted classifier and individual weak classifiers was proposed to
fine-tune the IB-CNN. Experimental results on four benchmark AU databases have
demonstrated that the IB-CNN yields significant improvement over the
traditional CNN and the boosting CNN without incremental learning, as well as
outperforming the state-of-the-art CNN-based methods in AU recognition. The
improvement is more impressive for the AUs that have the lowest frequencies in
the databases.

本研究提出了一种新颖的增量增强 CNN（IB-CNN），使其通过增量增强层将增强集成到 CNN 中，并选择下层的识别性神经元，在连续的小批量中进行增量更新。此外，还提出了一种新的损失函数，考虑了来自增量增强分类器和单个弱分类器的误差，以微调 IB-CNN，实验结果表明，IB-CNN 优于传统 CNN 和增强 CNN，并且在 AU 识别方面胜过 CNN-based 方法的最新技术。