Automatic facial action unit (AU) recognition is a challenging task due to
the scarcity of manual annotations. To alleviate this problem, a large amount
of efforts has been dedicated to exploiting various methods which leverage
numerous unlabeled data. However, many aspects with regard to some unique
properties of AUs, such as the regional and relational characteristics, are not
sufficiently explored in previous works. Motivated by this, we take the AU
properties into consideration and propose two auxiliary AU related tasks to
bridge the gap between limited annotations and the model performance in a
self-supervised manner via the unlabeled data. Specifically, to enhance the
discrimination of regional features with AU relation embedding, we design a
task of RoI inpainting to recover the randomly cropped AU patches. Meanwhile, a
single image based optical flow estimation task is proposed to leverage the
dynamic change of facial muscles and encode the motion information into the
global feature representation. Based on these two self-supervised auxiliary
tasks, local features, mutual relation and motion cues of AUs are better
captured in the backbone network with the proposed regional and temporal based
auxiliary task learning (RTATL) framework. Extensive experiments on BP4D and
DISFA demonstrate the superiority of our method and new state-of-the-art
performances are achieved.

本文提出了一种基于自监督学习方法的区域和时序辅助任务学习框架（RTATL），通过考虑面部动作单元（AU）的属性，设计了 RoI 修复和基于单张图像的光流估计两个辅助任务，以更好地捕捉 AU 的局部特征、相互关系和运动线索，并在 BP4D 和 DISFA 数据集上进行了广泛的实验，在 AU 识别方面取得了最新的最优性能。