A micro-expression is a spontaneous unconscious facial muscle movement that
can reveal the true emotions people attempt to hide. Although manual methods
have made good progress and deep learning is gaining prominence. Due to the
short duration of micro-expression and different scales of expressed in facial
regions, existing algorithms cannot extract multi-modal multi-scale facial
region features while taking into account contextual information to learn
underlying features. Therefore, in order to solve the above problems, a
multi-modal multi-scale algorithm based on transformer network is proposed in
this paper, aiming to fully learn local multi-grained features of
micro-expressions through two modal features of micro-expressions - motion
features and texture features. To obtain local area features of the face at
different scales, we learned patch features at different scales for both
modalities, and then fused multi-layer multi-headed attention weights to obtain
effective features by weighting the patch features, and combined cross-modal
contrastive learning for model optimization. We conducted comprehensive
experiments on three spontaneous datasets, and the results show the accuracy of
the proposed algorithm in single measurement SMIC database is up to 78.73% and
the F1 value on CASMEII of the combined database is up to 0.9071, which is at
the leading level.

本文提出了一种基于 Transformer 网络的多模态多尺度算法，通过微表情的运动和纹理两种特征完全学习微表情的局部多粒度特征，实现了不同尺度脸部局部区域特征的提取并在三个自然数据集上进行了综合实验。结果表明，该算法在单个测量 SMIC 数据库中的准确度高达 78.73％，在合并数据库的 CASMEII 上的 F1 值高达 0.9071，处于领先水平。