Human affective behavior analysis has received much attention in
human-computer interaction (HCI). In this paper, we introduce our submission to
the CVPR 2022 Competition on Affective Behavior Analysis in-the-wild (ABAW). To
fully exploit affective knowledge from multiple views, we utilize the
multimodal features of spoken words, speech prosody, and facial expression,
which are extracted from the video clips in the Aff-Wild2 dataset. Based on
these features, we propose a unified transformer-based multimodal framework for
Action Unit detection and also expression recognition. Specifically, the static
vision feature is first encoded from the current frame image. At the same time,
we clip its adjacent frames by a sliding window and extract three kinds of
multimodal features from the sequence of images, audio, and text. Then, we
introduce a transformer-based fusion module that integrates the static vision
features and the dynamic multimodal features. The cross-attention module in the
fusion module makes the output integrated features focus on the crucial parts
that facilitate the downstream detection tasks. We also leverage some data
balancing techniques, data augmentation techniques, and postprocessing methods
to further improve the model performance. In the official test of ABAW3
Competition, our model ranks first in the EXPR and AU tracks. The extensive
quantitative evaluations, as well as ablation studies on the Aff-Wild2 dataset,
prove the effectiveness of our proposed method.

本研究介绍了一种基于多模态特征和 Transformer 的统一框架，用于情感行为分析和表情识别任务，同时运用数据平衡和增强方法进一步提高模型性能。实验证明，该方法在 ABAW3 竞赛中在 EXPR 和 AU 方面排名第一。

基于 Transformer 的多模态信息融合用于面部表情分析

Transformer-based Multimodal Information Fusion for Facial Expression  Analysis

Recent vision-language (VL) studies have shown remarkable progress by
learning generic representations from massive image-text pairs with transformer
models and then fine-tuning on downstream VL tasks. While existing research has
been focused on achieving high accuracy with large pre-trained models, building
a lightweight model is of great value in practice but is less explored. In this
paper, we propose a smaller and faster VL model, MiniVLM, which can be
finetuned with good performance on various downstream tasks like its larger
counterpart. MiniVLM consists of two modules, a vision feature extractor and a
transformer-based vision-language fusion module. We design a Two-stage
Efficient feature Extractor (TEE), inspired by the one-stage EfficientDet
network, to significantly reduce the time cost of visual feature extraction by
$95\%$, compared to a baseline model. We adopt the MiniLM structure to reduce
the computation cost of the transformer module after comparing different
compact BERT models. In addition, we improve the MiniVLM pre-training by adding
$7M$ Open Images data, which are pseudo-labeled by a state-of-the-art
captioning model. We also pre-train with high-quality image tags obtained from
a strong tagging model to enhance cross-modality alignment. The large models
are used offline without adding any overhead in fine-tuning and inference. With
the above design choices, our MiniVLM reduces the model size by $73\%$ and the
inference time cost by $94\%$ while being able to retain $94-97\%$ of the
accuracy on multiple VL tasks. We hope that MiniVLM helps ease the use of the
state-of-the-art VL research for on-the-edge applications.

本文提出 MiniVLM，一个轻量、快速的视觉 - 语言模型，采用 two-stage efficient feature extractor 和 MiniLM 结构。MiniVLM 与大型模型相比，模型大小减少 73％，推理时间成本降低 94％，在多个视觉 - 语言任务上准确率保持 94-97％。希望 MiniVLM 可以用于边缘应用。