Automatic audio-visual expression recognition can play an important role in
communication services such as tele-health, VOIP calls and human-machine
interaction. Accuracy of audio-visual expression recognition could benefit from
the interplay between the two modalities. However, most audio-visual expression
recognition systems, trained in ideal conditions, fail to generalize in real
world scenarios where either the audio or visual modality could be missing due
to a number of reasons such as limited bandwidth, interactors' orientation,
caller initiated muting. This paper studies the performance of a state-of-the
art transformer when one of the modalities is missing. We conduct ablation
studies to evaluate the model in the absence of either modality. Further, we
propose a strategy to randomly ablate visual inputs during training at the clip
or frame level to mimic real world scenarios. Results conducted on in-the-wild
data, indicate significant generalization in proposed models trained on missing
cues, with gains up to 17% for frame level ablations, showing that these
training strategies cope better with the loss of input modalities.

研究了当其中一个模态缺失时，自动音视频表情识别中 transformer 模型的表现，通过消融实验和随机消融训练数据的策略，提高了模型的泛化性能。