Agents must monitor their partners' affective states continuously in order to
understand and engage in social interactions. However, methods for evaluating
affect recognition do not account for changes in classification performance
that may occur during occlusions or transitions between affective states. This
paper addresses temporal patterns in affect classification performance in the
context of an infant-robot interaction, where infants' affective states
contribute to their ability to participate in a therapeutic leg movement
activity. To support robustness to facial occlusions in video recordings, we
trained infant affect recognition classifiers using both facial and body
features. Next, we conducted an in-depth analysis of our best-performing models
to evaluate how performance changed over time as the models encountered missing
data and changing infant affect. During time windows when features were
extracted with high confidence, a unimodal model trained on facial features
achieved the same optimal performance as multimodal models trained on both
facial and body features. However, multimodal models outperformed unimodal
models when evaluated on the entire dataset. Additionally, model performance
was weakest when predicting an affective state transition and improved after
multiple predictions of the same affective state. These findings emphasize the
benefits of incorporating body features in continuous affect recognition for
infants. Our work highlights the importance of evaluating variability in model
performance both over time and in the presence of missing data when applying
affect recognition to social interactions.

本研究探讨了影响婴儿情绪识别准确度的因素，并证明在处理时间和数据缺失方面，采用多模态模型（脸部和身体特征）胜过单模态模型（仅脸部特征）以实现持续情绪识别。