In this work, we focus on a special group of human body language -- the
micro-gesture (MG), which differs from the range of ordinary illustrative
gestures in that they are not intentional behaviors performed to convey
information to others, but rather unintentional behaviors driven by inner
feelings. This characteristic introduces two novel challenges regarding
micro-gestures that are worth rethinking. The first is whether strategies
designed for other action recognition are entirely applicable to
micro-gestures. The second is whether micro-gestures, as supplementary data,
can provide additional insights for emotional understanding. In recognizing
micro-gestures, we explored various augmentation strategies that take into
account the subtle spatial and brief temporal characteristics of
micro-gestures, often accompanied by repetitiveness, to determine more suitable
augmentation methods. Considering the significance of temporal domain
information for micro-gestures, we introduce a simple and efficient
plug-and-play spatiotemporal balancing fusion method. We not only studied our
method on the considered micro-gesture dataset but also conducted experiments
on mainstream action datasets. The results show that our approach performs well
in micro-gesture recognition and on other datasets, achieving state-of-the-art
performance compared to previous micro-gesture recognition methods. For
emotional understanding based on micro-gestures, we construct complex emotional
reasoning scenarios. Our evaluation, conducted with large language models,
shows that micro-gestures play a significant and positive role in enhancing
comprehensive emotional understanding. The scenarios we developed can be
extended to other micro-gesture-based tasks such as deception detection and
interviews. We confirm that our new insights contribute to advancing research
in micro-gesture and emotional artificial intelligence.

本研究关注一类特殊的人体语言，即微小手势（MG），针对微小手势的动作识别和情感理解进行了研究，提出了多种增强策略和时空平衡融合方法，并通过实验证明了这些方法在微小手势识别和情感理解中的有效性。

基于微动作理解的无身份人工情绪智能

Identity-free Artificial Emotional Intelligence via Micro-Gesture  Understanding

We introduce a new dataset for the emotional artificial intelligence
research: identity-free video dataset for Micro-Gesture Understanding and
Emotion analysis (iMiGUE). Different from existing public datasets, iMiGUE
focuses on nonverbal body gestures without using any identity information,
while the predominant researches of emotion analysis concern sensitive
biometric data, like face and speech. Most importantly, iMiGUE focuses on
micro-gestures, i.e., unintentional behaviors driven by inner feelings, which
are different from ordinary scope of gestures from other gesture datasets which
are mostly intentionally performed for illustrative purposes. Furthermore,
iMiGUE is designed to evaluate the ability of models to analyze the emotional
states by integrating information of recognized micro-gesture, rather than just
recognizing prototypes in the sequences separately (or isolatedly). This is
because the real need for emotion AI is to understand the emotional states
behind gestures in a holistic way. Moreover, to counter for the challenge of
imbalanced sample distribution of this dataset, an unsupervised learning method
is proposed to capture latent representations from the micro-gesture sequences
themselves. We systematically investigate representative methods on this
dataset, and comprehensive experimental results reveal several interesting
insights from the iMiGUE, e.g., micro-gesture-based analysis can promote
emotion understanding. We confirm that the new iMiGUE dataset could advance
studies of micro-gesture and emotion AI.

本文介绍了一个面向情感人工智能研究的新数据集：iMiGUE，它专注于基于非语言的微小手势，旨在综合评估模型分析情感状态的能力，通过引入无监督学习方法解决了样本分布不平衡问题，并揭示了基于微小手势的情感分析研究的新见解。