In this study, we investigate how environmental factors, specifically the
scenes and objects involved, can affect the expression of emotions through body
language. To this end, we introduce a novel multi-stream deep convolutional
neural network named BEE-NET. We also propose a new late fusion strategy that
incorporates meta-information on places and objects as prior knowledge in the
learning process. Our proposed probabilistic pooling model leverages this
information to generate a joint probability distribution of both available and
anticipated non-available contextual information in latent space. Importantly,
our fusion strategy is differentiable, allowing for end-to-end training and
capturing of hidden associations among data points without requiring further
post-processing or regularisation. To evaluate our deep model, we use the Body
Language Database (BoLD), which is currently the largest available database for
the Automatic Identification of the in-the-wild Bodily Expression of Emotions
(AIBEE). Our experimental results demonstrate that our proposed approach
surpasses the current state-of-the-art in AIBEE by a margin of 2.07%, achieving
an Emotional Recognition Score of 66.33%.

本研究探讨了环境因素，特别是场景和物体如何通过身体语言影响情绪表达。为此，我们引入了一种名为 BEE-NET 的新型多流深度卷积神经网络，并提出了一种新的后期融合策略，在学习过程中将场所和物体的元信息作为先验知识纳入考虑。实验结果表明，我们的深度模型在情感识别领域取得了显著的突破，超过了当前最先进的技术，情绪识别得分达到了 66.33%。

BEE-NET：识别野外身体表达情感的深度神经网络

BEE-NET: A deep neural network to identify in-the-wild Bodily Expression  of Emotions

Multimodal-based action recognition methods have achieved high success using
pose and RGB modality. However, skeletons sequences lack appearance depiction
and RGB images suffer irrelevant noise due to modality limitations. To address
this, we introduce human parsing feature map as a novel modality, since it can
selectively retain effective semantic features of the body parts, while
filtering out most irrelevant noise. We propose a new dual-branch framework
called Ensemble Human Parsing and Pose Network (EPP-Net), which is the first to
leverage both skeletons and human parsing modalities for action recognition.
The first human pose branch feeds robust skeletons in graph convolutional
network to model pose features, while the second human parsing branch also
leverages depictive parsing feature maps to model parsing festures via
convolutional backbones. The two high-level features will be effectively
combined through a late fusion strategy for better action recognition.
Extensive experiments on NTU RGB+D and NTU RGB+D 120 benchmarks consistently
verify the effectiveness of our proposed EPP-Net, which outperforms the
existing action recognition methods. Our code is available at:
this https URL

采用人体解析特征图作为一种新的模态，结合骨架和人体解析模态，提出了一种新的双分支框架（EPP-Net），通过后期融合策略有效地结合两种高级特征以实现更好的动作识别。

探索人工解析模式用于动作识别

Explore Human Parsing Modality for Action Recognition

In this work, we propose an ensemble modeling approach for multimodal action
recognition. We independently train individual modality models using a variant
of focal loss tailored to handle the long-tailed distribution of the MECCANO
[21] dataset. Based on the underlying principle of focal loss, which captures
the relationship between tail (scarce) classes and their prediction
difficulties, we propose an exponentially decaying variant of focal loss for
our current task. It initially emphasizes learning from the hard misclassified
examples and gradually adapts to the entire range of examples in the dataset.
This annealing process encourages the model to strike a balance between
focusing on the sparse set of hard samples, while still leveraging the
information provided by the easier ones. Additionally, we opt for the late
fusion strategy to combine the resultant probability distributions from RGB and
Depth modalities for final action prediction. Experimental evaluations on the
MECCANO dataset demonstrate the effectiveness of our approach.

我们在这项工作中提出了一种用于多模态动作识别的集成建模方法，通过使用适应 MECCANO 数据集的长尾分布的变种焦点损失模型，独立训练各个模态的模型，并基于焦点损失的基本原理，提出了一种指数衰减的焦点损失模型，它能够根据数据集中的所有示例逐渐适应，并鼓励模型在关注稀疏的艰难样本的同时，利用简单样本提供的信息，最后采用后期融合策略将 RGB 和深度模态的概率分布进行组合，以进行最终的动作预测。在 MECCANO 数据集上进行的实验评估证明了我们方法的有效性。