Analyzing individual emotions during group conversation is crucial in
developing intelligent agents capable of natural human-machine interaction.
While reliable emotion recognition techniques depend on different modalities
(text, audio, video), the inherent heterogeneity between these modalities and
the dynamic cross-modal interactions influenced by an individual's unique
behavioral patterns make the task of emotion recognition very challenging. This
difficulty is compounded in group settings, where the emotion and its temporal
evolution are not only influenced by the individual but also by external
contexts like audience reaction and context of the ongoing conversation. To
meet this challenge, we propose a Multimodal Attention Network that captures
cross-modal interactions at various levels of spatial abstraction by jointly
learning its interactive bunch of mode-specific Peripheral and Central
networks. The proposed MAN injects cross-modal attention via its Peripheral
key-value pairs within each layer of a mode-specific Central query network. The
resulting cross-attended mode-specific descriptors are then combined using an
Adaptive Fusion technique that enables the model to integrate the
discriminative and complementary mode-specific data patterns within an
instance-specific multimodal descriptor. Given a dialogue represented by a
sequence of utterances, the proposed AMuSE model condenses both spatial and
temporal features into two dense descriptors: speaker-level and
utterance-level. This helps not only in delivering better classification
performance (3-5% improvement in Weighted-F1 and 5-7% improvement in Accuracy)
in large-scale public datasets but also helps the users in understanding the
reasoning behind each emotion prediction made by the model via its Multimodal
Explainability Visualization module.

通过提出一种多模态注意力网络，将跨模态注意力注入每个层级的特定模态中，从而整合多模态数据特征，实现情感识别，并通过多模态可解释性可视化模块使模型的情感预测能够理解其推理过程。

AMuSE: 面向群体对话的自适应多模态情感分析

AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in  Group Conversations

ICD coding is designed to assign the disease codes to electronic health
records (EHRs) upon discharge, which is crucial for billing and clinical
statistics. In an attempt to improve the effectiveness and efficiency of manual
coding, many methods have been proposed to automatically predict ICD codes from
clinical notes. However, most previous works ignore the decisive information
contained in structured medical data in EHRs, which is hard to be captured from
the noisy clinical notes. In this paper, we propose a Tree-enhanced Multimodal
Attention Network (TreeMAN) to fuse tabular features and textual features into
multimodal representations by enhancing the text representations with
tree-based features via the attention mechanism. Tree-based features are
constructed according to decision trees learned from structured multimodal
medical data, which capture the decisive information about ICD coding. We can
apply the same multi-label classifier from previous text models to the
multimodal representations to predict ICD codes. Experiments on two MIMIC
datasets show that our method outperforms prior state-of-the-art ICD coding
approaches. The code is available at this https URL

本研究提出了一种基于 Tree-enhanced Multimodal Attention Network (TreeMAN) 的方法，将 EHR 中的结构化医疗数据和文本特征融合成多模态表示。通过决策树学习，TreeMAN 能够更好地捕捉 ICD 编码的决定性信息。实验证明，该方法优于现有 ICD 编码方法。