Human multimodal emotion recognition (MER) aims to perceive human emotions
via language, visual and acoustic modalities. Despite the impressive
performance of previous MER approaches, the inherent multimodal heterogeneities
still haunt and the contribution of different modalities varies significantly.
In this work, we mitigate this issue by proposing a decoupled multimodal
distillation (DMD) approach that facilitates flexible and adaptive crossmodal
knowledge distillation, aiming to enhance the discriminative features of each
modality. Specially, the representation of each modality is decoupled into two
parts, i.e., modality-irrelevant/-exclusive spaces, in a self-regression
manner. DMD utilizes a graph distillation unit (GD-Unit) for each decoupled
part so that each GD can be performed in a more specialized and effective
manner. A GD-Unit consists of a dynamic graph where each vertice represents a
modality and each edge indicates a dynamic knowledge distillation. Such GD
paradigm provides a flexible knowledge transfer manner where the distillation
weights can be automatically learned, thus enabling diverse crossmodal
knowledge transfer patterns. Experimental results show DMD consistently obtains
superior performance than state-of-the-art MER methods. Visualization results
show the graph edges in DMD exhibit meaningful distributional patterns w.r.t.
the modality-irrelevant/-exclusive feature spaces. Codes are released at
https://github.com/mdswyz/DMD.

本文提出了一种去耦合的多模态蒸馏（DMD）方法，通过使用图蒸馏单元（GD-Unit）和构建动态图来实现灵活和自适应的多模态知识蒸馏，以提高每种模态的辨别特征。实验结果表明，DMD 方法优于最先进的 MER 方法，并且在 DMD 中的图边展现出与无关 / 专用空间以及有意义的分布模式。