Weakly-supervised audio-visual video parsing (WS-AVVP) aims to localize the
temporal extents of audio, visual and audio-visual event instances as well as
identify the corresponding event categories with only video-level category
labels for training. Most previous methods pay much attention to refining the
supervision for each modality or extracting fruitful cross-modality information
for more reliable feature learning. None of them have noticed the imbalanced
feature learning between different modalities in the task. In this paper, to
balance the feature learning processes of different modalities, a dynamic
gradient modulation (DGM) mechanism is explored, where a novel and effective
metric function is designed to measure the imbalanced feature learning between
audio and visual modalities. Furthermore, principle analysis indicates that the
multimodal confusing calculation will hamper the precise measurement of
multimodal imbalanced feature learning, which further weakens the effectiveness
of our DGM mechanism. To cope with this issue, a modality-separated decision
unit (MSDU) is designed for more precise measurement of imbalanced feature
learning between audio and visual modalities. Comprehensive experiments are
conducted on public benchmarks and the corresponding experimental results
demonstrate the effectiveness of our proposed method.

本文提出了一种利用动态梯度调制机制来平衡不同语音和视觉模态特征学习的方法，并设计了一种实现精确测量的模态分离决策单元来处理多模式混淆问题， 实验证明该方法的有效性。