Weakly-supervised audio-visual video parsing (WS-AVVP) aims to localize the
temporal extents of audio, visual and audio-visual event instances as well as
identify the corresponding event categories with only video-level category
labels for training. Most previous methods pay much attention to refining the
supervision for each modality or extracting fruitful cross-modality information
for more reliable feature learning. None of them have noticed the imbalanced
feature learning between different modalities in the task. In this paper, to
balance the feature learning processes of different modalities, a dynamic
gradient modulation (DGM) mechanism is explored, where a novel and effective
metric function is designed to measure the imbalanced feature learning between
audio and visual modalities. Furthermore, principle analysis indicates that the
multimodal confusing calculation will hamper the precise measurement of
multimodal imbalanced feature learning, which further weakens the effectiveness
of our DGM mechanism. To cope with this issue, a modality-separated decision
unit (MSDU) is designed for more precise measurement of imbalanced feature
learning between audio and visual modalities. Comprehensive experiments are
conducted on public benchmarks and the corresponding experimental results
demonstrate the effectiveness of our proposed method.

本文提出了一种利用动态梯度调制机制来平衡不同语音和视觉模态特征学习的方法，并设计了一种实现精确测量的模态分离决策单元来处理多模式混淆问题， 实验证明该方法的有效性。

多模态平衡感知梯度调控弱监督音视频解析

Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised  Audio-Visual Video Parsing

This paper focuses on the weakly-supervised audio-visual video parsing task,
which aims to recognize all events belonging to each modality and localize
their temporal boundaries. This task is challenging because only overall labels
indicating the video events are provided for training. However, an event might
be labeled but not appear in one of the modalities, which results in a
modality-specific noisy label problem. In this work, we propose a training
strategy to identify and remove modality-specific noisy labels dynamically. It
is motivated by two key observations: 1) networks tend to learn clean samples
first; and 2) a labeled event would appear in at least one modality.
Specifically, we sort the losses of all instances within a mini-batch
individually in each modality, and then select noisy samples according to the
relationships between intra-modal and inter-modal losses. Besides, we also
propose a simple but valid noise ratio estimation method by calculating the
proportion of instances whose confidence is below a preset threshold. Our
method makes large improvements over the previous state of the arts (e.g. from
60.0\% to 63.8\% in segment-level visual metric), which demonstrates the
effectiveness of our approach. Code and trained models are publicly available
at https://github.com/MCG-NJU/JoMoLD.

本文旨在解决弱监督下视听视频解析任务中的异态噪声标签问题，并提出了一种标签清理的训练策略。通过将每个模态中每个实例的损失进行排序并根据损失的关系选择噪声样本，我们的方法在视觉指标方面取得了明显进展，从而表明我们的方法是有效的。