Self-supervised pre-training recently demonstrates success on large-scale
multimodal data, and state-of-the-art contrastive learning methods often
enforce the feature consistency from cross-modality inputs, such as video/audio
or video/text pairs. Despite its convenience to formulate and leverage in
practice, such cross-modality alignment (CMA) is only a weak and noisy
supervision, since two modalities can be semantically misaligned even they are
temporally aligned. For example, even in the commonly adopted instructional
videos, a speaker can sometimes refer to something that is not visually present
in the current frame; and the semantic misalignment would only be more
unpredictable for the raw videos from the internet. We conjecture that might
cause conflicts and biases among modalities, and may hence prohibit CMA from
scaling up to training with larger and more heterogeneous data. This paper
first verifies our conjecture by observing that, even in the latest VATT
pre-training using only instructional videos, there exist strong gradient
conflicts between different CMA losses within the same video, audio, text
triplet, indicating them as the noisy source of supervision. We then propose to
harmonize such gradients, via two techniques: (i) cross-modality gradient
realignment: modifying different CMA loss gradients for each sample triplet, so
that their gradient directions are more aligned; and (ii) gradient-based
curriculum learning: leveraging the gradient conflict information on an
indicator of sample noisiness, to develop a curriculum learning strategy to
prioritize training on less noisy sample triplets. Applying those techniques to
pre-training VATT on the HowTo100M dataset, we consistently improve its
performance on different downstream tasks. Moreover, we are able to scale VATT
pre-training to more complicated non-narrative Youtube8M dataset to further
improve the state-of-the-arts.

本文讨论了跨模态对齐的效果和问题，提出了通过梯度调整和基于梯度的课程学习解决梯度冲突的方法，并将其应用于视频音频文本模态的预训练中以提高性能。