The Audio-Visual Video Parsing task aims to identify and temporally localize
the events that occur in either or both the audio and visual streams of audible
videos. It often performs in a weakly-supervised manner, where only video event
labels are provided, \ie, the modalities and the timestamps of the labels are
unknown. Due to the lack of densely annotated labels, recent work attempts to
leverage pseudo labels to enrich the supervision. A commonly used strategy is
to generate pseudo labels by categorizing the known video event labels for each
modality. However, the labels are still confined to the video level, and the
temporal boundaries of events remain unlabeled. In this paper, we propose a new
pseudo label generation strategy that can explicitly assign labels to each
video segment by utilizing prior knowledge learned from the open world.
Specifically, we exploit the large-scale pretrained models, namely CLIP and
CLAP, to estimate the events in each video segment and generate segment-level
visual and audio pseudo labels, respectively. We then propose a new loss
function to exploit these pseudo labels by taking into account their
category-richness and segment-richness. A label denoising strategy is also
adopted to further improve the visual pseudo labels by flipping them whenever
abnormally large forward losses occur. We perform extensive experiments on the
LLP dataset and demonstrate the effectiveness of each proposed design and we
achieve state-of-the-art video parsing performance on all types of event
parsing, \ie, audio event, visual event, and audio-visual event. We also
examine the proposed pseudo label generation strategy on a relevant
weakly-supervised audio-visual event localization task and the experimental
results again verify the benefits and generalization of our method.

通过利用先前从开放世界学到的知识，我们提出了一种新的伪标签生成策略，可以将标签明确分配给每个视频片段，进一步使用新的损失函数来利用这些伪标签，并采用标签去噪策略来提升视觉伪标签。我们在 LLP 数据集上进行了广泛实验，证明了每个提议设计的有效性，并在所有类型的事件解析上取得了最先进的视频解析性能，且在相关的弱监督音频 - 视觉事件定位任务上验证了我们方法的优势和泛化性。

通过分段伪标签提升弱监督音视频解析

Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise  Pseudo Labeling

This paper studies the multimedia problem of temporal sentence grounding
(TSG), which aims to accurately determine the specific video segment in an
untrimmed video according to a given sentence query. Traditional TSG methods
mainly follow the top-down or bottom-up framework and are not end-to-end. They
severely rely on time-consuming post-processing to refine the grounding
results. Recently, some transformer-based approaches are proposed to
efficiently and effectively model the fine-grained semantic alignment between
video and query. Although these methods achieve significant performance to some
extent, they equally take frames of the video and words of the query as
transformer input for correlating, failing to capture their different levels of
granularity with distinct semantics. To address this issue, in this paper, we
propose a novel Hierarchical Local-Global Transformer (HLGT) to leverage this
hierarchy information and model the interactions between different levels of
granularity and different modalities for learning more fine-grained multi-modal
representations. Specifically, we first split the video and query into
individual clips and phrases to learn their local context (adjacent dependency)
and global correlation (long-range dependency) via a temporal transformer.
Then, a global-local transformer is introduced to learn the interactions
between the local-level and global-level semantics for better multi-modal
reasoning. Besides, we develop a new cross-modal cycle-consistency loss to
enforce interaction between two modalities and encourage the semantic alignment
between them. Finally, we design a brand-new cross-modal parallel transformer
decoder to integrate the encoded visual and textual features for final
grounding. Extensive experiments on three challenging datasets show that our
proposed HLGT achieves a new state-of-the-art performance.

本文介绍了一种新的 Hierarchical Local-Global Transformer 模型来解决视频和文本之间的 fine-grained 语义对齐问题，并将其应用于 temporal sentence grounding 任务，通过跨模态平行变压器解码器将其编码为最终的基础。