Video localization tasks aim to temporally locate specific instances in
videos, including temporal action localization (TAL), sound event detection
(SED) and audio-visual event localization (AVEL). Existing methods
over-specialize on each task, overlooking the fact that these instances often
occur in the same video to form the complete video content. In this work, we
present UniAV, a Unified Audio-Visual perception network, to achieve joint
learning of TAL, SED and AVEL tasks for the first time. UniAV can leverage
diverse data available in task-specific datasets, allowing the model to learn
and share mutually beneficial knowledge across tasks and modalities. To tackle
the challenges posed by substantial variations in datasets
(size/domain/duration) and distinct task characteristics, we propose to
uniformly encode visual and audio modalities of all videos to derive generic
representations, while also designing task-specific experts to capture unique
knowledge for each task. Besides, we develop a unified language-aware
classifier by utilizing a pre-trained text encoder, enabling the model to
flexibly detect various types of instances and previously unseen ones by simply
changing prompts during inference. UniAV outperforms its single-task
counterparts by a large margin with fewer parameters, achieving on-par or
superior performances compared to state-of-the-art task-specific methods across
ActivityNet 1.3, DESED and UnAV-100 benchmarks.

UniAV 是一种统一的视听感知网络，可以联合学习时间动作定位（TAL）、声音事件检测（SED）和视听事件定位（AVEL）任务，并通过使用预训练的文本编码器设计统一的语言感知分类器，实现对各种类型实例的灵活检测。UniAV 通过更少的参数比单一任务模型，在 ActivityNet 1.3、DESED 和 UnAV-100 基准测试中取得与最先进的任务特定方法相当或优秀的性能。