This paper introduces the task of Auditory Referring Multi-Object Tracking
(AR-MOT), which dynamically tracks specific objects in a video sequence based
on audio expressions and appears as a challenging problem in autonomous
driving. Due to the lack of semantic modeling capacity in audio and video,
existing works have mainly focused on text-based multi-object tracking, which
often comes at the cost of tracking quality, interaction efficiency, and even
the safety of assistance systems, limiting the application of such methods in
autonomous driving. In this paper, we delve into the problem of AR-MOT from the
perspective of audio-video fusion and audio-video tracking. We put forward
EchoTrack, an end-to-end AR-MOT framework with dual-stream vision transformers.
The dual streams are intertwined with our Bidirectional Frequency-domain
Cross-attention Fusion Module (Bi-FCFM), which bidirectionally fuses audio and
video features from both frequency- and spatiotemporal domains. Moreover, we
propose the Audio-visual Contrastive Tracking Learning (ACTL) regime to extract
homogeneous semantic features between expressions and visual objects by
learning homogeneous features between different audio and video objects
effectively. Aside from the architectural design, we establish the first set of
large-scale AR-MOT benchmarks, including Echo-KITTI, Echo-KITTI+, and Echo-BDD.
Extensive experiments on the established benchmarks demonstrate the
effectiveness of the proposed EchoTrack model and its components. The source
code and datasets will be made publicly available at
this https URL

通过音频和视频融合的双流视觉变换框架，本研究提出了 EchoTrack 模型，解决了自动驾驶中的 Auditory Referring Multi-Object Tracking 问题，并建立了一套大规模的 AR-MOT 基准测试集。