Diverse actions give rise to rich audio-visual signals in long videos. Recent
works showcase that the two modalities of audio and video exhibit different
temporal extents of events and distinct labels. We address the interplay
between the two modalities in long videos by explicitly modelling the temporal
extents of audio and visual events. We propose the Time Interval Machine (TIM)
where a modality-specific time interval poses as a query to a transformer
encoder that ingests a long video input. The encoder then attends to the
specified interval, as well as the surrounding context in both modalities, in
order to recognise the ongoing action.
We test TIM on three long audio-visual video datasets: EPIC-KITCHENS,
Perception Test, and AVE, reporting state-of-the-art (SOTA) for recognition. On
EPIC-KITCHENS, we beat previous SOTA that utilises LLMs and significantly
larger pre-training by 2.9% top-1 action recognition accuracy. Additionally, we
show that TIM can be adapted for action detection, using dense multi-scale
interval queries, outperforming SOTA on EPIC-KITCHENS-100 for most metrics, and
showing strong performance on the Perception Test. Our ablations show the
critical role of integrating the two modalities and modelling their time
intervals in achieving this performance. Code and models at:
this https URL

在长视频中，多样的行为产生丰富的视听信号。我们通过显式地建模音频和视觉事件的时间范围，解决了长视频中这两种模态之间的相互作用。我们提出了一种时间间隔机器（TIM），其中以特定模态的时间间隔作为查询，将长视频输入馈入转换器编码器。然后，编码器会关注指定的时间间隔以及两种模态中的周围上下文，以识别正在进行的动作。我们在三个长音视频数据集上测试了 TIM，并在识别方面报告了最先进的结果。此外，我们展示了 TIM 可以适用于动作检测，并在大多数指标上超越了先前的最先进技术。我们的实验结果表明，在实现这种性能中，整合两种模态并对它们的时间间隔建模起着关键作用。