This paper strives for repetitive activity counting in videos. Different from
existing works, which all analyze the visual video content only, we incorporate
for the first time the corresponding sound into the repetition counting
process. This benefits accuracy in challenging vision conditions such as
occlusion, dramatic camera view changes, low resolution, etc. We propose a
model that starts with analyzing the sight and sound streams separately. Then
an audiovisual temporal stride decision module and a reliability estimation
module are introduced to exploit cross-modal temporal interaction. For learning
and evaluation, an existing dataset is repurposed and reorganized to allow for
repetition counting with sight and sound. We also introduce a variant of this
dataset for repetition counting under challenging vision conditions.
Experiments demonstrate the benefit of sound, as well as the other introduced
modules, for repetition counting. Our sight-only model already outperforms the
state-of-the-art by itself, when we add sound, results improve notably,
especially under harsh vision conditions.

本文介绍了一种将视听融合用于视频中重复活动计数的新方法，通过使用音频信息可以在遇到困难的视觉条件下获得更精确的结果。通过引入音频学习模块和可靠性评估模块的跨模态时间交互进行分析，已经在一个数据集上对该模型进行了学习和评估，并针对具有挑战性的视觉条件引入了数据集变体。实验表明，使用音频以及其他模块可显著提高模型的性能。