We present Audiovisual Moments in Time (AVMIT), a large-scale dataset of
audiovisual action events. In an extensive annotation task 11 participants
labelled a subset of 3-second audiovisual videos from the Moments in Time
dataset (MIT). For each trial, participants assessed whether the labelled
audiovisual action event was present and whether it was the most prominent
feature of the video. The dataset includes the annotation of 57,177 audiovisual
videos, each independently evaluated by 3 of 11 trained participants. From this
initial collection, we created a curated test set of 16 distinct action
classes, with 60 videos each (960 videos). We also offer 2 sets of pre-computed
audiovisual feature embeddings, using VGGish/YamNet for audio data and
VGG16/EfficientNetB0 for visual data, thereby lowering the barrier to entry for
audiovisual DNN research. We explored the advantages of AVMIT annotations and
feature embeddings to improve performance on audiovisual event recognition. A
series of 6 Recurrent Neural Networks (RNNs) were trained on either
AVMIT-filtered audiovisual events or modality-agnostic events from MIT, and
then tested on our audiovisual test set. In all RNNs, top 1 accuracy was
increased by 2.71-5.94\% by training exclusively on audiovisual events, even
outweighing a three-fold increase in training data. We anticipate that the
newly annotated AVMIT dataset will serve as a valuable resource for research
and comparative experiments involving computational models and human
participants, specifically when addressing research questions where audiovisual
correspondence is of critical importance.

我们提供了一个大规模的音频视觉动作事件数据集 (AVMIT)，它包含了被 11 名受训者独立评估的 57,177 个音频视觉视频的标注结果。这个数据集还附带了预先计算好的音频和视觉特征嵌入，并基于此进行了音频视觉事件识别性能的改进研究。通过在 6 个递归神经网络 (RNNs) 上进行训练和测试，我们发现专门使用音频视觉事件进行训练可以显著提高识别性能。我们有理由相信，这个新标注的 AVMIT 数据集将成为研究和比较实验的有价值资源，特别是在涉及到音频视觉对应关系的研究问题中。