We present the Moments in Time Dataset, a large-scale human-annotated
collection of one million short videos corresponding to dynamic events
unfolding within three seconds. Modeling the spatial-audio-temporal dynamics
even for actions occurring in 3 second videos poses many challenges: meaningful
events do not include only people, but also objects, animals, and natural
phenomena; visual and auditory events can be symmetrical in time ("opening" is
"closing" in reverse), and either transient or sustained. We describe the
annotation process of our dataset (each video is tagged with one action or
activity label among 339 different classes), analyze its scale and diversity in
comparison to other large-scale video datasets for action recognition, and
report results of several baseline models addressing separately, and jointly,
three modalities: spatial, temporal and auditory. The Moments in Time dataset,
designed to have a large coverage and diversity of events in both visual and
auditory modalities, can serve as a new challenge to develop models that scale
to the level of complexity and abstract reasoning that a human processes on a
daily basis.

Moments in Time 数据集是一个包含 100 万短视频的大型人工注释集合，主要用于研究动态事件的空间 - 音频 - 时间动态建模，包括 339 个类别的活动标注，可作为发展至人类日常处理的复杂性和抽象推理水平的模型的新挑战。