Video recognition has been advanced in recent years by benchmarks with rich
annotations. However, research is still mainly limited to human action or
sports recognition - focusing on a highly specific video understanding task and
thus leaving a significant gap towards describing the overall content of a
video. We fill this gap by presenting a large-scale "Holistic Video
Understanding Dataset"~(HVU). HVU is organized hierarchically in a semantic
taxonomy that focuses on multi-label and multi-task video understanding as a
comprehensive problem that encompasses the recognition of multiple semantic
aspects in the dynamic scene. HVU contains approx.~572k videos in total with 9
million annotations for training, validation, and test set spanning over 3142
labels. HVU encompasses semantic aspects defined on categories of scenes,
objects, actions, events, attributes, and concepts which naturally captures the
real-world scenarios.
We demonstrate the generalization capability of HVU on three challenging
tasks: 1.) Video classification, 2.) Video captioning and 3.) Video clustering
tasks. In particular for video classification, we introduce a new
spatio-temporal deep neural network architecture called "Holistic Appearance
and Temporal Network"~(HATNet) that builds on fusing 2D and 3D architectures
into one by combining intermediate representations of appearance and temporal
cues. HATNet focuses on the multi-label and multi-task learning problem and is
trained in an end-to-end manner. Via our experiments, we validate the idea that
holistic representation learning is complementary, and can play a key role in
enabling many real-world applications.

本文介绍了一个基于语义分类的大规模 Holistic 视频理解数据集 HVU，它实现了对多标签、多任务视频理解的全面考虑，以及介绍了一种新的综合外观和时间神经网络体系结构 HATNet，本文拓展了视频识别范畴，验证了 Holistic 表示学习在许多实际应用程序中的重要作用。