Recognizing instances at different scales simultaneously is a fundamental
challenge in visual detection problems. While spatial multi-scale modeling has
been well studied in object detection, how to effectively apply a multi-scale
architecture to temporal models for activity detection is still under-explored.
In this paper, we identify three unique challenges that need to be specifically
handled for temporal activity detection compared to its spatial counterpart. To
address all these issues, we propose Dynamic Temporal Pyramid Network (DTPN), a
new activity detection framework with a multi-scale pyramidal architecture
featuring three novel designs: (1) We sample input video frames dynamically
with varying frame per seconds (FPS) to construct a natural pyramidal input for
video of an arbitrary length. (2) We design a two-branch multi-scale temporal
feature hierarchy to deal with the inherent temporal scale variation of
activity instances. (3) We further exploit the temporal context of activities
by appropriately fusing multi-scale feature maps, and demonstrate that both
local and global temporal contexts are important. By combining all these
components into a uniform network, we end up with a single-shot activity
detector involving single-pass inferencing and end-to-end training. Extensive
experiments show that the proposed DTPN achieves state-of-the-art performance
on the challenging ActvityNet dataset.

本文提出了 “动态时间金字塔网络”（DTPN）来解决活动检测领域中的主要挑战，包括如何有效应用多尺度架构以及如何处理活动实例的固有时间尺度变化。通过将多种新设计结合为统一网络，实现了单次推理和端到端训练，从而取得了状态的最佳性能。