Existing action localization approaches adopt shallow temporal convolutional
networks (\ie, TCN) on 1D feature map extracted from video frames. In this
paper, we empirically find that stacking more conventional temporal convolution
layers actually deteriorates action classification performance, possibly
ascribing to that all channels of 1D feature map, which generally are highly
abstract and can be regarded as latent concepts, are excessively recombined in
temporal convolution. To address this issue, we introduce a novel concept-wise
temporal convolution (CTC) layer as an alternative to conventional temporal
convolution layer for training deeper action localization networks. Instead of
recombining latent concepts, CTC layer deploys a number of temporal filters to
each concept separately with shared filter parameters across concepts. Thus can
capture common temporal patterns of different concepts and significantly enrich
representation ability. Via stacking CTC layers, we proposed a deep
concept-wise temporal convolutional network (C-TCN), which boosts the
state-of-the-art action localization performance on THUMOS'14 from 42.8 to 52.1
in terms of mAP(\%), achieving a relative improvement of 21.7\%. Favorable
result is also obtained on ActivityNet.

本文提出了一种概念级别的时间卷积层（CTC），用于训练深度行动本地化神经网络，通过在每个概念上分别使用多个时间滤波器捕捉不同概念的共同时间模式，极大地丰富了表示能力，通过堆叠 CTC 层，提出了一种深度概念级别的时间卷积神经网络（C-TCN），在 THUMOS'14 上将 mAP (%) 从 42.8 提高到 52.1，相对提高了 21.7\%。