Inspired by the observation that humans are able to process videos
efficiently by only paying attention where and when it is needed, we propose an
interpretable and easy plug-in spatial-temporal attention mechanism for video
action recognition. For spatial attention, we learn a saliency mask to allow
the model to focus on the most salient parts of the feature maps. For temporal
attention, we employ a convolutional LSTM based attention mechanism to identify
the most relevant frames from an input video. Further, we propose a set of
regularizers to ensure that our attention mechanism attends to coherent regions
in space and time. Our model not only improves video action recognition
accuracy, but also localizes discriminative regions both spatially and
temporally, despite being trained in a weakly-supervised manner with only
classification labels (no bounding box labels or time frame temporal labels).
We evaluate our approach on several public video action recognition datasets
with ablation studies. Furthermore, we quantitatively and qualitatively
evaluate our model's ability to localize discriminative regions spatially and
critical frames temporally. Experimental results demonstrate the efficacy of
our approach, showing superior or comparable accuracy with the state-of-the-art
methods while increasing model interpretability.

通过引入可解释的时空注意力机制来提高视频动作识别的准确性和模型解释性，并使用一组正则化器对其进行约束。利用弱监督的方式仅使用分类标签，模型不仅提高了准确性，还能时空自动定位区分性区域。